U NIVERSAL G RAPH N EURAL N ETWORKS WITHOUT M ESSAGE P ASSING
Anonymous authors
Paper under double-blind review
A BSTRACT
Message Passing Graph Neural Networks (MP-GNNs) have become the de facto paradigm for learning on graph for years.
Nevertheless, recent works also obtain promising empirical results with other kinds of architectures like global selfattention and even MLPs.
This raises an important theoretical question: what is the minimal prerequisite for an expressive graph model?
In this work, we theoretically show that when equipped with proper position encodings, even a simple
Bag-of-Nodes (BoN) model (node-wise MLP followed by global readout) can be universal on graphs.
We name this model as Universal Bag-of-Nodes (UBoN).
Synthetic experiments on the EXP dataset show that UBoN indeed achieves expressive power beyond 1-WL test.
On real-world graph classiﬁcation tasks, UBoN also obtains comparable performance to MP-GNNs while enjoying better training and inference efﬁciency (50% less training time compared to GCN).
We believe that our theoretical and empirical results might inspire more research on simple and expressive GNN architectures.
I NTRODUCTION
In recent years, Message Passing Graph Neural Networks (MP-GNNs) become the prevailing paradigm in the development of Graph Neural Networks (GNNs).
Numerous representative GNNs belong to this family, such as GCN (Welling & Kipf, 2017), GraphSAGE (Hamilton et al., 2017),
GAT (Veli ˇ ckovi ´ c et al., 2018), and GIN (Xu et al., 2019).
However, it is well known that due to the nature of message passing, the expressive power of MP-GNNs is fundamentally limited by 1-WL test (Weisfeiler & Leman, 1968; Xu et al., 2019).
In other words, they can not distinguish a pair of non-isomorphic graphs whose 1-WL test fails, e.g., k -regular graphs.
Furthermore, the frequent message passing between nodes also introduces a signiﬁcant computation overhead in MP-GNNs, which is particularly problematic on large graphs, e.g., the neighbor exploration problem (Zeng et al., 2021).
Thus, it seems that existing GNNs are largely limited by the message passing scheme and have both theoretical and empirical demerits, which motivates us to explore efﬁcient and yet powerful alternative paradigms for designing GNNs.
Recently, there has been a surge of interest of directly applying Multi-Layer Perceptrons (MLPs) to graphs due to its simplicity and efﬁciency.
Nevertheless, since most of them still have to pre-process (Wu et al., 2019) or post-process (Huang et al., 2021)
the graphs using message-passing operations, these variants still inherent the theoretical limitations of MP-GNNs, as pointed out by Chen et al.
Recently, Hu et al.
(2021) show that with a proper training objective, MLPs alone can obtain comparable performance to MP-GNNs on node classiﬁcation tasks while being much more efﬁcient.
However, it is easy to see that, theoretically, these MLPs are even more limited than MP-GNNs in expressive power because they do not have structural information.
This fact draws our interest to the following problem: can we design a theoretically powerful graph MLP by simply incorporating the structural information as auxiliary input features?
In this paper, we give an afﬁrmative answer to this question by proposing a powerful graph MLP for graph classiﬁcation tasks, named Universal Bag-of-Nodes (UBoN).
UBoN only contains three simple modules: a node-level MLP, a global readout layer (e.g., sum), and a graph-level MLP.
As the name suggests, we prove that UBoN enjoys universal expressive power (the highest level of expressiveness among GNNs).
Consequently, UBoN is not only more powerful than canonical
MP-GNNs with 1-WL expressiveness, but also more powerful than high-order GNN variants with k -WL expressiveness, such as 1-2-3-GNN (Morris et al., 2019).
In fact, with universal expressiveness, UBoN is capable of distinguishing any two non-isomorphic graphs, making it one of the most powerful GNNs while being arguably the simplest.
The key part of the proposed UBoN lies in a newly designed permutation-invariant structural embedding, namely, the eigenvalue-aware spectral embedding (EASE).
EASE is built upon the canonical spectral embedding (SE) that utilizes the eigenvectors of the Laplacian matrix, a.k.a. , Laplacian embedding.
SE has demonstrated vital importance on many graph classiﬁcation tasks (Dwivedi et al., 2020; Yun et al., 2019;
Ying et al., 2019) while its drawback is that we cannot reconstruct the graph from it due to the loss of eigenvalue information.
Therefore, we propose to incorporate eigenvalues to SE, based on which we can already demonstrate the universality of UBoN. Further concerning the non-uniqueness of eigen-decomposition (Wang et al., 2022), we resolve this problem through two extensions of EASE that ensure its sign invariance and basis invariance.
We demonstrate that our proposed UBoN is not only theoretically powerful, but also practically efﬁcient for many graphlevel tasks, which offers a promising alternative paradigm to existing GNNs.
Our contribution is summarized as follows
: • In view of the limitations of MP-GNNs, we advocate for a simpler but yet more powerful paradigm for designing GNNs without using message passing.
In particular, we show that for the ﬁrst time, we can achieve universal expressiveness on graphs using a simple Bagof-Nodes classiﬁer, named Universal Bag-of-Nodes (UBoN).
• To achieve the universality of UBoN, we design a new kind of structural embedding that preserves all structural information, named eigenvalue-aware spectral embedding (EASE).
We show that a simple BoN can obtain universality with EASE, and further achieves sign and basis invariance through two extensions of EASE.
• We evaluate UBoN on both synthetic and real-world datasets.
On the one hand, UBoN indeed demonstrates expressive power beyond the 1-WL test.
On the other hand, UBoN also achieves comparable performance to MP-GNNs on ﬁve real-world graph classiﬁcation tasks, while requiring less training time and fewer parameters.
R ELATED WORK
Graph MLPs MLPs have been used in computer vision and shown promising results compared to prevalent models (Tolstikhin et al., 2021; Melas-Kyriazi, 2021).
Recently, several works proposed to use MLPs on graph representation learning.
Hu et al.
(2021) proposed a neighboring contrastive loss to allow MLPs to utilize the adjacency information implicitly.
Chen et al.
(2021) proposed graph-augmented MLPs to augment node features with multi-hop operators before applying learnable node-wise functions, but their expressive power is less powerful than GNNs.
Zhang et al.
(2022) brought GNNs and MLPs together via knowledge distillation and showed their model can even outperform the teacher GNNs on smaller datasets, but their model is less expressive than GNNs as well.
Compared with prior works, our proposed MLP-based model is universal and more efﬁcient.
Expressive Power of GNNs It has been shown that MP-GNNs are not more powerful than the WL test (Xu et al., 2019; Morris et al., 2019).
Since then, many attempts have been made to endow graph neural networks with expressive power beyond the 1-WL test, such as injecting random attributes (Murphy et al., 2019;
Sato et al., 2021;
You et al., 2019; Srinivasan & Ribeiro, 2020; Dwivedi et al., 2020), injecting positional encodings (Li et al., 2020;
You et al., 2021), and designing higher-order GNNs (Morris et al., 2019; Maron et al., 2019b;c;a; Chen et al., 2019).
Br¨uel Gabrielsson (2020) produced a framework for constructing universal function approximators on graph isomorphism classes.
Most of these attempts still rely on the redundant message passing modules, and suffer from high computational cost or sophisticated network structure.
In this paper, we propose a simple
MLP-based network that is computationally efﬁcient and has universal expressive power.
Positional Encodings Recently, there are many works that aim to improve the expressive power of
GNNs by injecting positional encoding (PE).
Murphy et al.
(2019) assigned each node an identiﬁer depending on its index.
Li et al.
(2020) proposed to use distances between nodes characterized by powers of the random walk matrix.
You et al.
(2019) proposed to use distances of nodes to a sampled anchor set of nodes.
Using random node features can also improve the expressiveness of GNNs, even making them universal (Sato et al., 2021; Abboud et al., 2021). Dasoulas et al. (2021) constructed node representations related to the structural identity of the neighborhood of each node.
More detailed discussion of these PE methods can be found in Appendix A. Compared with previous methods, EASE is computationally efﬁcient with fast convergence, while attaining universal expressiveness. Accelerating Graph Learning There have been many works that aim to speed up graph learning in many ways, such as implementing simpler network structure (Wu et al., 2019; Hu et al., 2021), optimizing the training procedure (Kaler et al., 2022), etc. Wu et al.
(2019) proposed a linear model that works well on node classiﬁcation, but as the authors pointed out in their paper, such linear networks do not work well on graph-level tasks. Hu et al. (2021) proposed an MLP model with NContrast loss that provides the model with structural information, but computing the NContrast loss is computationally expensive during training.
In our work, the computation of eigendecompositions of our MLP model happens during pre-processing, thus it is more computationally efﬁcient.
P RELIMINARIES
We denote a graph G =
( V , E , X ) where V is the vertex set of size n , E is the edge set, and
X ∈ R n × d are the input node features.
We denote A as the adjacency matrix, and let ˆ A =
D − 12 ˜ AD − 12 =
D − 12 ( I + A ) D − 12 be the normalized adjacency matrix, where I denotes the augmented self-loop, D is the diagonal degree matrix of ˜ A deﬁned by D i,i = (cid:80) nj =1 ˜
A i,j .
ˆ A can be decomposed as ˆ A = UΛ U T , where Λ = diag( λ ) is the diagonal matrix of eigenvalues, the i -th column of U is the eigenvector corresponding to the eigenvalue
λ
i , λ 1 ≤ · · · ≤ λ n .
We denote concatenation along the last dimension with brackets, i.e. , for A , B ∈ R n × d , [ A , B ] ∈ R n × 2 d .
The expressive power of GNNs is characterized by its ability to approximate permutation invariant functions on general graphs.
Let Ω ⊂ R n × d × R n × n be a compact set of graphs with [ X , ˆ A ] ∈ Ω .
Consider a function f acting on Ω .
We deﬁne its permutation invariance as follows.
Deﬁnition 1.
A function f ([ X , ˆ A ]) operating on graphs [ X , ˆ A ] ∈ Ω is permutation invariant if for all permutation matrix P ∈ R n × n , f ([ P X , P ˆ AP T ]) =
f
([ X , ˆ A ]) .
In graph-level tasks like graph classiﬁcation and graph regression, we learn a GNN to approximate the ground-truth function.
A most expressive GNN can approximate arbitrary permutation-invariant functions on graphs up to arbitrary precision, for which we say it has universal expressive power.
Deﬁnition 2.
Let Ω ⊂ R n × d × R n × n be a compact set of graphs with [ X , ˆ A ] ∈ Ω , where X are the node features and ˆ A is the normalized adjacency matrix.
A GNN is said to be universal , if for all continuous permutation invariant functions f deﬁned over Ω and arbitrary ε > 0 , there exists network parameters so that | GNN( X , ˆ A ) − f
([ X , ˆ A ]) | < ε , for all graphs [ X , ˆ A ] ∈ Ω .
3. E XPRESSIVE POWER LIMIT OF MP-GNN S
In recent years, Message-Passing Graph Neural Networks (MP-GNNs) have become the de facto paradigm for graph representation learning.
In MP-GNNs, graph convolution is often applied by message passing layers.
Consider an MP-GNN with K message passing layers.
Let h ( k ) v be the feature vector of node v at the k -th layer, we have h ( k ) v =
COMBINE ( k ) ( h ( k − 1) v , AGGREGATE ( k ) ( { h ( k − 1) u | u ∈ N ( v ) } )) , for k = 1 , 2 , . .
, K , where N ( v ) is the set of adjacent nodes to node v .
For graph classiﬁcation tasks, we need a readout layer to obtain the ﬁnal graph representation vector h G : h G =
READOUT( { h ( K ) v | v ∈ G} ) , and the prediction of graph label is y G
= g ( h G ) , where g can be a linear classiﬁer, a Multi-Layer Perceptron, etc .
For node classiﬁcation tasks, the label of node v can be predicted as y v =
f
( h v ) .
However, it is well-known that the expressive power of MP-GNNs is limited by the 1-WL test (Xu et al., 2019).
That is, if all node features are the same, and 1-WL test does not distinguish two graphs, then MP-GNNs will fail to distinguish them as well.
For example, consider the two graphs in Figure 1.
1-WL test cannot distinguish these two graphs, despite them being non-isomorphic.
Thus any MP-GNN will produce identical outputs with these graphs as inputs.
If a graph function assigns different values to these two graphs, MP-GNNs will fail to approximate it.
This fact inspires us to design more powerful GNNs with expressiveness beyond 1-WL, or even universal expressive power.
B UILDING UNIVERSAL GNN S WITHOUT MESSAGE PASSING
In view of the limitations of existing MP-GNNs, in this section, we resort to a new approach and design a simple
Bag-of-Node (BoN) classiﬁer with universal expressive power.
In Section 4.1, we introduce the vanilla BoN and discuss its limitations due to the lack of structural information.
Next, in Section 4.2, we develop a new kind of structural embedding named EASE, based on which we prove that the universality of our Universal BoN (UBoN).
In Section 4.3, we design two extensions that achieve the uniqueness of UBoN under different decompositions.
4. B AG OF N ODES (B O N)
As mentioned above, the expressive power of MP-GNNs is limited by 1-WL test, while the message passing modules in MP-GNNs also bring additional time consumption.
So a natural idea is to remove the message passing modules to achieve higher efﬁciency and more powerful expressiveness.
As an alternative to MP-GNNs, we explore the use of MLPs on graph-level tasks.
Given node features X ∈ R n × d , a naive approach to generate graph-level embeddings or predictions with MLP is to simply feed X to a node-level MLP, followed by a readout function and a graph-level MLP.
Both the node-level and graph-level MLPs are compositions of linear and activation layers.
The node-level MLP ﬁrst acts on each node feature independently to produce node-level embeddings
H ∈ R n × d (cid:48) , which is then passed to a readout function.
The readout function (e.g., sum ) aggregates all the node-level embeddings to produce a graph-level embedding h ∈ R d (cid:48) .
Finally, the graph-level
MLP acts on h to produce the ﬁnal prediction y .
The entire model can be formulated as:
[Equation]
Similar to Bag of Words (BoW) in natural language processing, the network in Equation 1 considers the node features of the graph as a multiset and aggregates them by a readout function disregarding their dependence.
We denote the model described in Equation 1 as Bag of Nodes (BoN).
By removing the message passing modules, BoN is free from computational costs caused by neighborhood aggregation in MP-GNNs.
However, as BoN has no information about the graph structure, it fails to achieve powerful expressiveness in terms of approximating graph functions dependent on the graph topology.
Thus, we need to add structural information to enhance its expressive power.
4. U NIVERSAL B O N (UB O N)
The lack of information about the graph structure limits the expressiveness of BoN. One way to address this issue is to incorporate the structural information as auxiliary input features, a.k.a. , positional encodings (PE).
Among different PEs, the classical Spectral embedding (SE) (Belkin &
Niyogi, 2001) has shown promising performance recently (Dwivedi et al., 2020; Yun et al., 2019).
SE utilizes the eigenvectors of the Laplacian matrix as positional encodings of nodes and is equivariant to different graph permutations.
Deﬁnition 3 (SE) .
The spectral embedding of a graph is constructed by (a partial set of) the eigenvectors U of the Laplacian matrix to the input node features, where L = UΛ U T is the eigendecomposition of the Laplacian matrix.
We can augment the input features X with the SE embedding, and get an augmented input
X (cid:48) =
[ X , U ] for GNNs.
In this way, SE provides the GNN with structural information about the graph, and provides signiﬁcant beneﬁts on graph-level tasks.
However, combining BoN networks with SE does not give the network universal expressive power, since SE lacks crucial information about eigenvalues of the graph.
For instance, consider two non-isomorphic graphs with identical
L i n ea r
A c ti v a ti on
L i n ea r
MLP s u m r ea dou t
L i n ea r
A c ti v a ti on
L i n ea r
MLP x (cid:48)x (cid:48)...
x (cid:48) n hh 2 .
h n h
X (cid:48) =
[ X , UΛ 12 ] ...
[Figure]
eigenvectors but different eigenvalues of the Laplacian matrix.
Although they are not isomorphic, the spectral embeddings of these two graphs are the same, which indicates the BoN network is not able to tell them apart.
For the BoN network to achieve universal expressive power, we propose eigenvalue-aware spectral embedding (EASE) to incorporate eigenvalues to SE.
Deﬁnition 4 (EASE) .
The eigenvalue-aware spectral embedding of a graph is constructed by a combination of the eigenvector and the eigenvalue matrix, i.e., UΛ 12 , where ˆ A =
UΛ U T is the eigen-decomposition of ˆ A .
Compared with SE, EASE provides the GNN with additional information of the eigenvalues.
As we will prove, incorporating EASE to BoN networks already grants universal expressive power.
By applying EASE, we propose a universal and permutation-invariant MLP-based network, named
Universal BoN (UBoN).
The eigenvalue-aware spectral embeddings of nodes are generated in the preprocessing period, which are then fed into a permutation-invariant BoN network (Section 4.1).
The outputs are used as the ﬁnal graph-level prediction.
The entire model is formulated as:
[Equation]
The overview of UBoN is shown in Figure 2.
Its universality is guaranteed as follows.
Theorem 1.
Let Ω ⊂ R n × d × R n × n be a compact set of graphs, [ X , ˆ A ] ∈ Ω .
Given a continuous invariant graph function f deﬁned over Ω and arbitrary ε > 0 , there exists a permutation-invariant
BoN network with parameters such that for all graphs [ X , ˆ A ] ∈ Ω ,
[Equation]
Theorem 1 shows UBoN can approximate arbitrary continuous invariant graph function, which indicates its universality on graph-level tasks.
In this way, as simple as it looks like, UBoN is not only more powerful than most MP-GNNs with 1-WL expressiveness, but also more expressive than higher-order GNNs with k -WL expressiveness.
Our proof is in Appendix B.1.
In practice, it is not necessary to use all eigenvectors and eigenvalues of the input graphs.
Extensive studies have shown that the high-frequency components of the graph are often unhelpful for learning good representations and improving the model’s generalization performance (Balcilar et al., 2021;
Hoang & Maehara, 2020).
Besides, using all eigenvectors and eigenvalues of the graph also increases the computational costs.
Thus, we propose to use the ﬁrst k low-frequency components only.
Taking the largest k eigenvectors reduces overﬁtting caused by high frequency components of ˆ A and improves the model’s generalization performance.
It also lessens the computational costs.
We can prove that under mild conditions, the loss of expressive power induced by this action can be upper bounded, as shown in the following theorem.
Theorem 2.
Let Ω ⊂ R n × d × R n × n be a compact set of graphs, [ X , ˆ A ] ∈ Ω .
Given an invariant graph function f deﬁned over Ω that can be ε -approximated by an L p -Lipschitz continuous function and arbitrary ε > 0 , for any integer 0 < k ≤ n , there exists a permutation-invariant BoN network with parameters such that for all graphs [ X , ˆ A ] ∈ Ω ,
[Equation]
([ X , ( UΛ 1 2 ) : , − k : , 0 ]) | < √ n − kL p λ n − k + ε. Here the L p -Lipschitz continuity of f is deﬁned using the Frobenius norm on the input domain, 0 ≤ λ 1 ≤ · · · ≤ λ n ≤ 2 are the eigenvalues of ˆ A , 0 ∈ R n × ( n − k ) .
We can see from Theorem 2 that the upper bound of the loss of expressive power decreases when k increases, and when k = n , our model becomes universal.
Proof of Theorem 2 is in Appendix B.2.
Compared to prior works on graph MLPs (Hu et al., 2021; Chen et al., 2021; Zhang et al., 2022), our approach has several advantages.
First, it overcomes the expressive power limitation of MLPs, making them universal on graph-level tasks.
Secondly, the structure of our network is fairly simple and easy to implement.
By removing the redundant message passing modules, our model costs less time to train and has less number of parameters, making them more efﬁcient than traditional
MP-GNNs, while still achieving comparable or even better performance on real-world datasets.
4. O N THE NON - UNIQUENESS OF SPECTRAL EMBEDDING
By Theorem 1, we have shown that a permutation-invariant MLP-based network can achieve universality on arbitrary graphs.
However, as we make use of the eigendecomposition of the adjacency matrix ˆ A , the non-uniqueness of the eigendecomposition could cause problems.
The non-uniqueness of decomposition lies in two aspects.
First is the ambiguity in sign.
For instance, if u λ i is an eigenvector corresponding to the eigenvalue
λ i , then
− u λ i is also an eigenvector corresponding to λ
i .
Second is the ambiguity in bases.
If a eigenvalue λ i has multiplicity degree d i > 1 , and columns of U λ i ∈ R n × d i are the orthonormal basis of its eigenspace, then for any orthogonal matrix Q ∈ R d i × d i , columns of U λ i Q are also the orthonormal basis of its eigenspace.
Prior works suggest that the two ambiguities can be potentially harmful for model’s stability (Lim et al., 2022;
Wang et al., 2022).
Below, we give their formal deﬁnitions.
Deﬁnition 5.
A function f acting on eigenvectors u 1 , u 2 , . .
, u n is sign invariant if for all sign choices s i ∈ { 1 , − 1 } , f ( u 1 , u 2 , . . .
, u n ) =
f ( s 1 u 1 , s 2 u 2 , . . .
, s n u n ) .
Deﬁnition 6.
A function f acting on eigenspaces
U 1 , U 2 , . . .
, U n ( U i ∈
R n × d i ) is basis invariant if for all orthogonal matrices
Q i ∈ R
d i × d i , f ( U 1 , U 2 , . . .
, U n ) =
f ( U 1 Q , U 2 Q , . .
U n Q ) .
One way to address the two ambiguities in sign and basis is to inject randomness. Inspired by Random Feature (Sato et al., 2021; Abboud et al., 2021) which generates random node features during training, we could use Random Sign (Dwivedi et al., 2020) and Random Basis.
Random Sign randomly ﬂips the signs of the eigenvectors during training to encourage the insensitivity to different sign choices. Similarly, Random Basis multiplies the eigenvectors U with a randomly sampled orthogonal matrix (Mezzadri, 2006) during each training period. UBoN enhanced with
Random Sign and Random Basis achieves sign and basis invariance in expectation. This is similar to Random Feature, which preserves permutation invariance in expectation (Abboud et al., 2021).
We could also address these problems from the perspective of model structures.
First, we consider sign invariance. We design a new sign-invariant extension of UBoN, named UBoN-sign, by combining both positive and negative variants of EASE.
We give its universality theorem as follows.
Theorem 3.
Let Ω ⊂ R n × d × R n × n be a compact set of graphs, [ X , ˆ A ] ∈ Ω .
Given a continuous invariant graph function f deﬁned over Ω , there exists a permutation-invariant and sign-invariant
BoN network with EASE that can approximate f to an arbitrary precision.
Then we consider the stronger basis-invariant property.
Suppose ˆ A has k unique eigenvalues λ 1 < λ 2 < · · · < λ k , and each λ i has multiplicity d i .
Let U
λ
i ∈ R n × d
i be the eigenvectors that form a orthonormal basis of the eigenspace corresponding to λ i .
We design another extension of
UBoN, named UBoN-basis, which also achieves basis invariance property by incorporating these different basis variants.
As the basis invariance is strictly stronger than sign invariance, UBoN-basis is also sign-invariant, and therefore, UBoN-basis can produce unique outputs for different eigendecomposition, while preserving the universal expressive power on graphs.
Theorem 4.
Let Ω ⊂ R n × d × R n × n be a compact set of graphs, [ X , ˆ A ] ∈ Ω .
Given a continuous graph function f deﬁned over Ω , there exists a permutation-invariant and basis-invariant BoN network with EASE that can approximate f to an arbitrary precision.
Theorem 3 and Theorem 4 indicate that by carefully designing the network structure, UBoN can be universal, permutation invariant, and sign/basis invariant at the same time.
Our designs of UBoN- sign and UBoN-basis follow the idea of SignNet and BasisNet in Lim et al.
More details and proofs can be found in Appendix B.3 and Appendix B.4.
E XPERIMENT
In this section, we evaluate the performance of UBoN on synthetic and real-world datasets.
The input node features are fed into a two-layer MLP network, followed by a permutation invariant readout function (e.g., sum , mean , max ) and a linear classiﬁer.
As mentioned in Section 4.2, we align the last dimension of input node features such that X (cid:
) ∈
R n × ( d + k ) .
We set k = 32 in all experiments.
Detailed hyperparameter settings are in Appendix C.
The experimental results show that UBoN achieves expressive power beyond the WL test and comparable performance with baselines.
5. E VALUATION ON SYNTHETIC DATASET
We conduct experiment on the E XP dataset proposed in Abboud et al.
(2021), which is designed to explicitly evaluate the expressive power of GNNs.
The dataset consists of a set of 1-WL indistinguishable non-isomorphic graph pairs, and each graph instance is a graph encoding of a propositional formula.
The classiﬁcation task is to determine whether the formula is satisﬁable (SAT).
Since the graph pairs in the E XP dataset are not distinguishable by 1-WL test, if a model shows above 50% accuracy on this dataset, it should have expressive power beyond the 1-WL algorithm.
The models we used on E XP dataset are as follows: an 8-layer GCN, GIN (Xu et al., 2019), PPGN (Maron et al., 2019a), 1-2-3-GCN-L (Morris et al., 2019), 3-GCN (Abboud et al., 2021), BoNRNI (BoN with Random Node Initialization (RNI)
(Abboud et al., 2021)), GCN-RNI (GCN with Random Node Initialization (RNI)), Linear-EASE (linear network with EASE), and UBoN. GCN and GIN belongs to the family of MP-GNNs whose expressive power is bounded by 1-WL, and RNI is a method to improve the expressive power of MP-GNNs.
We verify the expressive power gain of EASE-based models by comparing them with GCN, and evaluate their efﬁciency by comparing them with other expressive models.
We evaluate all models on the E XP dataset using 10-fold cross-validation, and train each model for 500 epochs per fold.
Mean test accuracy across all folds is measured and reported.
The results are reported in Table 1.
In addition, we also measure the learning curves of models to show their convergence rate, as shown in Figure 3.
[Table]
[Figure]
[Table] [Figure] In Table 1, we observe that vanilla GCN and BoN-RNI only achieves 50% accuracy, because they do not have expressive power beyond the 1-WL test.
UBoN achieves the best performance among all models with a near 100% accuracy, which demonstrates the expressive power of EASE-based models is beyond the 1-WL test.
Other models, namely Linear-EASE and GCN-RNI, also achieve comparable performance.
It’s worth mentioning that even a simple linear model (Linear-EASE) could achieve performance above 99%.
This indicates that the universal expressiveness of models with EASE is mostly from EASE itself, rather than the network structure.
From Figure 3, we ﬁnd that the convergence rate of EASE-based models are much faster than their
RNI-based counterpart.
This is because EASE-based models are deterministic, while RNI-based [Table] models are random and require more training epochs to converge.
The structures of UBoN and
Linear-EASE are simpler, which means they also train much faster than GCN-RNI.
5. E VALUATION ON REAL - WORLD DATASETS
We conduct graph classiﬁcation experiments on ﬁve real-world bioinformatics datasets: MUTAG,
PTC, COX2, PROTEINS, and ENZYMES (Yanardag & Vishwanathan, 2015).
We perform 10-fold cross-validation and report the average accuracy and standard deviations.
Our baselines include: (1)
MP-GNNs, including GCN (Welling & Kipf, 2017), GIN (Xu et al., 2019), FDGNN (Gallicchio & Micheli, 2020); (2) expressive models, including δ -2-LWL (Morris et al., 2020), 1-2-3-GNN (Morris et al., 2019), 3-hop GNN (Nikolentzos et al., 2020), Nested GIN (Zhang & Li, 2021); (3) graph MLPs, including Graph-MLP (Hu et al., 2021) and BoN. The results are listed in Table 2.
We also conduct experiments on larger scale datasets, and report the results in Appendix D.
As shown in Table 2, UBoN and its RS variant achieve overall comparable, and sometimes superior performance to previous MP-GNNs.
In particular, it outperforms the MP-GNN baselines on 4 out of 5 datasets.
On the PROTEINS dataset, although inferior to some models, UBoN is still superior to the expressive GNNs like δ -2 LWL, 1-2-3-GNN.
Notably, UBoN improves the vanilla BoN by a large margin, up to 38% on ENZYMES and 21% on MUTAG.
These results validate the strong expressive power and generalization capability of UBoN. Further comparing UBoN to its Random
Sign (RS) variant, we notice that RS can bring some additional beneﬁts in most cases, while the main improvement over vanilla BoN still comes from the proposed EASE in UBoN.
To further investigate the efﬁciency of UBoN, we train UBoN for 300 epochs on the PROTEINS dataset and summarize their training time.
The results are listed in Table 3.
We also count the total number of parameters of UBoN and some baselines, and report the results in Table
[Table]
[Table]
[Table]
As shown in Table 3 and Table [Table] UBoN is able to achieve comparable or better performance with signiﬁcantly less training time and number of parameters.
Removing the redundant message passing modules reduces the computational cost of UBoN [Table] but not its expressive power.
This exhibits the efﬁciency of UBoN [Table] and we believe it could motivate further researches on other alternatives to
MP-GNNs that can overcome the expressiveness bound of 1-WL test.
5. E MPIRICAL UNDERSTANDINGS OF UB O N
Effects of sign/basis invariance.
In Section 4.3, we discussed the sign/basis invariance issue of
BoN, and proved the universality of UBoN-sign and UBoN-basis.
Here we test the performances of UBoN-sign and UBoN-basis on ﬁve datasets and compare them with vanilla UBoN. As shown in Table 5, the two variants can bring some bonuses on certain datasets, while generally speaking, their performance is on the same level of UBoN and UBoN + RS.
This indicates that restricting the structure of UBoN to be explicitly sign/basis invariant does not bring signiﬁcant accuracy increase, while bringing additional computation overhead, particularly the UBoN-basis variant.
Therefore, we believe that the vanilla UBoN with random sign is enough for most real-world applications.
[Table]
Effects of number of eigenvectors.
In practice, we used only a part of all eigenvectors that corresponds to the largest k eigenvalues to generate EASE.
Here, we study the effects of k by changing k and keep other hyperparameters unchanged.
We observe in Table 6 that the performance of UBoN drops when k is too small or too large (except on ENZYMES).
This is because when k is too small, our model does not have access to enough structural information; and when k is too large, the high frequency eigenvectors that are unhelpful to classiﬁcation cause our model to overﬁt.
[Table]
Effects of eigenvalues in EASE.
In Section 4. [Table] we deﬁned two types of positional encoding methods, SE and EASE.
Here we test the performance of UBoN with SE and EASE to study the effects of using eigenvalues [Table] and report the results in Table 7.
We observe that removing eigenvalues in EASE lowers the performance of our model, showing that providing the model with information about eigenvalues is beneﬁcial for graph classiﬁcation.
[Table]
2 ± 5 .
C ONCLUSION
In this paper, we proposed eigenvalue-aware spectral embedding (EASE), a positional encoding method that improves the expressive power of graph learning models.
We prove that a simple MLPbased BoN network with EASE can achieve universal expressive power while preserving permutation invariance.
We name this model as UBoN. We also show that UBoN can further satisfy sign and basis invariance.
Extensive experiments show that UBoN achieves stronger expressive power and higher accuracy on most datasets with less training time and fewer parameters.
Our ﬁndings offer an alternative to the conventional MP-GNNs whose expressiveness is limited by the WL test.
A O THER POSITIONAL ENCODING METHODS
In this paper, we proposed EASE, which is a kind of positional encodings.
In the ﬁeld of graph representation learning, many other positional encoding methods have also been proposed.
Murphy et al.
(2019) proposed Relational Pooling (RP) that assigns each node with an identiﬁer that depends on the index ordering.
They showed that RP-GNN is strictly more expressive than the original WLGNN.
However, to ensure permutation equivariance, we have to account for all n !
node orderings, which is computationally expensive.
You et al.
(2019) proposed learnable position-aware embeddings by computing the distance of a target node to an anchor-set of nodes, and showed that P-GNNs have more expressive power than existing GNNs.
However, the expressive power of their model depends on the random selection of anchor sets.
Sato et al.
; Abboud et al.
(2021) proposed to use full or partial random node features and proved that their model has universal expressive power, but it has several defects: 1) the loss of permutation invariance, 2) slower convergence, and 3) poor generalization on unseen graphs (You et al., 2019; Loukas, 2020).
Li et al.
(2020) proposed Distance
Encoding (DE) that captures the distance between the node set whose representation is to be learned and each node in the graph.
They proved that DE can distinguish node sets embedded in almost all regular graphs where traditional GNNs always fail.
However, their approach fails on distance regular graphs, and computation of power matrices limits their model’s scalability.
In particular, we will show that with Random Node Initialization (RNI), (1) A linear network is universal on a ﬁxed graph, and (2) An MLP with just one additional message passing layer can be universal on arbitrary graphs.
In our work, we denote RNI as concatenating a random matrix to the input node features.
The random matrix can be sampled from Gaussian distribution, uniform distribution, etc.
Without loss of generality, we will assume that each entry of the random matrix is sampled independently from the standard Gaussian distribution N (0 , 1) .
Deﬁnition 7.
A GNN with RNI is deﬁned by concatenating a random matrix R to the input node features, i.e., X (cid:48) =
[ X , R ] , where X are the original node features, X (cid:48) are the modiﬁed node features and each entry of R is sampled independently from the standard Gaussian distribution N (0 , 1) .
The value of R is sampled at every forward pass of GNN.
To study the effects of RNI on the expressiveness of GNNs, we consider two types of tasks: tasks on a ﬁxed graph (e.g., node classiﬁcation) and tasks on arbitrary graphs (e.g. graph classiﬁcation).
On a ﬁxed graph, we aim to learn a function f : R n × d → R n × d (cid:48) that transforms the feature of each node v i to a presentation vector Z i, : ∈ R 1 × d (cid:48) .
We claim that a linear GNN with RNI in the form [ X , R ] W = Z (
[Equation]
is universal, where W ∈ R d × d (cid:48) are the network parameters, and Z ∈ R n × d (cid:48) is the desired output.
In other words, we have the universality theorem of linear GNNs with RNI on a ﬁxed graph:
Theorem 5.
On a ﬁxed graph G , a linear GNN with RNI deﬁned by Equation 2 is equivariant and can produce any prediction Z ∈ R n × d (cid:48) with probability 1 .
We prove Theorem 5 in Appendix B.5.
On arbitrary graphs, the target function is not only dependent on the node features, but on the graph structure ˆ A ∈ R n × n as well.
Let Ω ⊂ R n × d × R n × n be a compact set of graphs with [ X , ˆ A ] ∈ Ω , where X are the node features and ˆ A is the normalized adjacency matrix.
We wish to learn a function f : Ω → R that transforms each graph to its label.
Since f is also dependent on ˆ A , an MLP-based network with X (cid:48) as input is not expressive enough, and we need additional graph convolutional layers to obtain information about the graph structure.
However, Puny et al.
(2020) proved that with just one additional message passing layer, an MLP with RNI can approximate any continuous invariant graph function f .
Theorem 6 (Puny et al.
Given a compact set of graphs Ω ⊂ R n × d × R n × n , a GNN with one message passing layer, an MLP network and RNI can approximate an arbitrary continuous invariant graph function f : Ω → R to an arbitrary precision.
Theorem 6 is a direct inference of the proof of Proposition 1 in Puny et al.
(2020), where the authors constructed a RGNN that ﬁrst transfers the graph structural information to the node features through a message passing layer, and then approximates f with a DeepSets network, which is an MLP-based network.
B P ROOFS B. P ROOF OF T HEOREM
We ﬁrst prove that the EASE UΛ 12 is a real-valued matrix.
Lemma 1.
Suppose ˆ A is the normalized adjacency matrix of a graph G , and ˆ A =
UΛ U T is its spectral decomposition.
Then, UΛ 12 ∈ R n × n .
Proof.
Let Λ = diag( λ ) .
It sufﬁces to show that λ i ≥ 0 for i
= 1 , 2 , . .
. , n .
Let G =
( V , E ) , where V is the vertex set and E is the edge set.
For node i , we denote the degree of node i by d i .
For any x ∈ R n , we have
[Equation]
thus the Rayleigh quotient of ˆ A is bounded by x T ˆ Ax x T x ≥ 0 .
The Rayleigh quotient gives the lower bound of eigenvalues of ˆ A , therefore we have λ i ≥ 0 , and this completes the proof.
Our construction of universal MLPs on graphs involves transforming a graph invariant function into a set invariant function, thus we also give the deﬁnition of invariance on sets.
Let X
= { x 1 , x 2 , . . .
, x M } be a set, where x i ∈ X , and X =
X be the power set of X .
Consider a function f that transforms its input domain X to its range Y .
Since f acts on sets, we wish the output of f to be invariant to the ordering of elements in X .
Formally,
Deﬁnition 8.
A function acting on sets f : 2 X
→ Y is invariant if for any permutation π ,
[Equation]
Zaheer et al.
(2017) studied the property of invariant set functions and gave the following theorem.
Theorem 7 (Zaheer et al.
Assume the elements are from a compact set in R d , i.e. possibly uncountable, and the set size is ﬁxed to M .
Then any continuous function operating on a set X , i.e. f : R M × d → R which is permutation-invariant to the elements in X can be approximated arbitrarily close in the form of ρ (cid:0)(cid:80) x
∈ X φ ( x ) (cid:1) , for suitable transformations φ and ρ .
In practice, we can approximate ρ and φ with MLPs.
Using the universal approximation theorem of
MLPs (Hornik et al., 1989), we have the following corollary.
Corollary 1.
Any continuous invariant set function f ( X ) operating on a set X with ﬁxed set size having elements from a compact set in R d can be approximated by an MLP-based network.
Then we give the proof of Theorem 1.
Proof.
We can rewrite f such that it is a continuous set invariant function on the set consisting of the rows of its input:
[Equation]
Using the graph invariance property of f , we can verify that F is set invariant by observing:
[Equation]
By Lemma 1, the rows of [ X , UΛ 12 ] are in R d + n .
Using the conclusion from Corollary 1, we can approximate F arbitrarily close with a DeepSets network, and this completes the proof.
B. P ROOF OF T HEOREM
By Lemma 1, we know that λ i ≥ 0 for i
= 1 , 2 , . .
. , n .
Next we prove that λ i ≤ 2 .
Lemma 2.
Suppose ˆ A is the normalized adjacency matrix of a graph G , and λ 1 < · · · < λ n are its eigenvalues.
Then λ i ≤ 2 , for i = 1 , 2 , . .
. , n .
Proof.
In the proof of Lemma 1, we proved x T ( I + ˜ A ) x ≥ 0 .
Similarly, we have
[Equation]
Thus,
[Equation]
This shows that the Rayleigh quotient is bounded by x T ˆ Ax x T x ≤ 2 , therefore λ i ≤ 2 .
Then we give the proof of Theorem 2.
Proof.
Let 0 ≤ λ 1 ≤ · · · ≤ λ n ≤ 2 be the eigenvalues of ˆ A and u 1 , . .
u n be the corresponding eigenvectors.
Then
[Equation]
We also deﬁne
[Equation]
By Theorem 1 and the assumptions in our theorem, we know that there exists a permutation-invariant
BoN network such that
[Equation]
Since f can be approximated by an L p -Lipschitz continuous function
[Equation]
we have
[Equation]
Combining the two inequalities above gives us
[Equation]
B. P ROOF OF T HEOREM
Before proving Theorem 3, we ﬁrst give some basis topology and algebra deﬁnitions for our theoretical results.
Deﬁnition 9 (topological space and topological embedding) .
A topological space ( X , τ ) is deﬁned by a set X and a collection τ of subsets of X such that
The empty set and X belong to τ . 2.
Any arbitrary union of members of τ belongs to τ . 3.
The intersection of any ﬁnite number of members of τ belongs to τ .
We will omit τ and refer to a topological space as X .
A function f : X → Y is called a topological embedding if it is a homeomorphism from X to its image.
Deﬁnition 10 (group and topological group) .
A group is a set G along with a binary operation × on G such that
The binary operation satisﬁes associativity.
There exists an identity element e ∈ G such that e × a = a × e = a , for all a ∈ G . 3.
For each a ∈ G , there exists an inverse of a in G such that a × a − 1 = a − 1 × a = e .
A topological group is a topological space that is also a group such that the group operation and the inverse map are continuous.
Deﬁnition 11 (quotient space and quotient map) .
A group G may act on a set X by a function such that the output is also an element of X , which we denote as gx .
We deﬁne Gx = { gx : x ∈ X} as an equivalence class of X under G .
The quotient space is deﬁned by X /G
=
{ Gx : x ∈ X} , which is the set of all such equivalence classes.
The quotient map is a function π : X → X /G that maps elements in X to their equivalence classes.
Next we give the Decomposition Theorem (Lim et al., 2022), which is useful for representing and approximating invariant functions.
Theorem 8 (Decomposition Theorem) .
Let X 1 , . . .
, X k be topological spaces, and let G i be a topological group acting continuously on X i for each i .
Assume that there is a topological embedding ψ : X i /G i
→ R a i of each quotient space into a Euclidean space R a i for some dimension a i .
Then, for any continuous function f : X = X 1 × · · · × X k
→ R s that is invariant to the action of
G = G 1 × · · · × G k , there exists continuous functions
φ i : X i → R a i and a continuous function ρ : Z ⊆ R a → R s , where a = (cid:80) i a i such that
[Equation]
Furthermore: (1) each φ i can be taken to be invariant to G i , (2) the domain Z is compact if each
X i is compact, (3) if X i =
X j
and G i =
G j , then φ i can be taken to be equal to φ j .
See Lim et al.
for the proof of Theorem 8.
Using the Decomposition Theorem, we have the following lemma.
Lemma 3.
Suppose X ∈ R n × d , UΛ 12 ∈ R n × n .
A continuous function f is sign invariant w.r.t.
rows of UΛ 12 , i.e.,
[Equation]
for all s i ∈ { 1 , − 1 } , iff there exists continuous ρ and φ such that
[Equation]
X i, :
[Equation]
− ( UΛ 12 ) i, : (cid:1) | i ∈ [ n ] (cid:111)(cid:17) .
Proof.
The continuous function
[Equation]
acts on ( R d × R n ) n and is invariant to the action of ( { 1 } × { 1 , − 1 } ) n .
Since
R d / { 1 } ∼ = R d ,
[Equation] [Equation]
, −
[Equation]
} ∼ =
[Equation]
n , by Theorem 8, there exists continuous ρ and φ (cid:48) such that
[Equation]
Furthermore, we can take φ (cid:48) to be invariant to the action of { 1 } × { 1 , − 1 } , i.e. ,
[Equation]
Letting φ (cid:0) X i, : , ( UΛ 12 ) i, : (cid:1)
=
φ (cid:48) (cid:0) X i, : , ( UΛ 12 ) i, : (cid:1) / 2 , we are done.
In Lemma 3, f being permutation invariant indicates that ρ is also permutation invariant.
By Corollary 1 and the universal approximation theorem of MLPs (Hornik et al., 1989), we can approximate ρ with a DeepSets network and φ with an MLP.
This gives the proof of Theorem 3.
B. P ROOF OF T HEOREM
Before proving Theorem 4, we ﬁrst give the deﬁnitions of the Grassmannian and Stiefel manifold.
Deﬁnition 12 (Grassmannian) .
The Grassmannian, Gr( d, n ) , is a space that parameterizes all ddimensional linear subspaces of the n -dimensional vector space R n .
The Grassmannian Gr( d, n ) is a smooth manifold of dimension d ( n − d ) .
Deﬁnition 13 (Stiefel manifold) .
The Stiefel manifold St( d, n ) is the set of all orthonormal tuples [ v 1 · · · v d ] ∈ R n × d of d vectors in R n .
Then we give the First Fundamental Theorem of O ( d ) (Villar et al., 2021) and the Whitney Embedding Theorem (Whitney, 1944).
Theorem 9 (First Fundamental Theorem of O ( d ) ) .
A continuous function f : R n × d → R s is orthogonally invariant, i.e. f ( V Q ) =
f ( V ) for all Q ∈ O ( d ) , iff f ( V )
=
h ( V V T ) for some continuous h .
Theorem 10 (Whitney Embedding Theorem) .
Every smooth manifold M of dimension n > 0 can be smoothly embedded in R 2 n .
See Villar et al.
(2021) for the proof of Theorem 9, Whitney (1944) for the proof of Theorem 10.
Using the Decomposition Theorem, we have the following lemma.
Lemma 4.
Suppose X ∈ R n × d , UΛ 12 ∈ R n × n , where ˆ A =
UΛ U T is the spectral decomposition of ˆ A .
A continuous function f is basis invariant w.r.t.
eigenspaces, i.e.,
[Equation]
for all orthogonal Q i ∈ O
( d i ) ⊆
R d i × d i , iff there exists continuous ρ, ψ, φ d i such that
[Equation]
Proof.
The continuous function
[Equation]
acts on R n × d × (St( d i , n ) × R ) k and is invariant to the action of { 1 } × ( O ( d i ) × { 1 } ) n .
Since
[Equation]
and by Theorem 10, there is a topological embedding ψ d i : Gr( d i , n ) → R 2 d i ( n − d i ) , by Theorem 8, there exists continuous ρ, ψ, φ (cid:48)
d i such that
[Equation]
Furthermore, we can take φ (cid:48)
d i to be invariant to the action of O ( d i ) × { 1 } , i.e. ,
[Equation]
for all Q ∈ O ( d i ) .
By Theorem 9, there exists a continuous φ d i such that
[Equation]
This completes the proof.
In Lemma 4, if f is permutation invariant, i.e. , for all permutation matrices P ∈ R n × n , f ( X , { U λ i , λ i | i ∈
[ k ] } ) =
f ( P X , { P U
λ i , λ
i | i ∈
[ k ] } ) , by Theorem 8, ψ and φ (cid:48)
d i can be taken to be permutation invariant, thus φ d i is graph invariant.
Since ψ can be (cid:15) -approximated by an invariant set network and φ d i can be (cid:15) -approximated by an invariant graph network (Lim et al., 2022), Theorem 4 is proved.
B. P ROOF OF T HEOREM
We ﬁrst prove the following lemmas.
Lemma 5.
Let R ∈ R n × n be a random matrix, and each entry of R is sampled independently from the standard Gaussian distribution N (0 , 1) .
Then with probability 1 , R has full rank.
Proof.
We denote the ﬁrst column of R by R : , 1 .
It is linearly independent because R : , 1 = 0 with probability 1 .
Then we view R : , 1 as ﬁxed, and consider the second column R : , 2 .
The probability that
R : , 2 falls into the span of R : , 1 is 0 , thus with probability 1 , R : , 1 and R : , 2 are linearly independent.
Generally, let us consider the k -th column R : ,k .
The ﬁrst k − 1 columns of R forms a subspace in R n whose Lebesgue measure is 0 .
Thus R : ,k falls into this subspace with probability 0 .
By inference, we have all the columns of R are linearly independent with probability 1 , i.e. , P (rank( R ) = n ) = 1 .
Lemma 6.
Let A ∈ R s × n , B ∈ R s × m be two matrices.
Then the equation AX =
B has a solution iff.
rank( A ) = rank([ A , B ]) .
Proof.
First we prove the necessity.
Suppose AX =
B has a solution.
Then,
[Equation]
where M : ,i denotes the i -th column of matrix M .
This means each column of B can be expressed as a linear combination of the columns of A , and therefore each column of [ A , B ] can be expressed as a linear combination of the columns of A .
On the other hand, it is obvious that each column of A can be expressed as a linear combination of the columns of [ A , B ] .
Thus we have rank( A ) =
rank
[ A , B ]) .
Then we prove the sufﬁciency.
Since rank( A ) = rank
[ A , B ]) , and each column of A can be expressed as a linear combination of columns of [ A , B ] , we have the columns of A and the columns of [ A , B ] are equivalent.
Therefore, each column of B can be expressed as a linear combination of columns of A , i.e. , Ax = B : ,i has a solution for i = 1 , . . .
, m .
Thus the equation AX =
B has a solution.
Then we give the proof of Theorem 5.
Proof.
For any prediction Z ∈ R n × d (cid:48) , we wish to prove that with probability 1 , there exists parameters of a linear GNN with RNI W ∈ R d × d (cid:48)
such that
[Equation]
By Lemma 6, the necessary and sufﬁcient condition that Equation 3 has a solution W is rank([ X , R ])
= rank([ X , R , Z ]) .
By Lemma 5, with probability 1 , rank( R ) = n , therefore rank([ X , R ])
= rank([ X , R , Z ])
= n .
Thus, in conclusion, with probability 1 , there exists parameters of a linear GNN with RNI W ∈
R d × d (cid:48) such that the GNN produces Z .
We can also prove linear GNNs’ equivariance by observing that for any permutation matrix P ∈ R n × n ,
[Equation]
P
[Equation]
P
[Equation]
, where we used P R = R because each entry of P R is also sampled from the standard Gaussian matrix N (0 , 1) .
C H YPERPARAMETER SETTINGS
We use Optuna (Akiba et al., 2019) to optimize the hyperparameters of our models.
In experiments on synthetic dataset, the hyperparameters of our models are as follows:
• GCN-SE : the learning rate λ = 0 . 000135737656276214 , the hidden dimension w = 87 , the number of GCN layers l = 9 , whether to normalize the GCN layers normLayers =
False , the weight decay of Adam optimizer (Kingma & Ba, 2015) wd = 0 .
• UBoN : the learning rate λ = 0 .
002385602941230316 , the hidden dimension of the ﬁrst linear layer w 1 = 60 , the hidden dimension of the second linear layer w 2 = 76 , the dropout rate (Srivastava et al., 2014) p = 0 .
13592575703525184 , the weight decay of Adam optimizer wd = 0 . 0005 .
• Linear-SE : the learning rate λ = 0 .
0006867736568978745 , the hidden dimension w = 109 , the weight decay of Adam optimizer wd = 0 .
In experiments on real-world datasets, we tune the following hyperparameters: the learning rate λ , the hidden dimension of the ﬁrst linear layer w the hidden dimension of the second linear layer w 2 , whether to use layer norm (Ba et al., 2016) normLayers , whether to normalize the adjacency matrix normAdj , the dropout rate p and the weight decay of Adam optimizer wd .
The values are listed in Table 8 and Table 9.
[Table] [Table]
We use ELU non-linearities (Clevert et al., 2016) in all experiments.
We set the batch size to beon PROTEINS and 20 on all other datasets.
D E XPERIMENT RESULTS ON LARGE SCALE DATASETS
We conduct experiments on the following three large scale datasets: DD (Yanardag & Vishwanathan, 2015), ogbg-molhiv and ogbg-moltoxcast (Hu et al., 2020).
On the DD dataset, we use the same settings as in Section 5.2, i.e. , perform 10-fold cross-validation and report the average accuracy.
On the OGB datasets, we use the settings provided by the OGB team, i.e. , split the datasets into train, validation and test set, and report the test accuracy at the best validation epoch. We compare the performance of UBoN with GIN (Xu et al., 2019), which is a widely used MP-GNN, and also with vanilla BoN to show the effects of UBoN. The results are listed in Table 10. As shown in Table 10, UBoN greatly improves the performance of BoN, demonstrating the efﬁcacy of applying EASE, while also achieving comparable performance with its MP-GNN counterpart. On the other hand, the number of parameters of UBoN is signiﬁcantly less than GIN, showing that our UBoN is indeed a promising efﬁcient alternative to message passing GNNs.
We also report the dataset details and the pre-processing and training time (300 epochs) of UBoN in
Table 11.
[Table]
[Table]
In Table
[Table]
it can be seen that even on datasets with larger number of graphs or nodes
[Table]
the preprocessing time is relatively less than training time, showing that the computation cost of eigendecomposition is acceptable.
On the other hand, the main training cost of our UBoN is simply an
MLP network, which is very fast to train without any message passing steps.
This demonstrates the efﬁciency and scalability of UBoN.