A S IMPLE AND G ENERAL G RAPH N EURAL N ETWORK WITH S TOCHASTIC M ESSAGE P ASSING
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph neural networks (GNNs) are emerging machine learning models on graphs.
One key property behind the expressiveness of existing GNNs is that the learned node representations are permutation-equivariant.
Though being a desirable prop- erty for certain tasks, however, permutation-equivariance prevents GNNs from being proximity-aware, i e., preserving the walk-based proximities between pairs of nodes, which is another critical property for graph analytical tasks.
On the other hand, some variants of GNNs are proposed to preserve node prox- imities but they fail to maintain permutation-equivariance.
How to empower
GNNs to be proximity-aware while maintaining permutation-equivariance re- mains an open problem.
In this paper, we propose Stochastic Message Passing
(SMP), a general and simple GNN to maintain both proximity-awareness and permutation-equivariance properties.
Speciﬁcally we augment the existing GNNs with stochastic node representations learned to preserve node proximities.
Though seemingly simple, we prove that such a mechanism can enable GNNs to preserve node proximities in theory while maintaining permutation-equivariance with cer- tain parametrization.
Extensive experimental results demonstrate the effectiveness and efﬁciency of SMP for tasks including node classiﬁcation and link prediction
I NTRODUCTION
Graph neural networks (GNNs), as generalizations of neural networks in analyzing graphs, haveattracted considerable research attention.
GNNs have been widely applied to various applicationssuch as social recommendation (Ma et al., 2019), physical simulation (Kipf et al., 2018), and proteininteraction prediction (Zitnik & Leskovec, 2017).
One key property of most existing GNNs is permutation-equivariance, i.e., if we randomly permutate the structure, the
GNNsare permutated accordingly.
Mathematically, permutation-equivariance reﬂects one basic symmetric group of graph structures.
Although it is a desirable property for tasks such as node or graphclassiﬁcation (Keriven & Peyr´e, 2019; Maron et al., 2019b), permutation-equivariance also prevents
GNNs from being proximity-aware, i.e., permutation-equivariant GNNs cannot preserve walk-basedproximities between nodes such as the shortest distance or high-order proximities (see Theorem 1).
Pairwise proximities between nodes are crucial for graph analytical tasks such as link prediction (Hu et al., 2020;
You et al., 2019).
To enable a proximity-aware GNN, Position-aware GNN(P-GNN) (You et al., 2019) 1 proposes a sophisticated GNN architecture and shows better performance for proximity-aware tasks.
But P-GNN needs to explicitly calculate the shortest distance between nodes and its computational complexity is unaffordable for large graphs.
Moreover, P-GNNcompletely ignores the permutation-equivariance property.
Therefore, it cannot produce satisfactoryresults when permutation-equivariance is helpful.
In real-world scenarios, both proximity-awareness and permutation-equivariance are indispensableproperties for GNNs.
Firstly, different tasks may require different properties.
For example, recommendation applications usually require the model to be proximity-aware (Konstas et al., 2009) whilepermutation-equivariance is a basic assumption in centrality measurements (Borgatti, 2005).
Even for the same task, different datasets may have different requirements on these two properties.
Takinglink prediction as an example, we observe that permutation-equivariant GNNs such as GCN (Kipf &
Welling, 2017) or GAT (Velickovic et al., 2018) show better results than P-GNN in coauthor graphs,but the opposite in biological graphs (please see Section 5.2 for details).
Unfortunately, in the current
GNN frameworks, these two properties are contradicting, as we show in Theorem 1.
Whether thereexists a general GNN to be proximity-aware while maintaining permutation-equivariance remainsan open problem.
In this paper, we propose Stochastic Message Passing (SMP), a general and simple GNN to preserve both proximity-awareness and permutation-equivariance properties.
Speciﬁcally, we augmentthe existing GNNs with stochastic node representations learned to preserve proximities.
Thoughseemingly simple, we prove that our proposed SMP can enable GNNs to preserve walk-based proximities in theory (see Theorem 2 and Theorem 3).
Meanwhile, SMP is equivalent to a permutationequivariant GNN with certain parametrization and thus is at least as powerful as those GNNs inpermutation-equivariant tasks (see Remark 1).
Therefore, SMP is general and ﬂexible in handlingboth proximity-aware and permutation-equivariant tasks, which is also demonstrated by our extensive experimental results.
Besides, owing to the simple structure, SMP is computationally efﬁcient,with a running time roughly the same as those of the most simple GNNs such as SGC (Wu et al.
and is at least an order of magnitude faster than P-GNN on large graphs.
Ablation studiesfurther show that a linear instantiation of SMP is expressive enough as adding extra non-linearitiesdoes not lift the performance of SMP on the majority of datasets.
Our contributions are as follows.
• We propose SMP, a simple and general GNN to handle both proximity-aware and permutationequivariant graph analytical tasks.•
We prove that SMP has theoretical guarantees in preserving walk-based proximities and is atleast as powerful as the existing GNNs in permutation-equivariant tasks.• Extensive experimental results demonstrate the effectiveness and efﬁciency of SMP.
We showthat a linear instantiation of SMP is expressive enough on the majority of datasets.
R ELATED W ORK
We brieﬂy review GNNs and their permutation-equivariance and proximity-awareness property.
The earliest GNNs adopt a recursive deﬁnition of node states (Scarselli et al., 2008; Gori et al.,2005) or a contextual realization (Micheli, 2009).
GGS-NNs (Li et al., 2016) replace the recursivedeﬁnition with recurrent neural networks (RNNs).
Spectral GCNs (Bruna et al., 2014) deﬁned graphconvolutions using graph signal processing (Shuman et al., 2013; Ortega et al., 2018) with ChebNet (Defferrard et al., 2016) and GCN (Kipf & Welling, 2017) approximating the spectral ﬁlters using a K -order Chebyshev polynomial and the ﬁrst-order polynomial, respectively.
MPNNs (Gilmeret al., 2017), GraphSAGE (Hamilton et al., 2017), and MoNet (Monti et al., 2017) are proposedas general frameworks by characterizing GNNs with a message-passing function and an updatingfunction.
More advanced variants such as GAT (Velickovic et al., 2018), JK-Nets (Xu et al., 2018b),
GIN (Xu et al., 2018a), and GraphNets (Battaglia et al., 2018) follow these frameworks.
Li et al.
(Li et al., 2018), Xu et al.
(Xu et al., 2018a), Morris et al.
(Morris et al., 2019), and
Maron et al.
(Maron et al., 2019a) show the connection between GNNs and the Weisfeiler-Lehmanalgorithm (Shervashidze et al., 2011) of graph isomorphism tests, in which permutation-equivarianceholds a key constraint.
Maron et al.
(Maron et al., 2019b) and Keriven et al.
(Keriven & Peyr´e, 2019)analyze the permutation-equivariance property of GNNs more theoretically.
To date, most of theexisting GNNs are permutation-equivariant and thus are not proximity-aware.
The only exceptionis P-GNN (You et al., 2019), which proposes to capture the positions of nodes using the relativedistance between the target node and some randomly chosen anchor nodes.
However, P-GNN cannotsatisfy permutation-equivariance and is computationally expensive.
Very recently, motivated by enhancing the expressive power of GNNs in graph isomorphism testsand distributed computing literature (Angluin, 1980; Linial, 1992; Naor & Stockmeyer, 1995),some studies suggest assigning unique node identiﬁers for GNNs (Loukas, 2020) such as one-hot
IDs (Murphy et al., 2019) or random numbers (Dasoulas et al., 2019;
Sato et al., 2020; Corso et al.,2020).
For example, Sato et al.
(Sato et al., 2020) novelly show that random numbers can enhance
GNNs in tackling two important graph-based NP problems with a theoretical guarantee, namely the minimum dominating set and the maximum matching problem,
2020) empirically show the effectiveness of random features in the graph matching problem.
Our work differsin that we systematically study how to preserve permutation-equivariance and proximity-awarenesssimultaneously in a simple yet effective framework, which is a new topic different from these existing works.
Besides, we theoretically prove that our proposed method can preserve walk-basedproximities by using the random projection literature. We also demonstrate the effectiveness of ourmethod on various large-scale benchmarks for both node- and edge-level tasks, while no similarresults are reported in the literature.
The design of our method is also inspired by the random projection literature in dimensionality reduction (Vempala, 2005) and to the best of our knowledge, we are the ﬁrst to study random projectionin the scope of GNNs.
More remotely, our deﬁnition of node proximities is inspired and inheritedfrom graph kernels (G¨artner et al., 2003; Borgwardt & Kriegel, 2005), network embedding (Perozziet al., 2014; Grover & Leskovec, 2016), and general studies of graphs (Newman, 2018).
M ESSAGE - PASSING GNN S
We consider a graph G =
( V , E , F ) where V = { v 1 , ..., v N } is the set of N = |V| nodes, E ⊆ V ×Vis the set of M =
|E| edges, and F ∈ R N × d 0 is a matrix of d 0 node features.
The adjacency matrixis denoted as A , where its i th row, j th column and an element denoted as A i, : , A : ,j , and A i,j ,respectively.
In this paper, we assume the graph is unweighted and undirected.
The neighborhoodof node v i is denoted as N i and ˜ N i =
N i ∪ { v i } .
The existing GNNs usually follow a message-passing framework (Gilmer et al., 2017), where the l thlayer adopts a neighborhood aggregation function AGG ( · ) and an updating function UPDATE ( · ) :
[Equation]
where h ( l ) i ∈
R d l is the representation of node v i in the l th layer, d l is the dimensionality, and m ( l ) iare the messages.
We also denote H ( l ) =
[ h ( l ) 1 , ..., h ( l ) N ] and [ · , · ] is the concatenation operation.
Thenode representations are initialized as node features, i.e., H (0) =
F .
We denote a GNN following
Eq.
(1) with L layers as a parameterized function as follows 2 :
[Equation]
where H ( L ) are ﬁnal node representations learned by the GNN and W denotes all the parameters.
One key property of the existing GNNs is permutation-equivariance.
Deﬁnition 1 (Permutation-equivariance) .
Consider a graph G =
( V , E , F ) and any permutation
P : V → V so that G (cid:48) =
( V , E (cid:48) , F (cid:48) ) has an adjacency matrix A (cid:48)
=
PAP T and a feature matrix
F (cid:48)
= PF , where P ∈ { 0 , 1 } N × N is the permutation matrix corresponding to P , i.e., P i,j = 1 iff
P ( v i ) = v j .
A GNN satisﬁes permutation-equivariance if the node representations are equivariantwith respect to P , i.e.,
[Equation]
It is known that GNNs following Eq.
(1) are permutation-equivariant (Maron et al., 2019b).
Deﬁnition 2 (Automorphism) .
A graph G is said to have (non-trivial) automorphism if there existsa non-identity permutation matrix P (cid:54) =
I N so that
A =
PAP T and F = PF .
We denote thecorresponding automorphic node pairs as C G =
(cid:83) P (cid:54) =
I N { ( i, j ) | P i,j (cid:54) = 0 , i (cid:54) = j }
Corollary 1.
Using Deﬁnition 1 and 2, if a graph has automorphism, a permutation-equivariant
GNN will produce identical node representations for automorphic node pairs:
[Equation]
Since the node representations are used for downstream tasks, the corollary shows that permutationequivariant GNNs cannot differentiate automorphic node pairs.
A direct consequence of Corollary 1 is that permutation-equivariant GNNs cannot preserve walk-based proximities between pairsof nodes.
The formal deﬁnitions are as follows
Deﬁnition 3 (Walk-based Proximities) .
For a given graph G =
( V , E , F ) , we use a matrix S ∈ 136 R N × N to denote walk-based proximities between pairs of nodes deﬁned as:
[Equation]
5) where v i (cid:32) v j denotes walks from node v i to v j and S ( · ) is an arbitrary real-valued function.
Thelength of a walk-based proximity is the maximum length of all the walks of the proximity.
Typical examples of walk-based proximities include the shortest distance (You et al., 2019), the highorder proximities (a sum of walks weighted by their lengths) (Zhang et al., 2018), and random walkprobabilities (Klicpera et al., 2019). Next, we give a deﬁnition of preserving walk-based proximities.
Deﬁnition 4.
For a given walk-based proximity, a GNN is said to be able to preserve the proximityif, for any graph G
=
( V , E , F ) , there exist parameters W G and a decoder function F de ( · ) so that∀ (cid:15) > 0 :
[Equation] ) where [Equation] Note that we do not constrain the GNN architecture as long as it follows the message-passing framework in Eq.
(1), and the decoder function is also arbitrary.
In fact, both the GNN and the decoderfunction can be arbitrarily deep and with sufﬁcient hidden units.
Theorem 1.
The existing permutation-equivariant GNNs cannot preserve any walk-based proximityexcept the trivial solution that all node pairs have the same proximity.
The formulation and proof of the theorem are given in Appendix A.1.
Since walk-based proximitiesare rather general and widely adopted in graph analytical tasks such as link prediction, the theoremshows that the existing permutation-equivariant GNNs cannot handle these tasks well.
T HE M ODEL
A GNN F RAMEWORK USING S TOCHASTIC M ESSAGE P ASSING
A major shortcoming of permutation-equivariant GNNs is that they cannot differentiate automorphicnode pairs.
To solve that problem, we need to introduce some mechanism as “symmetry breaking”,i.e.
, to enable GNNs to distinguish these nodes.
To achieve this goal, we sample a stochastic matrix
E ∈ R N × d where each element follows an i.i.d.
normal distribution N (0 , 1) .
The stochastic matrixcan provide signals in distinguishing the nodes because they are randomly sampled without beingaffected by the graph automorphism.
In fact, we can easily calculate that the Euclidean distancebetween two stochastic signals divided by a constant √ 2 follows a chi distribution
χ d :
[Equation]
When d is reasonably large, e.g., d > 20 , the probability of two signals being close is very low.
Then, inspired by the message-passing framework, we apply a GNN on the stochastic matrix so thatnodes can exchange information of the stochastic signals:
[Equation]
We call ˜ E the stochastic representation of nodes.
Using the stochastic matrix and message-passing,˜ E can be used to preserve node proximities (see Theorem 2 and Theorem 3).
Then, to let our modelstill be able to utilize node features, we concatenate ˜ E with the node representations from another
GNN with node features as inputs:
[Equation]
where F output ( · ) is an aggregation function such as a linear function or simply the identity mapping.
In a nutshell, our proposed method augments the existing GNNs with a stochastic representationlearned by message-passings to differentiate different nodes and preserve node proximities.
There is also a delicate choice worthy mentioning, i.e., whether the stochastic matrix E is ﬁxed orresampled in each epoch.
By ﬁxing E , the model can learn to memorize the stochastic representationand distinguish different nodes, but with the cost of unable to handle nodes not seen during training.
On the other hand, by resampling E in each epoch, the model can have a better generalizationability since the model cannot simply remember one speciﬁc stochastic matrix.
However, the noderepresentations are not ﬁxed (but pairwise proximities are preserved; see Theorem 2).
In these cases,˜ E is more capable of handling pairwise tasks such as link prediction or pairwise node classiﬁcation.
In this paper, we use a ﬁxed E for transductive datasets and resample E for inductive datasets.
Time Complexity From Eq.(10), the time complexity of our framework mainly depends on thetwo GNNs in learning the stochastic and permutation-equivariant node representations.
In this paper,we instantiate these two GNNs using simple message-passing GNNs such as GCN (Kipf & Welling,2017) and SGC (Wu et al., 2019)
(see Section 4.2 and Section 4.3).
Thus, the time complexity ofour method is the same as these models, which is O ( M ) , i.e., linear with respect to the number ofedges.
We also empirically compare the running time of different models in Appendix 5.5.
Besides,many acceleration schemes for GNNs such as sampling (Chen et al., 2018a;b; Huang et al., 2018)or partitioning the graph (Chiang et al., 2019) can be directly applied to our framework.
A L INEAR I NSTANTIATION
Based on the general framework shown in Eq.
(10), we attempt to explore its minimum modelinstantiation, i.e., a linear model.
Speciﬁcally, inspired by Simpliﬁed Graph Convolution (SGC) (Wuet al., 2019), we adopt a linear message-passing for both GNNs, i.e.,
[Equation]
where
˜ A =
( D + I ) − 12 ( A + I )
( D + I ) − 12 is the normalized graph adjacency matrix with self-loopsproposed in GCN (Kipf & Welling, 2017) and K is the number of propagation steps.
We also set
F output ( · ) in Eq.
(11) as a linear mapping or identity mapping.
Though seemingly simple, we show that such an SMP instantiation possesses a theoretical guaranteein preserving the walk-based proximities.
Theorem 2.
An SMP in Eq.
(11) with the message-passing matrix ˜ A and the number of propagationsteps K can preserve the walk-based proximity ˜
A K ( ˜ A K ) T with high probability if the dimensionality of the stochastic matrix d is sufﬁciently large, where the superscript T denotes matrix transpose.
The theorem is regardless of whether E are ﬁxed or resampled.
The mathematical formulation and proof of the theorem are given in Appendix A.2.
In addition, weshow that SMP is equivalent to a permutation-equivariant GNN with certain parametrization.
Remark 1.
Suppose we adopt F output ( · ) as a linear function with the output dimensionality thesame as F GNN (cid:48) .
Then, Eq.
(10) is equivalent to the permutation-equivariant F GNN (cid:48)
( A , F ; W (cid:48) )
if theparameters in F output ( · ) are all-zeros for ˜ E and an identity matrix for H ( L ) .
The result is straightforward from the deﬁnition.
Then, we have the following corollary.
Corollary 2.
For any task, Eq.
(10) with the aforementioned linear F output ( · ) is at least as powerfulas the permutation-equivariant F GNN (cid:48)
( A , F ; W (cid:48) ) , i.e., the minimum training loss of using H in Eq.
(10) is equal to or smaller than using H ( L ) =
F GNN (cid:48)
( A , F ; W (cid:48) ) .
In other words, SMP will not hinder the performance 4 even the tasks are permutation-equivariantsince the stochastic representations are concatenated with the permutation-equivariant GNNs followed by a linear mapping.
In these cases, the linear SMP is equivalent to SGC (Wu et al., 2019).
Combining Theorem 2 and Corollary 2, the linear SMP instantiation in Eq.
(11) is capable of handling both proximity-aware and permutation-equivariant tasks.
4. N ON - LINEAR E XTENSIONS
One may question whether a more sophisticated variant of Eq.
(10) can further improve the expressiveness of SMP.
There are three adjustable components in Eq.
(10): two GNNs in propagating thestochastic matrix and node features, respectively, and an output function.
In theory, adopting nonlinear models as either component is able to enhance the expressiveness of SMP.
Indeed, if we usea sufﬁciently expressive GNN in learning ˜ E instead of linear propagations, we can prove a moregeneral version of Theorem 2 as follows.
Theorem 3.
An SMP variant following Eq.
(10) with F GNN ( A , E ; W ) containing L layers canpreserve any length-
L walk-based proximity if the message-passing and updating functions in the GNN are sufﬁciently expressive.
The proof of the theorem is given in Appendix A.3.
Similarly, we can adopt more advanced methodsfor F output ( · ) such as gating or attention so that the two GNNs are more properly integrated.
Although non-linear extensions of SMP can, in theory, increase the model expressiveness, they alsotake a higher risk of over-ﬁtting due to model complexity, not to mention that the computational costwill also increase.
In practice, we ﬁnd in ablation studies that the linear SMP instantiation in Eq.
(11)works reasonably well on most of the datasets (please refer to Section 5.4 for further details).
E XPERIMENTS5. E XPERIMENTAL S ETUPS
Datasets We conduct experiments on the following ten datasets: two simulation datasets, Gridand Communities (You et al., 2019), a communication dataset Email (You et al., 2019), two coauthor networks, CS and Physics (Shchur et al., 2018), two protein interaction networks, PPI (Hamilton et al., 2017) and
PPA (Hu et al., 2020), and three GNN benchmarks, Cora , CiteSeer , and
PubMed (Yang et al., 2016).
We only report the results of three benchmarks for the node classiﬁcation task and the results for other tasks are shown in Appendix B due to the page limit.
Moredetails of the datasets including their statistics are provided in Appendix C.1.
These datasets covera wide spectrum of domains, sizes, and with or without node features.
Since Email and PPI containmore than one graph, we conduct experiments in an inductive setting on these two datasets, i.e., thetraining, validation, and testing set are split with respect to different graphs.
Baselines We adopt two sets of baselines.
The ﬁrst set is permutation-equivariant GNNs including
GCN (Kipf & Welling, 2017), GAT (Velickovic et al., 2018), and SGC (Wu et al., 2019), which arewidely adopted GNN architectures.
The second set contains P-GNN (You et al., 2019), the onlyproximity-aware GNN to date.
We use the P-GNN-F version.
In comparing with the baselines, we mainly evaluate two variants of SMP with different F output ( · ) :
SMP-Identity, i.e., F output ( · ) as an identity mapping, and SMP-Linear, i.e., F output ( · ) as a linearmapping.
Note that both variants adopt linear message-passing functions as SGC.
We conduct moreablation studies with different SMP variants in Section 5.4.
For fair comparisons, we adopt the same architecture and hyper-parameters for all the methods(please refer to Appendix C.2 for the details).
For datasets without node features, we adopt a constant vector as the node features.
We experiment on two tasks: link prediction and node classiﬁcation.
Additional experiments on graph reconstruction, pairwise node classiﬁcation, and running timecomparison are provided in Appendix B.
We repeat the experiments 10 times for datasets except for
PPA and 3 times for PPA, and report the average results.
5. L INK P REDICTION
Link prediction aims to predict missing links of a graph.
Speciﬁcally, we split the edges into 80%-10%-10% and use them for training, validation, and testing, respectively.
Besides adopting those realedges as positive samples, we obtain negative samples by randomly sampling an equal number ofnode pairs that do not have edges.
For all the methods, we set a simple classiﬁer:
Sigmoid( H Ti H j ) ,i.e., use the inner product to predict whether a node pair ( v i , v j ) forms a link, and use AUC (areaunder the curve) as the evaluation metric.
One exception to the aforementioned setting is that on the
[Table] PPA dataset, we follow the splits and evaluation metric (i [Table] e., Hits@100) provided by the dataset (Huet al., 2020).
The results except PPA are shown in Table 2.
We make the following observations.
• Our proposed SMP achieves the best results on ﬁve out of the six datasets and is highly competitive (the second-best result) on the other (Physics).
The results demonstrate the effectiveness ofour proposed method on link prediction tasks.
We attribute the strong performance of SMP to itscapability of maintaining both proximity-awareness and permutation-equivariance properties.•
On Grid, Communities, Email, and PPI, both SMP and P-GNN outperform the permutationequivariant GNNs, proving the importance of preserving node proximities.
Although SMP issimpler and more computationally efﬁcient than P-GNN, SMP reports even better results.• When node features are available (CS, Physics, and PPI), SGC can outperform GCN and GAT.The results re-validate the experiments in SGC (Wu et al
that the non-linearity in GNNs isnot necessarily indispensable.
Some plausible reasons include that the additional model complex-
ity brought by non-linear operators makes the models tend to overﬁt and also difﬁcult to train (seeAppendix B.6).
On those datasets, SMP retains comparable performance on two coauthor graphsand shows better performance on PPI, possibly because node features on protein graphs are lessinformative than node features on coauthor graphs for predicting links, and thus preserving graph
[Table]
structure is more beneﬁcial on PPI.• As Email and PPI are conducted in an inductive setting, i.e., using different graphs for training/validation/testing, the results show that SMP can handle inductive tasks as well. [Table]
[Table]
The results on PPA are shown in Table 1.
SMPagain outperforms all the baselines, showing thatit can handle large-scale graphs with millions ofnodes and edges.
PPA is part of a recently released Open Graph Benchmark (Hu et al., 2020).
The superior performance on PPA further demonstrates the effectiveness of our proposed methodin the link prediction task.
5. N ODE C LASSIFICATION
Next, we conduct experiments of node classiﬁcation, i.e., predicting the labels of nodes.
Sincewe need ground-truths in the evaluation, we onlyadopt datasets with node labels.
Speciﬁcally, for CS and Physics, following (Shchur et al., 2018), we adopt 20/30 labeled nodes per class for training/validation and the rest for testing.
For Communities, we adjust the number as 5/5/10 labelednodes per class for training/validation/testing.
For Cora, CiteSeer, and PubMed, we use the defaultsplits that came with the datasets.
We do not adopt Email because some graphs in the dataset are toosmall to show stable results and exclude PPI as it is a multi-label dataset.
We use a softmax layer on the learned node representations as the classiﬁer and adopt accuracy,i.e., how many percentages of nodes are correctly classiﬁed, as the evaluation criteria.
We omit theresults of SMP-Identity for this task since the node representations in SMP-Identity have a ﬁxeddimensionality that does not match the number of classes.
[Table]
The results are shown in Table 3.
From the table, we observe that SMP reports nearly perfect resultson Communities.
Since the node labels are generated by graph structures on Communities and thereare no node features, the model needs to be proximity-aware to handle it well.
P-GNN, which showspromising results in the link prediction task, also fails miserably here.
On the other ﬁve graphs, SMP reports highly competitive performance.
These graphs are commonlyused benchmarks for GNNs.
P-GNN, which completely ignores permutation-equivariance, performspoorly as expected.
In contrast, SMP can manage to recover the permutation-equivariant GNNsand avoid being misled, as proven in Remark 1.
In fact, SMP even shows better results than itscounterpart, SGC, indicating that preserving proximities is also helpful for these datasets.
5. A BLATION S TUDIES
We conduct ablation studies by comparing different SMP variants, including SMP-Identity, SMPLinear, and the additional three variants as follows:
• SMP-MLP: we set F output ( · ) as a fully-connected network with 1 hidden layer.• SMP-Linear-GCN feat : we set F GNN (cid:48)
( A , F ; W (cid:48) ) in Eq.
(10) to be a GCN (Kipf & Welling,2017), i.e., induce non-linearity in message passing for features.
F output ( · ) is still linear.• SMP-Linear-GCN both : we set both F GNN ( A , E ; W ) and F GNN (cid:48)
( A , F ; W (cid:48) ) to be a GCN (Kipf& Welling, 2017),
i
e., induce non-linearity in message passing for both features and stochasticrepresentations.
F output ( · ) is linear.
We show the results for link prediction tasks in Table 4.
The results for node classiﬁcation andpairwise node classiﬁcation, which imply similar conclusions, are provided in Table 10 and Table 11in Appendix B.5.
We make the following observations.
• In general, SMP-Linear shows good-enough performance, achieving the best or second-best results on six datasets and highly competitive on the other (Communities).
SMP-Identity, which does not have parameters in the output function, performs slightly worse.
The results demonstrate the importance of adopting a learnable linear layer in the output function, which is consistent with Remark 1.
SMP-MLP does not lift the performance in general, showing that addingextra complexities in F output ( · ) brings no gain in those datasets.• SMP-Linear-GCN feat reports the best results on Communities, PPI, and PPA, indicating thatadding extra non-linearities in propagating node features are helpful for some graphs.• SMP-Linear-GCN both reports the best results on Gird with a considerable margin.
Recall thatGrid has no node features.
The results indicate that inducing non-linearities can help the stochastic representations capture more proximities, which is more helpful for featureless graphs.
5. E FFICIENCY C OMPARISON
To compare the efﬁciency of different methods quantitatively, we report the running time of differentmethods in Table 5.
The results are averaged over 3,000 epochs on an NVIDIA TESLA M40 GPU
[Table]
[Table]
[Table]
with 12 GB of memory.
The results show that SMP is computationally efﬁcient, i.e., only marginallyslower than SGC and comparable to GCN.
P-GNN is at least an order of magnitude slower exceptfor the extremely small graphs such as Grid, Communities, or Email with no more than a thousandnodes.
In addition, the expensive memory cost makes P-GNN unable to work on large-scale graphs. 5. M ORE EXPERIMENTAL RESULTS Besides the aforementioned experiments, we also conduct experiments on the following tasks: graphreconstruction (Appendix B.1), pairwise node classiﬁcation (Appendix B.2), and comparing withone-hot IDs (Appendix B.3).
Please refer to the Appendix for experimental results and corresponding analyses.
C ONCLUSION
In this paper, we propose SMP, a general and simple GNN to maintain both proximity-awarenessand permutation-equivariance properties.
We propose to augment the existing GNNs with stochasticnode representations learned to preserve node proximities.
We prove that SMP can enable GNN topreserve node proximities in theory and is equivalent to a permutation-equivariant GNN with certainparametrization.
Experimental results demonstrate the effectiveness and efﬁciency of SMP.
Ablationstudies show that a linear SMP instantiation works reasonably well on most of the datasets.
A T HEOREMS AND P ROOFSA. T HEOREM 1
Here we formulate and prove Theorem 1.
Theorem 1.
For any walk-based proximity function S ( · ) , a permutation-equivariant GNN cannotpreserve S ( · ) , except the trivial solution that all node pairs have the same proximity, i.e., S i,j =c, ∀ i, j , where c is a constant.
Proof.
We prove the theorem by contradiction.
Assume there exists a non-trivial S ( · ) which apermutation-equivariant GNN can preserve.
Consider any graph G =
( V , E , F ) and denote N =|V| .
We can create G (cid:48)
=
( V (cid:48) , E (cid:48) , F (cid:48) ) with |V (cid:48)
| = 2 N so that:
[Equation]
else ,
[Equation]
(cid:48)
[Equation]
[Equation]
Basically, we generate two “copies” of the original graph, one indexing from 1 to N , and the otherindexing from N + 1 to 2 N .
By assumption, there exists a permutation-equivariant GNN which canpreserve S ( · ) in G (cid:48) and we denote the node representations as H (cid:48)
( L ) =
F GNN ( A (cid:48) , F (cid:48) ; W G (cid:48) ) .
It iseasy to see that node v (cid:48)
i and v (cid:48) i + N in G (cid:48) form an automorphic node pair.
Using Corollary 1, theirrepresentations will be identical in any permutation-equivariant [Equation] , i.e., [Equation]
[Equation]
Also, note that there exists no walk from the two copies, i
[Equation]
e. (cid:8) v (cid:48) i (cid:32) v (cid:48)
j (cid:9) =
(cid:8) v (cid:48)
j (cid:32) v (cid:48) i (cid:9)
= ∅ , ∀ i ≤
N, j > N .
As a result, for ∀ i ≤ N, j ≤ N, ∀ (cid:15) > 0 , we have:
[Equation]
We can prove the same for ∀ i > N, j > N .
The equation naturally holds if i ≤ N, j > N ori > N, j ≤ N since (cid:8) v (cid:48)
i (cid:32) v (cid:48)
j (cid:9) = ∅ .
Combining the results, we have ∀ (cid:15) > 0 , ∀ i, j , | S
i,j − S ( ∅ ) | <2 (cid:15) .
Since (cid:15) can be arbitrarily small, the equation shows that all node pairs have the same proximityc =
S ( ∅ ) , which leads to a contraction and ﬁnishes our proof.
Notice that in our proof, G (cid:48) can be constructed for any graph, so rather than designing one speciﬁccounter-example, we have shown that there always exists an inﬁnite number of counter-examples byconstructing automorphisms in the graph.
Some may ﬁnd that our counter-examples in the above proof will lead to multiple connected components. Next, we give an alternative proof maintaining one connected component (assuming theoriginal graph is connected) under the assumption that the walk-based proximity is of ﬁnite length. Proof. Similar to the previous proof, we assume there exists a non-trivial S ( · ) which a permutationequivariant GNN can preserve.
Besides, we assume the length of S ( · ) is upper bounded by l max ,where l max is any ﬁnite number, i.e., ∀ i, j ,
[Equation]
Then, for a connected graph G =
( V , E , F ) , we create G (cid:48)
=
( V (cid:48) , E (cid:48) , F (cid:48) ) similar to Eq.
Specifically, denoting ˜ N = N + l max , we let G (cid:48) have 3 ˜ N nodes so that:
E (cid:48)
i,j =  
E i,j if i, j ≤ N if N ≤ i, j ≤ ˜ N + 1 , | j − i | =
E
[Equation]
if ˜ N < i, j ≤ ˜ N + N if ˜ N + N
[Equation]
i, j
[Equation]
[Equation]
N + 1 , | j − i | =
E
[Equation]
N if 2 ˜ N < i, j ≤ 2 ˜ N + N if 2 ˜ N + N ≤ i, j, | j − i | = if i = 3 ˜ N, j = 1 or j = 3 ˜ N,
i = else , F (cid:48)
i, : =  
F i, : if i ≤ N if N < i ≤ ˜ N
F i − ˜ N, : if ˜ N < i ≤ ˜ N + N if ˜ N + N < i ≤ 2 ˜ N
F i − 2 ˜ N, : if 2 ˜ N < i ≤ 2 ˜ N + N if 2 ˜ N + N < i .
Intuitively, we create three “copies” of G and three “bridges” to connect the copies and thus make
G (cid:48) also connected.
It is also easy to see that nodes v (cid:48)
i , v (cid:48) i + ˜ N , and v (cid:48)
i +2 ˜ N all form automorphic nodepairs and thus we have:
[Equation]
{+Next, we can see that the nodes in G (cid:48) are divided into six parts (three copies and three bridges),which we denote as V (cid:48) 
+}
= { v 1 , ..., v N } , V (cid:48)
= { v N +1 , ..., v ˜ N } , V
[Equation]
3 = (cid:8) v ˜ N +1 , ..., v ˜ N + N (cid:9) , V (cid:48) 4 = 538 (cid:8) v ˜ N + N +
[Equation]
, ..., v 2 ˜ N (cid:9) , V (cid:48) 5 = (cid:8) v 2 ˜ N +1 , ..., v 2 ˜ N + N (cid:9) , and V (cid:48) 6 = (cid:8) v 2 ˜ N + N +1 , ..., v 3 ˜ N (cid:9) .
Since V (cid:48) 2 , V (cid:48) 4 , V (cid:48)
6are bridges with length l max , any walk crosses these bridges will have a length large than l max .
Forexample, let us focus on v i ∈ V (cid:48) 1 , i.e., i ≤ N .
If v j is in V (cid:48) 3 , V (cid:48) 4 , or V (cid:48) 5 (i.e., ˜ N < j ≤ 2 ˜ N + N ), anywalk v i (cid:32) v j will either pass the bridge V (cid:48) 2 or V (cid:48) 6 and thus has a length larger than l max .
As a result,we have:
[Equation]
If v j ∈ V (cid:48) 1 or v j ∈ V (cid:48) 2 , i.e., j ≤ ˜ N , we can use the fact that v j and v j + ˜ N forms an automorphic nodepair similar to Eq.
(14), i.e., ∀
[Equation]
> 0 , we have
[Equation]
Similarly, if v j ∈ V (cid:48) 6 , i.e., 2 ˜ N + N < j , we can use the fact that v j and v j − ˜ N forms an automorphicnode pair to prove the same inequality.
Thus, we prove that if i ≤ N, ∀ (cid:15) > 0 , ∀ j , | S i,j − S ( ∅ ) | < 2 (cid:15) .
The same proof strategy can be applied to i > N .
Since (cid:15) can be arbitrarily small, the results showthat all node pairs have the same proximity S ( ∅ ) , which leads to a contraction and ﬁnishes ourproof.
A. T HEOREM 2
Here we formulate and prove Theorem 2.
Note that some notations and deﬁnitions are introduced in
Appendix A.1.
Theorem 2.
For the walk-based proximity S =
˜
A K ( ˜ A K ) T , SMP can preserve the proximity withhigh probability if the dimensionality of the stochastic matrix is sufﬁciently large, i.e., ∀ (cid:15) > 0 , ∀ δ >0 , there ∃ d 0 so that any d > d 0 :
[Equation]
where H are the node representation obtained from SMP in Eq.
The result holds for anystochastic matrix and thus is regardless of whether E is ﬁxed or resampled during each epoch.
Proof.
Our proof is mostly based on the standard random projection theory.
Firstly, since we haveproven in Theorem 1 that the permutation-equivariant representations cannot preserve any walkbased proximity, here we prove that we can preserve the proximity only using ˜ E , which can be easily achieved by ignoring H ( L ) in F output ([˜ E , H ( L ) ]) , e.g., if we set F output as a linear function, themodel can learn to set the corresponding weights for H ( L ) as all-zeros.
We set the decoder function as a normalized inner product:
[Equation]
Then, denoting a i = ˜
A Ki, : and recalling ˜ E = ˜
A K E , we have:
[Equation]
Since E is a Gaussian random matrix, from the Johnson-Lindenstrauss lemma (Vempala, 2005) (inthe inner product preservation forum, e.g., see Corollary 2.1 and its proof in (Sham & Greg, 2020)),∀ 0 < (cid:15) (cid:48) < 12 , we have:
[Equation]
By setting (cid:15) (cid:48) = (cid:15) [-max i (cid:107) a i (cid:107) , we have (cid:15) > (cid:15) (cid:48) 
-][Equation]
(cid:107) , we have (cid:15) > (cid:15)
[Equation]
2 ( (cid:107) a i (cid:107) + (cid:107) a j (cid:107) ) and:
[Equation]
which leads to the theorem by solving and setting d 0 as follows:
[Equation]
A. T HEOREM 3
Here we formulate and prove Theorem 3.
Note that some notations and deﬁnitions are introduced in
Appendix A.1.
Theorem 3.
For any length- L walk-based proximity, i.e.,
[Equation]
where len ( · ) is the length of a walk, there exists an SMP variant in Eq.
(10) with F GNN ( A , E ; W )containing L layers (including the input layer) to preserve that proximity if the following conditionshold: (1) The stochastic matrix E contains unique signals for different nodes, i.e. E i, : (cid:54) =
E j, : , ∀ i (cid:54) =
j . (2)
The message-passing and updating functions in learning ˜ E are bijective.
The decoderfunction F de ( · ) also takes E as inputs and is universal approximation.
Proof.
Similar as Theorem 2, we only utilize ˜ E during our proof.
We use e ( l ) i , 0 ≤ l < L todenote the node representations in the l th layer of F GNN ( A , E ; W ) , i.e., e (0) i
= E i, : and e ( L − 1) i =
˜
E i, : .
Our proof strategy is to show that the stochastic node representations can remember all theinformation about the walks.
Firstly, as the message-passing and updating function are bijective by assumption, we can recoverfrom the node representations in each layer all their neighborhood representations in the previouslayer.
Speciﬁcally, there exist F ( l ) ( · ) , 1 ≤ l < L such that:
[Equation]
For notation conveniences, we split the function into two parts, one for the node itself and the otherfor its neighbors:
[Equation]
For the ﬁrst function, if we successively apply such functions from the l th layer to the input layer, wecan recover the input features of the GNN, i.e., E .
Since the stochastic matrix E contains a uniquesignal for different nodes, we can decode the node ID from e (0) i , i.e., there exists F (0) self (cid:16) e (0) i ; E (cid:17)
= i .
For brevity, we denote applying such l + 1 functions to get the node ID as
[Equation]
For the second function, we can apply F ( l − 1) neighbor to the decoded vector set so that we can recover theirneighborhood representations in the ( l − 2) th layer, etc.
Next, we show that for e ( l − 1) j , there exists a length- l walk v i (cid:32) v j = ( v a 1 , v a 2 , ..., v a l ) , wherev a 1 = v i , v a l = v j if and only if F (0: l − 1) self (cid:16) e ( l − [Equation] j (cid:17) =
a l = j
and there exists e ( l − 2) , ..., e (0) suchthat:
[Equation]
This result is easily veriﬁed as:
[Equation]
a 1 , v a 2 , ..., v a l ) is a walk
[Equation]
i
[Equation]
Note that all the information is encoded in ˜ E , i.e., we can decode { v i (cid:32) v j | len ( v i (cid:32) v j ) ≤ L } frome ( L − 1) j by successively applying F ( l ) self ( · ) , F ( l ) neighbor ( · ) .
We can also apply F (0:
L − 1) self to e ( L − 1) i to getthe start node ID i .
Putting it together, we have:
[Equation]
where F ( · ) is composed of F ( l ) self ( · ) , 0 ≤ l < L and F ( l ) neighbor ( · ) , 1 ≤ l < L .
Applying the proximityfunction S ( · ) , we have:
[Equation]
We ﬁnish the proof by setting the real decoder function F de ( · ) to arbitrarily approximate this desiredfunction S ( F ( · , · )) under the universal approximation assumption.
B A DDITIONAL E XPERIMENTAL R ESULTSB. G RAPH R ECONSTRUCTION
To verify that our proposed SMP can indeed preserve node proximities, we conduct experiments ofgraph reconstruction (Wang et al., 2016), i.e., using the node representations learned by GNNs toreconstruct the edges of the graph.
Graph reconstruction corresponds to the ﬁrst-order proximitybetween nodes, i.e., whether two nodes directly have a connection, which is the most straightforward node proximity (Tang et al., 2015).
Speciﬁcally, following Section 5.2, we adopt the innerproduct classiﬁer Sigmoid( H Ti H j ) and use AUC as the evaluation metric.
To control the impactof node features (i.e., since many graphs exhibit assortative mixing, even models only using nodefeatures can reconstruct the edges to a certain extent), we do not use node features for all the models.
We report the results in Table 6.
The results show that SMP greatly outperforms permutationequivariant GNNs such as GCN and GAT in graph reconstruction, clearly demonstrating that SMPcan better preserve node proximities.
PGNN shows highly competitive results as SMP.
However,similar to other tasks, the intensive memory usage makes PGNN unable to handle medium-scalegraphs such as Physics and PubMed.
[Table]
B. P AIRWISE N ODE C LASSIFICATION
Besides standard node classiﬁcation experiments reported in Section 5.3, we follow (You et al.
and experiment on pairwise node classiﬁcation, i.e., predicting whether two nodes have thesame label.
Compared with standard node classiﬁcation, pairwise node classiﬁcation focuses moreon the relations between nodes and thus requires the model to be proximity-aware to perform well.
Similar to link prediction, we split the positive samples (i.e., node pairs with the same label) into an80%-10%-10% training-validation-testing set with an equal number of randomly sampled negativepairs.
For large graphs, since the possible positive samples are intractable (i.e. O ( N 2 ) ), we use arandom subset.
Since we also need node labels as the ground-truth, we only conduct pairwise nodeclassiﬁcation on datasets when node labels are available.
We also exclude the results of PPI sincethe dataset is multi-label and cannot be used in a pairwise setting (You et al., 2019).
Similar to
Section 5.2, we adopt a simple inner product classiﬁer and use AUC as the evaluation metric.
The results are shown in Table 7.
We observe consistent results as link prediction in Section 5.2, i.e.,
SMP reports the best results on four datasets and the second-best results on the other three datasets.
These results again verify that SMP can effectively preserve and utilize node proximities whenneeded while retaining comparable performance when the tasks are more permutation-equivariantlike, e.g., on CS and Physics.
[Table]
B. C OMPARISON WITH U SING ID S
We further compare SMP with augmenting GNNs using a one-hot encoding of node IDs, i [Table] e., theidentity matrix.
Intuitively, since the IDs of nodes are unique, such a method does not suffer from theautomorphism problem and should also enable GNNs to preserve node proximities.
However, theoretically speaking, using such a one-hot encoding has two major problems.
Firstly, the dimensionality of the identity matrix is N × N [Table] and thus the number of parameters in the ﬁrst message-passinglayer is also on the order of O ( N ) .
Therefore, the method will inevitably be computationally expensive and may not be scalable to large-scale graphs.
A large number of parameters will also morelikely lead to the overﬁtting problem.
Secondly, the node IDs are not transferable across differentgraphs, i.e., the node v 1 in one graph and the node v 1 in another graph do not necessarily share asimilar meaning.
But as the parameters in the message-passings depend on the node IDs (since theyare input features), such a mechanism cannot handle inductive tasks well.
[Table] We also empirically compare such a method with SMP and report the results in Table 8.
The resultsshow that SMP-Linear outperforms GCN onehot in most cases.
Besides, GCN onehot fails to handle
Physics, which is only a medium-scale graph, due to the heavy memory usage.
One surprising resultis that GCN onehot outperforms SMP-Linear on Grid, the simulated graph where nodes are placed ona 20 × 20 grid.
A plausible reason is that since the edges in Grid follow a speciﬁc rule, using aone-hot encoding gives GCN onehot enough ﬂexibility to learn and remember the rules, and the modeldoes not overﬁt because the graph has a rather small scale.
B. A DDITIONAL L INK P REDICTION R ESULTS
We further report the results of link prediction on three GNN benchmarks: Cora, CiteSeer, and
PubMed.
The results are shown in Table 9.
The results show similar trends as other datasets presented in Section 5.2, i.e., SMP reports comparable results as other permutation-equivariant GNNswhile PGNN fails to handle the task well.
[Table]
B. A DDITIONAL A BLATION S TUDIES
We report the ablation study results for the node classiﬁcation task and pairwise node classiﬁcationtask in Table 10 and Table 11, respectively.
The results again show that SMP-Linear generallyachieves good-enough results on the majority of the datasets and adding non-linearities does notnecessarily lift the performance of SMP.
We also compare whether the stochastic signals E are ﬁxed or not during different training epochsfor our proposed SMP. For brevity, we only report the results for the link prediction task in Table 12.
The results show that ﬁxing E usually leads to better results on transductive datasets (recall thatdatasets except Email and PPI are transductive) and resampling E leads to better results on inductivedatasets in general. The results are consistent with our analysis in Section 4.1. [Table]
[Table]
B. C OMPARISON OF P ERMUTATION - EQUIVARIANT GNN S FOR LINK PREDICTION
To investigate the performance of linear and non-linear variants of permutation-equivariant GNNsfor the link prediction task, we additionally report both the training accuracies and the testing accuracies of SGC, GCN, and GAT in Table 13.
Notice that to ensure a fair comparison, we do not adoptthe early stopping strategy here so that different models have the same number of training epochs(otherwise, if a model tends to overﬁt, the early stopping strategy will terminate the training processwhen the number of training epochs is small and result in a spurious underﬁtting phenomena).
The results show that non-linear variants of GNNs (GCN and GAT) are more likely to overﬁt, i.e.,the margins between the training accuracies and the testing accuracies are usually larger, than thelinear variant SGC.
Besides, though possessing extra model expressiveness, non-linear GNNs arealso difﬁcult to train, i.e., the training accuracies of GCN and GAT are not necessarily higher than
SGC.
The results are consistent with the literature Wu et al.
; He et al.
C E XPERIMENTAL D ETAILS FOR R EPRODUCIBILITYC. D ATASETS
• Grid
(You et al., 2019):
A simulated 2D grid graph with size 20 × 20 and no node feature.• Communities (You et al., 2019):
A simulated caveman graph (Watts, 1999) composed of 20communities with each community containing 20 nodes.
The graph is perturbed by rewiring 1%edges randomly.
It has no node feature and the label of each node indicates which communitythe node belongs to.• Email 8 (You et al., 2019): Seven real-world email communication graphs.
Each graph has sixcommunities and each node has an integer label indicating the community the node belongs to.• Coauthor Networks 9 (Shchur et al., 2018):
Two networks from Microsoft academic graph inCS and Physics with their nodes representing authors and edges representing co-authorshipsbetween authors.
The node features are embeddings of the paper keywords of the authors.• PPI 8 (Hamilton et al., 2017): 24 protein-protein interaction networks.
Each node has a 50- [Table] dimensional feature vector.
[Table] • PPA 10 (Hu et al., 2020):
A network representing biological associations between proteins from58 different species.
The node features are one-hot vectors of the species that the proteins aretaken from.
[Table] [Table] • Cora , CiteSeer , PubMed 11 (Yang et al., 2016):
Three citation graphs where nodes correspondto papers and edges correspond to citations between papers.
The node features are bag-of-wordsand the node labels are the ground truth topics of the papers.
[Table] We summarize the statistics of datasets in Table 14.
C. H YPER - PARAMETERS
We use the following hyper-parameters:
• All datasets except PPA: we uniformly set the number of layers for all the methods as 2, i.e., 2message-passing steps, and set the dimensionality of hidden layers as 32,
i
[Table] e., H ( l ) ∈ R N × 32 ,for all 1 ≤ l ≤ L (for GAT, we use 4 heads with each head containing 8 units).
We use Adamoptimizer with an initial learning rate of 0.01 and decay the learning rate by 0.1 at epoch 200.The weight decay is 5e-4.
We train the model for 1,000 epochs and evaluate the model every 5epochs.
We adopt an early-stopping strategy by reporting the testing performance at the epochwhich achieves the best validation performance.
For SMP, the dimensionality of the stochasticmatrix is d = 32 .
For P-GNN, we use the P-GNN-F version, which uses the truncated 2-hopshortest path distance instead of the exact shortest distance.•
PPA: as suggested in the original paper (Hu et al., 2020), we set the number of GNN layersas 3 with each layer containing 256 hidden units and add a three-layer MLP after taking theHadamard product between pair-wise node embeddings as the predictor, i.e., MLP( H i (cid:12) H j ) .We use Adam optimizer with an initial learning rate of 0.01.
We set the number of epochs fortraining as 40, evaluate the results on validation sets every epoch, and report the testing resultsusing the model with the best validation performance.
We also found that the dataset had issueswith exploding gradients and adopt a gradient clipping strategy by limiting the maximum p2norm of gradients as 1.0.
The dimensionality of the stochastic matrix in SMP is d = 64 .
C. H ARDWARE AND S OFTWARE C ONFIGURATIONS
All experiments are conducted on a server with the following conﬁgurations.
• Operating System: Ubuntu
18.04.1 LTS• CPU:
Intel(R) Xeon(R)
CPU E5-2699 v4 @
2.20GHz• GPU: NVIDIA TESLA M40 with 12 GB of memory 11 https://github.com/kimiyoung/planetoid/tree/master/data [Table] Software: Python 3.6.8, PyTorch [-1.4.0, PyTorch Geometric 1.4.3, NumPy 1.18.1, Cuda 10.
-]1.4.
[Table]
, PyTorch
[Table]