PF-GNN: D IFFERENTIABLE PARTICLE FILTERING BASED APPROXIMATION OF UNIVERSAL GRAPH REPRESENTATIONS
A BSTRACT
Message passing Graph Neural Networks (GNNs) are known to be limited in expressive power by the 1-WL color-reﬁnement test for graph isomorphism.
Other more expressive models either are computationally expensive or need preprocessing to extract structural features from the graph.
In this work, we propose to make
GNNs universal by guiding the learning process with exact isomorphism solver techniques which operate on the paradigm of Individualization and Reﬁnement (IR), a method to artiﬁcially introduce asymmetry and further reﬁne the coloring when 1-WL stops.
Isomorphism solvers generate a search tree of colorings whose leaves uniquely identify the graph.
However, the tree grows exponentially large and needs hand-crafted pruning techniques which are not desirable from a learning perspective.
We take a probabilistic view and approximate the search tree of colorings ( i.e. embeddings) by sampling multiple paths from root to leaves of the search tree.
To learn more discriminative representations, we guide the sampling process with particle ﬁlter updates, a principled approach for sequential state estimation.
Our algorithm is end-to-end differentiable, can be applied with any GNN as backbone and learns richer graph representations with only linear increase in runtime.
Experimental evaluation shows that our approach consistently outperforms leading GNN models on both synthetic benchmarks for isomorphism detection as well as real-world datasets.
I NTRODUCTION
In recent years, Graph Neural Networks (GNNs) have emerged as learning models of choice on graph structured data.
GNNs operate on a message passing paradigm (Kipf & Welling, 2016; Defferrard et al., 2016; Veliˇckovi´c et al., 2017; Gilmer et al., 2017), where nodes maintain latent embeddings which are updated iteratively based on their neighborhood.
This way of representation learning on graphs provides the necessary inductive bias to encode the structural information of graphs into the node embeddings.
The process of message passing in GNNs is equivalent to vertex color-reﬁnement procedure or the 1-dimensional Weisfeiler-Lehman (WL) test used to distinguish non-isomorphic graphs (Xu et al., 2018; Morris et al., 2019).
Consequently, GNNs suffer from the limitations of 1-WL color-reﬁnement in their expressive power.
In each step of 1-WL color-reﬁnement, two vertices get different colors if the colors of neighboring vertices are different.
The procedure stabilizes after a few steps when the colors cannot be further reﬁned.
Due to the symmetry in graph structures, certain non-isomorphic graphs induce same colors upon 1-WL reﬁnement.
Higher-order WL reﬁnement and their neural k -GNN versions break some of the symmetry by operating on k -tuples of nodes.
They are more expressive but require exponentially increasing computation time and hence, are not practical for large k .
Motivated by the fact that a fully expressive graph representation learning model should be able to produce embeddings that can distinguish any two non-isomorphic graphs (Chen et al., 2019b), we turn to exact graph isomorphism solvers for better inductive biases in our learning algorithm.
Most of the practical graph isomorphism solvers use 1-WL in combination with the traditional technique of individualization and reﬁnement (IR) (McKay & Piperno, 2014; Junttila & Kaski, 2011) for coloring the graph.
Individualization is the process of artiﬁcially introducing asymmetry by recoloring a vertex and thereby, distinguishing it from the rest of the vertices.
Reﬁnement refers to 1-WL reﬁnement which can propagate this information to recolor the rest of the graph.
The two graphs shown in Fig.
1 are not distinguishable after 1-WL reﬁnement but induce different colorings after one IR step.
The IR process is repeated for each reﬁnement until every vertex gets a unique color.
However, in order to maintain permutation-invariance, whenever a vertex is individualized, other vertices that have the same color need to individualized as well and thereafter reﬁned.
This process generates a search tree with colorings as nodes, and can grow exponentially in worst case.
In this work, we propose to learn graph representations with the inductive bias of the search tree generated by the graph-isomorphism solvers.
However, generating the complete search tree is computationally expensive.
Isomorphism-solvers prune the search tree by detecting automorphisms on the ﬂy as they generate the tree.
Nevertheless, detecting automorphisms is non-trivial from the perspective of end-to-end discriminative learning and hence, we need to approximate it.
To this end, we ﬁrst deﬁne a universal neural graph representation based on the search tree of colorings.
Then we take a probabilistic view and approximate it by sampling multiple paths from root to leaves of the search tree.
We observe that the process of sampling a vertex, individualizing it and the subsequent reﬁnement resembles the state transition in sequential state estimation problems.
Hence, we model our sampling approach on the principled technique of Sequential Monte Carlo or Particle Filtering.
We introduce Particle Filter Graph Neural Networks (PF-GNN), an end-to-end learnable algorithm which maintains a weighted belief distribution over a set of K graph colorings/embeddings.
With each step of IR, PF-GNN transitions to a new set of embeddings.
It then updates the belief by re-weighting each particle with a discriminatively learned function that measures the quality of the reﬁnement induced after the transition.
With this inductive bias, the belief evolves over time to be more discriminative of the input graph.
After a few steps, we can use the belief along with the embeddings to generate the ﬁnal representation of the graph.
Our approach is simple, efﬁcient, parallelizable, easy to implement and can learn representations to distinguish non-isomorphic graphs beyond 1-WL GNNs.
We evaluate PF-GNN over diverse set of datasets on tasks which are provably not learnable with 1-WL equivalent GNNs.
Furthermore, our experiments on real-world benchmark datasets show the strong performance of PF-GNN over other more expressive GNNs.
R ELATED W ORK
It was established that GNNs are limited in expressive power, and cannot go beyond 1-WL test for graph isomorphism by Xu et al.
and Morris et al.
Later analysis of GNN has shown other limits of GNNs like counting substructures and detecting graph properties (Arvind et al., 2020;
Chen et al., 2020; Loukas, 2019; Dehmamy et al., 2019; Srinivasan & Ribeiro, 2019).
Chen et al.
(2019b) further formalizes the intuition that there is equivalence between learning universal graph representations and solving the graph isomorphism problem.
Thereafter, many models have been proposed that are more expressive than 1-WL.
Prominent ones are k -GNNs and their equivalent models (Morris et al., 2019; Maron et al., 2019; Vignac et al., 2020; Morris et al., 2020), but they are difﬁcult to scale beyond 3-WL.
Other methods need to preprocess the graph to ﬁnd substructures (Bouritsas et al., 2020; Li et al., 2020), which are not task-speciﬁc and may incur more computation time.
Another way to improve expressivity of GNNs is to introduce randomness (You et al., 2019;
Sato et al., 2020; Abboud et al., 2020; Zambon et al., 2020).
However, adding uniform randomness interferes with the learning task, and hence these models have not shown good performance on real-world datasets.
In PF-GNN, randomness is controlled as only a subset of nodes are sampled.
Furthermore, since all components are learned discriminatively, the representation is tuned towards the end task.
Our approach of learning with particle ﬁlter updates is inspired from recent works which make particle ﬁlters differentiable (Karkus et al., 2018; Ma et al., 2019; 2020).
P RELIMINARIES
Let G n be the set of all graphs of n vertices with vertex set V = { 1 , 2 , . .
. , n } and edge set E ,
Isomorphism between any two graphs G , H ∈ G n is a bijection f : V G → V H such that
( u, v ) ∈
E G ⇐⇒ ( f ( v ) , f ( u ))
∈
E H .
An automorphism of G is an isomorphism that maps G onto itself.
One way of identifying non-isomorphic graphs is by generating unique colorings for graphs based on their structures in a permutation-invariant way and then, comparing them.
A colouring of the graph G ∈ G n is a surjective function π : V → N , which assigns each vertex to a natural number.
The number of colors is denoted by | π | .
The set of vertices sharing the same color form a color cell in the coloring.
We denote the set of colored vertices with π = { p 1 , p 2 , . . .
, p k } where p i is a color cell.
A coloring in which every vertex gets a distinct color is called a discrete colouring i.e. | π | = n .
For any two colorings π, π (cid:48) , we say that π (cid:48) is ﬁner than or equal to π , written π (cid:48)
(cid:22) π , if π ( v ) <
π ( w ) ⇒ π (cid:48) ( v ) <
π (cid:48) ( w ) for all v, w ∈ V .
i.e. each cell of π (cid:48) is a subset of a cell of π , but the converse is not true.
Coloring is also loosely called a partition , since it partitions V into cells.
A coloring is an equitable partition when any two vertices of the same color are adjacent to the same number of vertices of each color.
1-dimensional Weisfeiler Lehman test is a simple and fast procedure to color graphs.
It starts with the same initial color for all vertices.
Then, it iteratively reﬁnes the coloring of the graph by mapping the the tuple of the color of a vertex and its neighbors to a distinct new color.
i.e at step t , π t +1 ( v ) =
HASH (cid:16) π t ( v ) , {{ π t ( u ) , u ∈ N ( v ) }} (cid:17) , where {{}} denotes a multiset and N ( v ) is the set of adjacent vertices of v .
The algorithm terminates when π forms an equitable partition.
3. S EARCH T REE OF COLORINGS
Equitable coloring cannot be further reﬁned due to the symmetry in the graph structure.
To break the symmetry, exact isomorphism solvers employ the technique of individualization-reﬁnement to generate further reﬁned colorings.
Individualization is the technique of breaking the symmetry in the graph by distinguishing a vertex with a new unique color.
Once a vertex is individualized, 1-WL reﬁnement is used to reﬁne the coloring until a further reﬁned equitable partition is reached.
However, this comes at a cost.
In order to maintain permutation-invariance, we have to individualize and reﬁne all the vertices with the same color.
As a result, we get as many reﬁned colorings as the number of vertices with the chosen color.
This procedure can be repeated iteratively for each of the reﬁned coloring and the process takes shape of a rooted search tree of equitable colorings.
The search tree has initial 1-WL coloring at the root.
Then a non-singleton color cell called the target cell is chosen and all vertices of the target cell are individualized in parallel and thereafter, colorings are reﬁned.
The reﬁned equitable colorings form the child nodes of the root.
After sufﬁcient repetitions, we get a set of discrete colorings at the leaves.
This search tree is unique to the isomorphism class of graphs, i.e. all non-isomorphic graphs produce distinct search trees, and consequently, discrete colorings at the leaves.
An example of search tree is shown in
P ROPOSED M ETHOD
In this section, we ﬁrst deﬁne a universal representation of any n -vertex graph, then propose a practical approximation of the representation.
For any arbitrary n -vertex graph G , we aim to learn its neural representation f ( G ) that can uniquely identify G .
To achieve this, we can construct a computation graph which mimics the search tree of colorings.
We propose to design a neural equivalent of the search tree.
To do this, we can model the target-cell selector using a learnable function that outputs scalar scores on the vertices based on their embeddings, and then individualize those which have the maximum score.
Note that node embeddings in GNN are analogous to colors in 1-WL reﬁnement.
We can then adopt GNN to produce reﬁned set of embeddings.
If we have discrete embeddings after T iterations, then f ( G ) can be computed as,
[Equation]
where I is a sequence of vertices individualized iteratively ( I identiﬁes a single path from root to a leaf in the search tree) and the summation is over all such paths.
π I T is the discrete coloring (discrete embeddings) of G after individualizing and reﬁning with vertices in I .
ψ is an multiset function approximator and ρ is an MLP.
Theorem 1 presents the expressive power of f ( G ) .
Theorem 1.
Consider any n -vertex graph G with no distinct node attributes.
Assume we use universal multiset function approximators for target-cell selection and graph pooling function ψ , and a GNN with 1-WL equivalent expressive power for color reﬁnement, then the representation f ( G ) in Eqn.is permutation-invariant and can uniquely identify G .
Proof of Theorem 1 is provided in Appendix A.3.
4. P ROBABILISTIC REPRESENTATION
The graph representation in Eqn. 1 is computationally expensive to obtain as the size of the search tree can grow exponentially with the number of vertices.
Graph isomorphism solvers like Nauty and
Traces (McKay & Piperno, 2014) prune the search tree by detecting automorphisms in order to run faster.
To speed up, we instead propose to approximate the representations for the set of colorings at the leaves as suggested by Theorem 1 using sampling techniques, and approximate the pruning process by learning to down-weight certain subtrees.
By sequentially sampling the vertices to individualize and reﬁne, we can get a sample set of paths/trajectories from root to leaf in the search tree.
To facilitate the approximation, we use the expectation of the embeddings instead of the sum of the embeddings.
Assume we need T steps to generate discrete colorings.
Let I be the sequence of vertices selected to individualize in one trajectory from root to leaf.
Then, we have the following expectation
[Equation]
where Π I is the set of all paths in the search tree.
In this case, we want the E [ ψ ( G , π I T )] to be different if the set of leaves is different; function ψ with this property exists if the number of leaves and distinct leaf representations are bounded, e.g. if we limit the size of graphs we consider.
To see this, observe that each distinct graph will have a distinct canonical embedding as one of the leaves of its search tree (McKay & Piperno, 2014), and ψ can set one of the components of its output vector to be the indicator function of this canonical embedding.
To approximate ˜ f ( G ) , we can initialize a set of K colorings and in each step, sample a vertex to individualize and reﬁne the colorings.
After multiple such steps, we average the K embeddings to obtain the ﬁnal graph embedding.
We bound the approximation error of the sampling process in the following theorem.
Theorem 2.
Assume we need D dimensional embeddings for some D > 0 , in order to uniquely represent any graph of n -vertices.
Consider two n -vertex graphs G 1 and G 2 whose universal representations in Eqn.
2 are f 1 and f 2 after T levels of the search tree.
Let the max-norm distance between f 1 and f 2 be at least d i.e. generating the full level- T search tree for both graphs will separate them by d .
Assume that the values in embeddings generated by the tree are strictly bounded in the range of [ − M, M ] .
Then, with probability of at least 1 − δ , the approximate embeddings, generated by sampling K paths from root to leaves, are separated in max-norm distance by d ± (cid:15) provided the number of root to leaf paths sampled are at least
[Equation]
The proof is provided in Appendix A.3.
Further reﬁnement of the method can be made in practice.
First, we learn the function used to select the node for individualization so that it is adaptive to the end-task.
Additionally, the learned function may select different nodes with different probabilities, so the probability of each path down the tree is different.
To speed up, practical solvers prune subtrees when they are unlikely to be helpful.
We propose to approximate this process by learning how to de-emphasize certain subtrees using particle ﬁlter methods as described in the next section.
4. P ARTICLE F ILTER G RAPH N EURAL N ETWORK Particle Filtering:
Particle ﬁltering is a monte-carlo technique that sequentially estimates the posterior distribution of the state in non-linear dynamical systems when partial observations are made.
Consider a process where we receive an observation o t at each time step when it transitions from state x t to a new state x t +
and we have a transition model p ( x t +1 | x t ) , an observation model p ( o t | x t ) and a sequence of observations [ o 1 , o 2 . . .
o T ] .
Then particle ﬁlters can approximate the posterior distribution of p ( x T | o t =1: T ) by maintaining a belief over the state space with a set of particles and their corresponding weights.
The belief b ( x t ) is given by (cid:104) x kt , w kt (cid:105) k
=1: K where (cid:80) k
=1: K w kt = 1 .
At every time step t , each particle transitions to a new state x kt +1 by sampling from p ( x kt +1 | x kt ) .
We then receive an observation o t +1 and the conditional likelihood of o t +1 is multiplied with the weight w kt before normalizing the weights i.e. w kt
+
=
η
· p ( o t +1 | x kt +1 ) · w kt , where η is the normalizing factor.
Usually, the belief update is followed by a resampling step where a new set of particles are sampled from the distribution of (cid:104) w kt +1 (cid:105) and the weights are reset to 1 /K .
This step handles the problem of particle degeneracy i.e. when most weights are nearly zero.
It helps in concentrating on most promising particles.
Refer to the Appendix for detailed explanation.
Belief of graph colorings: The task of estimating the expected coloring of a graph G in Eqn.highly resembles the problem of state estimation in particle ﬁltering algorithm.
If we consider a graph coloring as a state, then sampling a vertex from a distribution, individualizing it and reﬁning the graph coloring can be seen as a stochastic transition from one state to the next.
We do not receive a separate observation after every transition.
Instead, we use a function of the current graph embeddings to up-weight or down-weight each particle in order to approximate the pruning of non-promising subtrees.
The function is akin to unnormalized observation likelihood p ( o t +1 | x kt +1 ) in particle ﬁlters.
Speciﬁcally, we approximate the expectation in Eqn.
2 with a set of k particles and the associated set of weights which form a belief distribution over the set of particles.
[Equation]
The belief b t ( G ) at time step t approximates the posterior distribution over the set of colorings of graph
G .
At each t , we update the belief with a stochastic transition of colorings, and the weights with an observation function.
We further do resampling to focus on and expand the most promising colorings.
Finally, we compute the mean coloring from the belief which gives an estimate of Eqn. 2.
Note that each step can be computed in parallel for all particles and the whole process is discriminatively trained end-to-end.
We call our model Particle Filter - Graph Neural Network (PF-GNN) and give the pseudo-code in Algorithm 1.
Next, we describe the main steps of the algorithm in detail.
Initialization: We ﬁrst initialize the node embeddings with node attributes or with a constant value when attributes are not present.
Then, we run a 1-WL equivalent GNN for a ﬁxed number of iterations to get the initial coloring/embeddings of the graph.
Next, we instantiate our belief with k copies of the embeddings along with uniformly distributed weights i.e ∀ k, w k 1 = 1 /K .
Our belief after initialization is then (cid:104) ( G , H k 1 ) , w k 1 (cid:105) k
=1: K , where H k 1 represents the n × d matrix of node embeddings of the k th particle at time step 1 .
Transition step: In the transition step, for each particle, we pick a vertex, individualize it and reﬁne the coloring.
For this, we learn a policy function P ( V| H kt ; θ ) which is a discrete distribution on the set of vertices conditioned on the node embeddings H kt .
This can be modeled either as a separate
GNN or an MLP on node embeddings, which gives a non-negative score for each vertex.
We can then normalize the scores and sample a vertex from the distribution.
Next, we individualize the sampled vertex by transforming its embeddings with another parameterized MLP.
In this way, recoloring of vertices is also learnt from the data.
Now that the symmetry in the recolored graph is broken, we reﬁne the embeddings H kt with a GNN for a ﬁxed number of iterations.
This is repeated for every particle to get new set of particles (cid:104)
H kt +1 (cid:105) k
=1: K .
Speciﬁcally, the transition step involves:
[Equation] [Equation]
where M kt is a mask matrix of ones.
Since we are operating on a set of k particles, we need to share
GNN parameters across the k transitions.
However, we are free to use separate GNN parameters for each step t .
Also, it means increasing the number of particles does not lead to increase in parameters.
Particle weights update step:
In particle ﬁltering, an observation is received after every transition.
Then an observation function is applied to estimate the likelihood of the observation, which will be used to update the particle weights.
In our case, since the new information (or observation) is contained in the reﬁnement, we model the observation as a latent variable conditioned on the reﬁned coloring.
A learnable observation function f obs ( H kt +1 ; θ o ) is used to measure the value of the reﬁned embeddings.
Speciﬁcally, f obs ( H kt +1 ; θ o ) is a set function approximator which outputs a non-negative score on the set of node embeddings.
The output of f obs is then multiplied with the weights w kt , and the set of k weights are normalized to get the updated distribution over k colorings.
[Equation]
Note that in the absence of intermediate observations, following Ma et al.
(2020), we have used a discriminative function f obs ( H kt +1 ; θ o ) to up/down-weight the state.
Since it is discriminatively trained to optimize the end task, it performs the same function as p ( o | s ) in particle ﬁlters.
Resampling step: A crucial step in the particle ﬁlter algorithm is the resampling operation.
The weights of most particles tend to degenerate ( i.e. , become near zero).
To overcome this, we need to resample K new particles from the discrete distribution of (cid:104) w kt +
(cid:105) k
=1: K and set their weights to 1 /K .
Since this is a non-differentiable operation, we adopt the differentiable soft-resampling strategy proposed in Karkus et al.
Speciﬁcally, we sample new particles from the convex combination of particle weights and uniform distribution i.e., q t ( k ) =
αw kt + (1 − α )1 /K , where α ∈ [0 , 1] is a tunable parameter.
The new weights can then be computed by importance weighting, w (cid:48)
kt =
p t ( k ) q t ( k ) =
[Equation]
kt +(
[Equation]
)1 /K , where p t ( k ) is the weights distribution.
This is a differentiable approximation which provides non-zero gradients for the full particle chain with trade off between desired sampling distribution ( α = 1 ) and the uniform sampling distribution ( α = 0 ).
Readout: After T iterations of individualization-reﬁnement over the set of k particles, we have the belief (cid:104) ( G , H kT ) , w kT (cid:105) k
=1: K .
We can then adopt sufﬁciently expressive function over the belief as it is a probability distribution on the set of embeddings.
Ma et al.
(2019) proposed Moment Generating
Function (MGF) features to readout.
However, we found that they are numerically unstable in our experiments.
Hence instead, we take the mean particle (cid:80) k
=1: K w kT H kT for all t ∈ { 1 , . . .
, T } and concatenate the embeddings before readout for the ﬁnal prediction.
Training: We introduced randomness in two parts of our algorithm, one where we sample for individualization, and the other during resampling of weights.
Since our resampling step is differentiable, we only need to optimize the expected loss under P ( I ) , which gives us a REINFORCE type gradient update along with task-speciﬁc loss.
Let ˜ y be the prediction, y the target and I be the sequence of vertices individualized, then the expected loss over K such sequences is,
[Equation]
and the gradient of the expected loss is ∇
[Equation]
On the number of paths needed to get good embeddings with less variance: Since PF-GNN is a sampling based approach, we would want a small number of paths ( K ) to represent the search-tree with less variance.
We now discuss how PF-GNN can generate good embeddings with small K .
Firstly, K can be small whenever most colorings are repeated at each level of the tree due to the presence of automorphisms in the graph.
For example, in Figure 2, all the leaves are rotated versions of the same coloring.
Exact isomorphism solvers ﬁnd automorphisms as well as less promising subtrees on the ﬂy and prune the search-tree.
This pruning is approximated in PF-GNN by reweighting with observation function followed by resampling.
Furthermore, our individualization function is learnt end-to-end and will adapt to reduce variance if it helps the end performance.
Finally, we can tune T (which may increase variance) and K (which reduces variance) to ﬁt the dataset for optimal performance.
E XPERIMENTS
We evaluate PF-GNN on a set of three separate experiments i.e. graph isomorphism test, graph property detection and on real world datasets.
Our experiments are set up to ﬁrst evaluate the expressivity of
PF-GNN and then to compare with other leading GNN models on real world datasets
[Table]
5. G RAPH ISOMORPHISM DETECTION
[Table] [Table]
We ﬁrst evaluate the expressiveness of PF-GNN on three graph isomorphism test datasets. 1)
SR25 (Mckay, 2021).
The task is to distinguish all 105 pairs of non-isomorphic strongly regular graphs, which are one of the hard class of graphs for the graph isomorphism problem.
2) EXP (Abboud et al., 2020) consists of 600 pairs of non-isomorphic graphs.
For both SR25 and EXP, we train the models with all graphs and test whether the models can learn embeddings that distinguish each pair.
Following Balcilar et al.
(2021), we use 3-layer models for all the backbones.
We run experiments with 10 different seed and report the average accuracy and standard deviation.
3) CSL (Murphy et al., 2019) consists of 150 4-regular graphs divided evenly into 10 isomorphism classes.
The task is to classify the graphs into isomorphism classes that can only be accomplished by models with higher expressive power than 1-WL.
As in Dwivedi et al.
; Murphy et al.
(2019), we follow 5-fold cross validation for evaluation.
PF-GNN is a method that can be easily applied with any GNN as backbone.
In Tab. 1, we compare the isomorphism detection results for models with and without PF-GNN, including GCN (Kipf & Welling, 2016), GAT (Veliˇckovi´c et al., 2017),
GIN (Xu et al., 2018) and GNNML1 (Balcilar et al., 2021).
These models are known to be at most 1-WL, and are expected to fail in the task.
After augmenting these models with PF-GNN, most of them can achieve 100% accuracy.
We further evaluate our method against several strong and more expressive models such as PPGN (Maron et al., 2019),
GNNML3 (Balcilar et al., 2021) in Tab 1 and Ring-GNN (Chen et al., 2019b), RP-GNN (Murphy et al., 2019) and 3-WL GNN (Dwivedi et al., 2020) in Tab. 2 for the CSL dataset.
As shown in Tab.and Tab. 2, our proposed method consistently outperforms them experimentally.
Note that PPGN is equivalent to 3-WL in expressive power.
Interestingly, PF-GNN outperforms both PPGN as well as recently proposed GNNML3 on SR25 dataset.
This shows that PF-GNN is able to distinguish graphs which even 3-WL equivalent models are not able to distinguish.
Note that our setting differs from Balcilar et al.
(2021) where there is no learning and PPGN gets 100% accuracy for EXP dataset.
5. G RAPH PROPERTIES :
[Table]
In this experiment, we evaluate
PF-GNN on two property detection tasks where 1-WL GNNs fail.
One is counting the number of triangles in TRIANGLES dataset (Knyazev et al., 2019) and the other is ﬁnding the node clustering co-efﬁcient in LCC dataset (Sato et al., 2020).
TRIANGLES (Knyazev et al., 2019) is a
[Table] [Table]
[Table]
large synthetic dataset with 45000 graphs.
The task is to count the number of triangles in each graph.
LCC dataset (Sato et al., 2020) has graphs with nodes labeled with their local clustering co-efﬁcient.
TRIANGLES has 10 graph-level classes and LCC has 3 classes at node-level.
Both datasets have two different test sets.
The test sets Large and X for TRIANGLES and LCC respectively come with graphs with nearly 100 nodes whereas the training set has less than 25 nodes.
Better prediction in these test sets suggests that the model is able to generalize better.
Our baselines are 1-WL GNNs, Chebnet and RNI (Sato et al., 2020; Abboud et al., 2020).
RNI is 1-WL GNN but with randomly initialized node features which makes it a universal approximator on graphs.
We also compare with RNI on partial levels of individualization as in Abboud et al.
The empirical results in Tab.
3 show that PF-GNN outperforms all RNI models as well as more expressive Chebnet on both datasets.
The difference is more evident in TRIANGLES-Large where the baselines are not able to generalise to graphs with more nodes.
Comparison with RNI suggests that learning the subset of nodes to individualize is helpful for generalization towards the end-task.
5. R EAL WORLD BENCHMARKS
We now evaluate PF-GNN on various real-world benchmark datasets including ZINC (Jin et al., 2018; Dwivedi et al., 2020), ALCHEMY (Chen et al., 2019a) and QM9 (Ruddigkeit et al., 2012;
Ramakrishnan et al., 2014) and Ogbg-molhiv (Hu et al., 2020).
For ZINC and ALCHEMY, we use the ﬁxed training set of 10 K graphs and 1 K graphs each for testing and validation.
QM9 dataset consists of nearly 130000 graphs.
As is the standard practice, we use the 10 K each for testing and validation, and rest of the graphs for training.
For ogbg-molhiv (Hu et al., 2020), we use full 9-dimensional node features and adopt the standard scaffold split.
We report mean absolute error for the former three datasets, and ROC-AUC for ogbg-molhiv.
Details are in Appendix A.5.5.
[Table]
For ZINC, we compare with neural k -WL and their sparse versions (Morris et al., 2020) along with other recent models including
PNA (Corso et al., 2020), DGN (Beani et al., 2021).
For Alchemy, we compare with sparse k -GNNs.
Scores of PNA on Alchemy are reported by running their open source code.
Tab.
4 shows consistent improvement of error rates in both Zinc and Alchemy datasets.
In both the datasets, PF-GNN shows strong performance in comparison to all reported models.
For QM9, we compare with most of the strong baselines including sparse k -GNN, 3 -IGN (Maron et al., 2018) and Dimenet (Klicpera et al., 2020).
In Tab. 5, we report MAE scores averaged over all 12 targets.
PFGNN achieves best performance compared to all the baseline models.
Note that in this setting all targets are jointly trained and include position co-ordinates used either as node or edge features.
It has been observed that QM9 results are strongly correlated with positional features (Klicpera et al., 2020).
This is not an appropriate setting to evaluate PF-GNN, as the main motivation of PF-GNN is to identify nodes without distinguishing features.
Therefore, we also report
[Table]
results in Tab. 6 for the setting where no positional features are used.
We compare with the baseline results reported in Brockschmidt (2020).
PF-GNN outperforms GNN-FiLM (Brockschmidt, 2020) in most of the targets.
For Ogbg-molhiv, we compare with some of the strong baselines including
GCN, GIN, PNA, DGN and GSN (Bouritsas et al., 2020).
In Tab. 7, we report ROC-AUC scores which shows PF-GNN achieves competitive performance against the baselines.
5. A BLATION STUDIES
Performance:
We ﬁrst run ablative experiments of PF-GNN with different number of IR steps ( T ) and number of particles ( K ).
As presented in Tab. 8, best performance is achieved for TRIANGLESlarge at T = 3 and K = 16 , and for CSL at T = 4 and K = 8 .
Note that a single IR layer is insufﬁcient to reach the best performance on both TRIANGLES-large and CSL datasets.
Subsequent addition of IR steps is beneﬁcial up to a few rounds but negatively impacts the performance on the end task after T = 4 for these two tasks.
This is not the case for the number of particles K .
With higher K , the score either improves or remains the same.
This empirically veriﬁes that higher number of samples help in better approximation of the true expectation of the embeddings.
[Table]
We further study two ablative models of PF-GNN, 1) without resampling updates and 2) training without the policy gradient loss.
Tab.
9 shows that the test performance of
PF-GNN on TRIANGLES-large dataset depends on both the resampling operation as well as training with the policy loss.
Interestingly, the policy loss seems to help signiﬁcantly which suggests that learning the nodes to individualize is important.
Furthermore, TRIANGLES-large test set contains graphs with larger number of nodes as compared to the graphs in training data.
Hence, the results indicate that both operations can help our model to generalize better.
Additional ablation experiments are included in the Appendix A.
Runtime: PF-GNN is efﬁcient in terms of its runtime given its expressive power.
Since, we run a
GNN in each step for ﬁxed iterations, the runtime of PF-GNN is bounded by O ( nT ) depth T with parallel computation of K paths.
We further empirically evaluate the increase in runtime of PF-GNN compared to the 1-WL GNN model.
Fig.
3 plots the ratio of increase in runtime compared to the base
GNN, as we increase number of particles and depth.
The results conﬁrm that the runtime increase is only linear as we keep increasing T .
Note that we do not need longer depth as a few individualizations are sufﬁcient to break most of the symmetry in graphs.
Additional runtime results in Appendix A.5.3.
C ONCLUSION
Message passing GNNs are known to be not fully expressive for learning functions on graph structured data.
In this paper, we ﬁrst deﬁne a way of learning universal representations of graphs by guiding the learning process with exact isomorphism solver techniques.
However, learning these representations can take exponential time in worst case.
To be practically useful, we proposed PF-GNN, a differentiable particle ﬁltering based model to approximately learn the universal representations.
PF-GNN learns highly discriminative representations while being efﬁcient in runtime requirements.
It is ﬂexible and can be used with any GNN as a backbone.
Experimental results on various benchmarks demonstrate that PF-GNN can distinguish even those graphs not distinguishable by 3-WL equivalent models and gives competitive results on real-world datasets.
E THICS S TATEMENT
Our work is a generic method to improve the expressive power of any GNN backbone models.
As our work applies to GNNs generally, its ethical impacts (both positive and negative) will be on the broad approach of GNNs rather than on any speciﬁc application.
R EPRODUCIBILITY S TATEMENT
The complete proofs for Theorem 1 and Theorem 2 are included in Appendix A.3.
For experiments, the settings are stated in Section 5, and more details of the model architectures and hyperparameter for training are speciﬁed in Appendix A.5.5.
Details of the datasets used can be found in Appendix A.5.7.
To further facilitate reproducibility of the work, we provide publicly available code at https://github.com/pfgnn/PF-GNN. A CKNOWLEDGEMENT This research is supported in part by the National Research Foundation, Singapore under its AI Singapore Program (AISG Award
No: AISG2-RP-2020-016)
A A PPENDIX A. P ARTICLE FILTER ALGORITHM
A Particle ﬁlter estimates the posterior distribution of the states of a Markov process when noisy/partial observations are received.
It uses a set of samples or particles to represent the posterior distribution of the state given a series of observations.
One of the main beneﬁt of particle ﬁltering is that the particle set can approximate any arbitrary distributions,
Consider a Markov process with the state transition probability given by p ( x t | x t − 1 ) where x t is the state at time step t .
For every transition, we receive an observation o t described by observation probability of p ( o t | x t ) .
Given a series of observations o t =1: T = { o 1 , . . .
, o T } , particle ﬁlters estimate the posterior probability p ( x T | o t =1: T ) , of the state x T conditioned on the sequence of observations.
Speciﬁcally, particle ﬁlters sequentially approximate the posterior distribution by maintaining a belief over the states b ( x ) with a set of K particles , i.e. , weighted samples from the probability distribution,
[Equation]
10) where (cid:80)
k w k t
=1 , K is the number of particles, w k t is the particle weight and x k t is the particle state at time step t .
The particles are sequentially updated with three steps: transition step, observation update step and the resampling step.
Transition step:
At each time step, the particles transition to new states depending on the current state.
For each of the
K particles, the new state is computed by sampling from the probabilistic transition model given by,
[Equation]
(11) where p ( x t | x kt − 1 ) is the probability of state x t given the previous state x kt − 1 .
Observation update step:
After a transition step, we receive an observation which is a noisy signal for the state of the system.
Therefore, after every transition, the weights of the belief need to be updated with the observation received.
The observation compatibility function given by f obs ( o t | x t ) , measures the conditional likelihood of the observation given the state.
Since, we have K particles, we measure the observation likelihood for each one of them by computing
[Equation]
The weights of the particle set are updated by mulitplying with observation model and then normalizing the weights across the K -particle set.
[Equation]
13) where η = (cid:80) j
=1: K f jt w jt − 1 is the normalization factor.
Resampling step:
A very common issue in sequential state estimation with a set of particles is the problem of particle degeneracy when most of the weight tends to be concentrated in one or two particles.
This issue is resolved by resampling a set of new particles from the weights distribution.
The new set of particles are sampled from the weighted distribution (cid:104)
˜ w kt (cid:105) k
=1: K with repetition.
Following the resampling operation, the weights are reset to a uniform distribution by setting,
[Equation]
The new belief represented by the particle set (cid:104) w kt , x kt (cid:105) approximates the same distribution, but devotes its space to the most promising particles.
At any step t , the state can be estimated by taking the weighted mean of the particle set,
[Equation]
A. D ERIVATION OF THE GRADIENT UPDATE
The prediction ˜ y T generated by the PF-GNN depends on multiple sampling steps.
Therefore, in order to optimize the parameters, we minimize the expected loss under the random variables sampled during the process.
The two random variables in PF-GNN are the particles selected during resampling step with weights (cid:104) w kt (cid:105) k
=1: K and the sequence of individualization vertices
I .
Since, we use softresampling and are able to backpropagate through the weights (cid:104) w kt (cid:105) k
=1: K , we only optimize the expected loss with random variable I .
Let y be the prediction target, I be the sequence of vertices individualized in one particle chain from root to leaf, π I i =1: T be the colorings generated in one sequence of individualizations I .
Then the expected loss is over all such sequences.
[Equation]
where P ( I|G , π I t =1: T ; θ ) is the probablity of the full chain for t = 1 : T of the sequence I and
L (˜ y
( π I t =1: T ) , y ; θ ) is the loss of the ﬁnal model prediction ˜
y and the target y .
Note that the both
P ( I|· ; θ ) and L (˜ y, y ; θ ) share some of parameters.
Consequently, we cannot separately learn both the functions.
Taking the gradient of the loss, we have:
[Equation]
In the second step, we used the product rule of differentiation and in last step, we have used the identity ∇ p ( I|· )
= p ( I|· ) · ∇ log p ( I|· ) .
This gives us two additive expectation terms under the distribution of P ( I|G , π I t =1: T ; θ ) to optimize in the loss function.
[Equation]
We optimize the expectations by the set of K samples of I drawn from the distribution of
P ( I|G , π I t =1: T ; θ ) .
A. P ROOF FOR T HEOREMS
Theorem 1.
Consider a n -vertex graph G with no distinct node attributes.
Assume we use universal multiset function approximators for target-cell selection and graph pooling function ψ , and a GNN with 1-WL equivalent expressive power for color reﬁnement, then the representation f ( G ) in Eqn.is permutation-invariant and can uniquely identify G .
Proof.
First, note that the set of leaves generated by the search tree of the exact graph-isomorphism solvers for a given n -vertex graph respects permutation-equivariance and contains the canonical representation as the largest element, hence uniquely identiﬁes the graph.
Please refer McKay &
Piperno (2014) for a detailed proof.
To complete the proof, observe that with sufﬁciently expressive networks, we can approximate the target-cell selector and individualization algorithms in the exact solver arbitrarily closely.
Using 1-WL equivalent GNN, we can also approximate the reﬁnement operation arbitrarily closely.
Repeated applying these operations give an equivalent set of leaves as those produced by the exact solver.
Finally, using a universal multiset approximator ψ in conjunction with an MLP ρ which is also universal approximator allows us to distinguish distinct sets of leaves from non-isomorphic graphs.
Theorem 2.
Assume we need D dimensional embeddings for some D > 0 , in order to uniquely represent any graph of n -vertices.
Consider two n -vertex graphs G 1 and G 2 whose universal representations in Eqn.
2 are f 1 and f 2 after T levels of the search tree.
Let the max-norm distance between f 1 and f 2 be at least d i.e. generating the full level- T search tree for both graphs will separate them by d .
Assume that the values in embeddings generated by the tree are strictly bounded in the range of [ − M, M ] .
Then, with probability of at least 1 − δ , the approximate embeddings, generated by sampling K paths from root to leaves, are separated in max-norm distance by d ± (cid:15) provided the number of root to leaf paths sampled are at least
[Equation]
Proof.
Observe that we independently sample K paths from root to leaf of the search tree.
After
T steps in the tree, if the mean of the embeddings is f T ( G ) =
| Π I | (cid:80) i ψ ( G , π Ti ) , then we have K independant samples to approximate the mean.
For clarity, we omit G and refer actual expectation as f T and sample approximation with f T .
Note that by the Hoeffding bound (Hoeffding, 1994; Shalev-Shwartz & Ben-David, 2014) for mean of K independent random variables Y 1 , . . .
, Y K in range [ − M, M ] with expected value µ , we have
[Equation]
Assume we need D dimensions for some D > 0 , in order to fully represent any n -vertex graph.
Since, we are independently sampling from the leaves of the tree, the Hoeffding bound holds for each of the estimated value in the D -dimensional vector.
Hence, for any i th value in the graph embedding,
[Equation]
[Equation]
Since this bound holds for every value of the embedding.
we can bound the maximum error for any of the D estimates with the “union bound” of probability for all the D values simultaneously.
[Equation] [Equation]
This gives us the max-norm distance between the estimated and the expected embeddings.
Using ζ = (cid:15)/ 2 , we get
[Equation] [Equation]
Now, assume two non-isomorphic graphs G 1 and G 2 , are separated in max-norm distance by d i.e.
(cid:107) f 1 T − f 2 T
[Equation]
=
d , at level T of the search tree.
With at least K samples, we can union bound the error in distance between the two graphs by (cid:15) . [Equation]
[Equation]
Bounding the failure probability with δ , we get the minimum samples needed for the approximation as:
[Equation]
A. PF-GNN A LGORITHM
1: Input: Graph G E ) ;
; P
· ; θ ) 2: Output:
, b
G ) 3:
Initialize H = I 4: b 1 ( G ) = (cid:104) H 1 = GNN ( H 0 ) , w k 1 = 1 /K (cid:105) k
=1: K (initial belief) 5: for t = 1 to T do 6: for k = 1 to K do 7: v ∼ P ( V| H kt ; θ ) 8:
˜ H kt = f ind ( H kt , v ) 9:
H kt +1 =
GNN ( ˜ H kt ) (transition update) 10: end for 11: (cid:104) ˜
w kt (cid:105) k =1: K = η ˙ (cid:104) f obs ( H kt +1 ; θ o ) · w kt (cid:105) k
=1: K (observation) 12: (cid:104) w kt {++
 (cid:105)+} k =1: K ∼ (cid:104) ˜ w kt
(cid:105) k
=1: K (resampling step) 13: end for 14: b T ( G ) = (cid:104) H T , w k T (cid:105) k
=1: K (ﬁnal belief) 15: ˜ f
( G ) =
CONCAT ( (cid:80) k
=1: K w kt H kt ) t =1: T 16: ˜
y = READOUT ( ˜ f
( G ))
17: Return ˜ y, b T ( G )
A. A DDITIONAL E XPERIMENTS
In this subsection, we provide further experiments in order to evaluate various components of the
PF-GNN algorithm
. A.5. I MPACT OF INDIVIDUALIZATION FUNCTION :
In PF-GNN, individualization method involves transforming the selected node embedding via an MLP.
In this ablation study, we seek to learn whether learning the individualization method is useful for the end performance.
For this, we replace the learnable MLP with non-learnable random features sampled from standard Normal distribution as used in Abboud et al.
and compare the performance on
TRIANGLES and CSL datasets.
Moreover, we compare both methods of individualization for various partial levels of node-feature perturbations as partial individualizations were noted to be helpful in
RNI (Abboud et al., 2020).
In partial RNI, only part of embedding dimensions are perturbed.
For example, 50% refers to 50% embedding dimensions of the selected node are perturbed in each IR step.
PF-GNN with T=3 and K=4 was used in the experiments.
We train all models with randomized individualization till convergence in this experiment.
[Table]
Results in Tab.
10 suggest that learnable MLP is better than random perturbations for node individualization.
For partial individualization, MLP performs similarly or better than random features.
For full individualization, the difference is less clear in CSL dataset as both methods give similar result.
However, very small levels of partial random perturbation seem to harm performance in both datasets.
A.5. I MPACT OF SEQUENTIAL INDIVIDUALIZATION :
We now study the impact of sequential individualization of PF-GNN.
In the ablation study of the main paper, we show in Tab. 9 that sequential individualization without learning signiﬁcantly reduces performance i.e. which nodes are selected in each IR step is important.
We further examine the effects of sequential individualization compared to simultaneous individualization.
For this, we compare following ablation models of PF-GNN: 1) PF-GNN with T=3 steps of IR, 2) PF-GNN-A:
PF-GNN with a single IR step but 3 randomly chosen nodes individualized in the same step.
PF-GNN-B: single IR step but 6 random nodes are individualized at once 4) PF-GNN C: single IR step with 9 nodes individualized.
Note that we individualize same number of nodes for PF-GNN method (sequential) and PF-GNN-A method (simultaneous).
We train all models till convergence in this experiment.
[Table]
Results in Tab.
11 show that sequentially individualizing selected nodes is better than individualizing a subset of nodes at once on both the datasets.
Since most nodes are indistinguishable initially, it is hard to learn the optimal subset in one step.
With sequential IR, the graph-coloring is reﬁned in each step and the model can learn to successively pick nodes which are likely to generate the best reﬁnement.
Therefore, “Sequential IR process with learning" helps in ﬁnding those subset of nodes.
Furthermore in CSL dataset, performance goes down as we increase the number of nodes individualized at once, suggesting that larger unguided randomness may not be helpful.
Overall, the results suggest that PF-GNN’s data-driven sequential reﬁnement improves GNN’s expressivity while adding minimal randomness.
A.5. R UNTIME ANALYSIS
In this subsection, we further study the running time behaviour of PF-GNN in contrast to the base
GNN models.
Assuming GNN takes linear time for bounded-degree graphs O ( n ) , runtime of PFGNN is O ( nT ) for T number of IR steps.
Note that the computation of K paths can be parallelized.
In Tab. 12, we provide additional empirical runtime results of PF-GNN models for different K and
T values for the Zinc and Alchemy datasets.
We also compare with the runtime of the standalone backbone GNN model.
The numbers in brackets indicate the ratio of the runtime of PF-GNN w.r.t that of backbone GNN.
For fair comparison, all models are trained on a single GPU with a batch size of 64 with PNA convolution as the base GNN.
The results indicate that runtime increase is only linear in T .
[Table]
Furthermore, PF-GNN is a ﬂexible model.
We can reduce the number of GNN iterations within each
IR step, if we want to keep the same time as the base GNN.
Even so, we can still get performance improvements with PF-GNN.
Below we report results of the standalone base GNN model withiterations and PF-GNN with T = 1 and T = 2 but 6 GNN message passing steps in total.
Results in Tab.
13 show that PF-GNN improves upon GNN with the same budget of total GNN layers.
Additionally, the results further empirically support that the improvements are a result of IR process and not merely addition of more GNN layers.
[Table]
A.5. G RAPH PROPERTY TESTING
In the main paper, we studied graph properties like triangle counting and local clustering co-efﬁcient and found the PF-GNN outperforms 1-WL equivalent GNNs.
In this subsection, we study another set of fundamental graph properties that GNNs cannot capture.
These properties are connectivity, bipartiteness and planarity (Kriege et al., 2018).
For all three tasks, we generate synthetic datasets of 100 graphs for training and 40 graphs for testing with equal number of positive and negative samples.
Each graph consists of 20 nodes each.
For connectivity dataset, we ﬁrst generate two random connected graphs which forms a single instance of non-connected graph and add an edge between the two to make it a positive sample of connected graph.
For planar datasets, we generate random regular graphs and check for planarity.
We repeat till we have equal number of planar and non-planar graphs.
For bipartiteness, we generate a random bipartite graph and create a positive sample by adding an edge between two groups of vertices and negative sample by adding an edge within a group of vertices.
Tab.
14 shows that GIN model fails on all three tasks while PF-GNN reaches near optimal accuracy on each one of them.
[Table]
A.5. H YPERPARAMETERS AND ARCHITECTURE DETAILS
In this subsection, we report the hyperparameters used in all experiments from Section 5 in Tab. 15.
The architecture of PF-GNN is speciﬁed by the backbone model, number of particles and number of IR steps.
We also report the hyperparameters used for training, including batch size, number of embedding dimensions and the weight for policy loss.
The learning rate is set at 1e-3 for SR25 and
EXP, and we use a scheduler to reduce it based on the improvement of metrics for other datasets.
[Table]
Baseline results: The baseline results mentioned in Tab. 1 are reproduced from the open sourced code from Balcilar et al
[Table]
(2021) and those in Tab. 2 are from Dwivedi et al.
We reproduced the results from Knyazev et al.
for Tab. 3.
For Tab.
[Table]
we reproduced results for PNA from the open sourced code and rest of the results are reported from Morris et al.
(2020) for k -GNNs and Beani et al.
(2021) for DGN.
For QM9 results in
Tab. [Table] we report results from Morris et al.
; Klicpera et al.
(2020) and from Brockschmidt (2020) for Tab. 6.
For Tab.
[Table]
we get the baseline results from the OGB leaderboard (Hu et al., 2020)
[Table]
A.5. A DDITIONAL E MPIRICAL R ESULTS
Two additional sets of experiments are done to complement our empirical evaluation.
In Abboud et al.
(2020), they use a different setting for EXP dataset as compared to our experiment in Tab. 1, which is 10-fold cross-validation.
We follow their setting and make further comparison with more expressive baselines in Tab. 16.
We also include one additional ablation study on number of IR steps and number of particles.
We test the performance for each architecture on the easier TRIANGLES-orig test set, where the number of nodes for each graph is similar to that of the graphs in training set.
As shown in Tab. 17, models with
L=1 get substantially less accuracy than the deeper ones.
Furthermore, increasing both K and T helps with the performance.
[Table] [Table]
This dataset (Mckay, 2021) contains 15 strongly regular graphs, where each of them has 25 nodes and each node has 12 neighbors.
5 common neighbors are shared by connected nodes while 6 common neighbor are shared by disconnected nodes.
There are in total 105 pairs of non-isomorphic graphs.
The task is to distinguish all pairs.
This is a challenging task because strongly regular graphs are known to be one of the hardest class of graphs for detecting isomorphism.
It is to be noted that SR dataset is used as one of the benchmarks for isomorphism detection task.
[Table]
This dataset (Abboud et al., 2020) consists of 600 pairs of non-isomorphic graphs.
Each graph in in
EXP dataset encodes a propositional formula.
Classifying the graph is equivalent to determining if the propositional formula is satisﬁable or not.
For both SR25 and EXP, we train the models with all graphs and test whether the models can learn embeddings that distinguish each pair.
Following, Balcilar et al.
we use 3-layer models for all the backbones in Tab. 3, and 8 IR steps with 4 particles for models with PF-GNN.
We run experiments with 10 different seed and report the average accuracy and standard deviation.
CSL:
This dataset from Murphy et al.
(2019) consists of 150 4-regular graphs with edges forming a cycle and contain skip links.
The 150 graphs are divided evenly into 10 isomorphism classes based on the skip length.
All graphs are 1-WL equivalent.
The task is to classify the graphs into isomorphism classes that can only be accomplished by models with higher expressive power than 1-WL.
As in Dwivedi et al.
; Murphy et al.
(2019), we follow 5-fold cross validation for evaluation.
PF-GNN results are obtained with 3 IR steps, 8 particles and GIN as backbone.
TRIANGLES:
The TRIANGLES dataset, released by (Knyazev et al., 2019), consists of 45000 graphs and the task is to count the number of triangles in each graph.
Interestingly, the dataset comes with two test splits; test-orig and test-large, both of which have 5000 graphs.
The training set as well as the test-orig set consists of graphs with < 25 nodes while the test-large set has < 100 nodes in each graph.
Better prediction in test-large suggests that the model is able to generalize better.
It is a graph classiﬁcation problem with 10 classes.
LCC:
LCC dataset (Sato et al., 2020) consists of 2000 random 3-regular graphs with 1000 graphs each for training and testing.
The nodes are labelled with their local clustering coefﬁcient (LCC) value.
LCC measures how close a node’s neighbours are to being a clique.
It is yet another property that 1-WL
GNNs fail to identify.
It is a node level classiﬁcation problem with 3 classes.
The dataset consists of two test sets N and X .
The training set and test- N set have graphs with 20 nodes and test- X has graphs with 100 nodes.
A good performance on LCC-X further shows the generalization ability of the model to larger graphs.
ZINC:
ZINC is a real-world molecular dataset consisting of more than 250K graphs.
We select the popularly used subset made available by Dwivedi et al.
(2020) which contains 12000 graphs.
The dataset comes with standard split of 10000 graphs for training set and 1000 graphs each for testing and validation sets.
It contains graphs along with their constrained solubility values.
The task is regression on each graph to predict the constrained solubility values.
For evaluation, we measure the mean absolute error on the predicted values.
ALCHEMY:
The Alchemy dataset Chen et al.
(2019a) consists of nearly 200K graphs along with their 12 quantum mechanical properties.
We chose the subset provided by Morris et al.
The dataset comes with 12000 graphs.
The test set and the validation set contain 1000 graphs each.
The task is to regress on the 12 quantum mechanical properties and evaluate on mean absolute error.
QM
QM9 Ramakrishnan et al.
(2014) is a large scale molecular dataset consisting of nearlygraphs.
Each graph consists of a set of node features associated with atoms along with bonds encoded as edge features.
Additionally, we are provided with 3D node position coordinates for each atom.
The task is graph regression on a set of 12 quantum-chemical properties associated with each graph.
We follow the convention of randomly splitting the dataset with 10000 graphs each in testing and validation set.
The rest of the graphs are used for training.
Interestingly, the node position coordinates have been shown to be very useful in Klicpera et al.
However, we conduct two separate experiments, one with node position coordinates and other without the position features.
OGBG-MOLHIV
Ogbg-Molhiv dataset consists of 41127 molecular graphs.
The task is binary classiﬁcation for molecular property prediction.
We use the full 9-dimensional node features, and adopt the standard scaffold split provided by Hu et al.
A. E XTENDED PRELIMINARIES
In this subsection, we provide an extended discussion on the graph coloring process and the search tree which PF-GNN is designed to approximate.
For a detailed explanation, please refer to McKay & Piperno (2014).
Below ﬁgures are adapted from McKay & Piperno (2021), a visual guide to the practical graph isomorphism solvers from the authors of Nauty and Traces.
Coloring and 1-WL reﬁnement:
Graph Neural Networks have been shown to be neural analogues of vertex color reﬁnement technique with neural embeddings being equivalent to the colors.
Graph coloring is technique of assigning graph vertices with unique numbers in a permutation invariant manner; thereby ordering the vertices.
1-WL algorithm is one way coloring vertices and is also called naive-vertex reﬁnement.
In 1-WL
reﬁnement, initially all nodes are assigned with the same color.
In each iteration, 1-WL reassigns colors of each node based on its neighbourhood.
As long as two vertices have different multiset of colors in their neighbourhood, they will be assigned different colors.
However, after some iterations, the colors stop changing.
At this point, the coloring is called equitable coloring .
Equitable coloring
An equitable coloring is one where every two vertices of the same colour are adjacent to the same number of vertices of each colour.
Fig.
4 shows some of the example equitable colorings.
The colorings divides the set of vertices into disjoint sets based on the node colors.
Hence, equitable colorings are also loosely called equitable partitions.
An equitable coloring where each node has unique color is call discrete coloring .
Identifying graphs:
In order to uniquely identify graphs, we need to color every vertex uniquely in a permutation invariant manner.
For this, we need to generate discrete coloring in a permutation invariant way.
Due to symmetry in the graph structures, equitable colorings do not allow 1-WL process to further reﬁne the colorings.
Therefore, GNNs are also limited in their expressive power to distinguish all non-isomorphic graphs.
This bottleneck can be broken with a technique called individualization and reﬁnement .
Individualization and reﬁnement:
Individualization and reﬁnement is a technique of breaking the symmetry in the equitable colorings by recoloring a node with a unique color.
Once, a node is distinguished from rest of the vertices, this information can be propagated the 1-WL/GNN message passing.
Fig.
5 shows an example of the
IR process.
Initially a vertex is individualized by coloring it Green.
Then in the reﬁnement stage, its neighbours will be distinguished from other vertices.
Thereafter, rest of the vertices are similarly distinguished iteratively till a ﬁner equitable coloring is reached.
equitable coloring of the graph.
Search tree
The IR process is not a permutation invariant operation as the reﬁned coloring depends on the individualized node in the graph.
Therefore, in order to preserve permutation invariance, we need to repeat the IR process for all the nodes which have the same color i.e. , belonging to the same color cell in the equitable coloring.
This gives rise to a search-tree where the equitable colorings form tree-nodes.
The search tree is a unique representation of the isomorphism class of the input of graph i.e. , all isomorphic graphs generate the same search tree and non-isomorphic graphs generate different search trees.
Fig.
2 shows the example of one such search tree.