K-Plex Cover Pooling for Graph Neural Networks
Abstract
We introduce a novel pooling technique which borrows from classical results in graph theory that is non-parametric and generalizes well to graphs of different nature and connectivity pattern.
Our pooling method, named KP LEX P OOL , builds on the concepts of graph covers and k -plexes, i e. pseudo-cliques where each node can miss up to k links.
The experimental evaluation on molecular and social graph classiﬁcation shows that KP LEX P OOL achieves state-of-the-art performances, supporting the intuition that well-founded graph-theoretic approaches can be effectively integrated in learning models for graphs
Introduction and Related Works
Graph neural networks allow for the adaptive processing of topology-varying structures representing complex data which comprises atomic information entities (the nodes) and their relationships (the edges).
The neural processing of graphs typically leverages message passing between neighboring nodes [2] to collect and exchange information on the context of nodes.
Such process is made effective and efﬁcient, also on cyclic structures, by feedforward neural layers with node-level weight-sharing, an approach popularized under the term graph convolutions [14], but previously known as contextual structure processing [18, 1].
Graph pooling methods provide mechanisms for structure reduction that are intended to ease the diffusion of such a context between nodes farther in the graph.
These methods realize structure reduction layers that are interleaved between graph convolutions to provide a multi-resolution view of the input graph.
This is intended to extract coarser and more abstract representations of the graph as we go deeper in the network.
The deﬁnition of a robust, general, and efﬁcient graph pooling mechanism is made difﬁcult by the irregular nature of the data and by the lack of a reference ordering of nodes between samples.
Approaches in literature are addressing the problem from a (more or less explicit) community discovery perspective by considering node connectivity patterns (topological approaches) [22, 17] or by aggregating the nodes based on their similarity in the neural embedding space (adaptive approaches)
Our work proposes a novel link between pooling operators and two consolidated concepts in graph theory and combinatorial algorithms, namely, k -plexes and graph covers.
The former provides a ﬂexible formalization for a community of nodes as a densely interconnected and cohesive subgraph, which relaxes the deﬁnition of a clique by allowing nodes to miss up to k links each.
The latter relates to a soft-partition of nodes whose union covers all nodes on the original graph.
We argue that both concepts are necessary to realize an effective and general graph pooling mechanism as they permit to summarize the overall community structure by taking a small but meaningful set of highly connected components, represented by the k -plexes.
We introduce KP LEX P OOL , a novel pooling method using only topological graph features, which is not parameterized nor its outcomes depend on the speciﬁc predictive task.
We show how
KP LEX P OOL can be effectively integrated into deep graph networks, with excellent performances that generalize well to structures of different nature and topology.
Our contribution is not just a straightforward application of k -plexes to deep learning for graphs.
Rather, KP LEX P OOL is the result of the careful design and integration of different graph enumeration and simpliﬁcation mechanisms speciﬁcally crafted to obtain a hierarchical structure coarsening algorithm well suited to promote effective context diffusion in neural message passing.
This goal is quite challenging as, for instance, a previous attempt by [16] clearly shows that the straightforward application of clique discovery does not yield an effective graph pooling method.
We hope that our work can further stimulate the community interests at the cross-roads between graph algorithmics and machine learning, as for the
Weisfeiler-Lehman graph kernel case [21].
Overall, our contribution is novel in: i) the use of k -plex communities to deﬁne pooling mechanisms in neural processing systems; ii) the deﬁnition of the notion of k -plex cover; iii) the deﬁnition of an efﬁcient algorithm to implement the concepts above.
K-Plex Cover Graph Pooling
Preliminaries Given an undirected graph G , let V =
V ( G ) be its node set and E
= E ( G ) be its edge set, where v ( G ) =
| V | = n and e ( G ) =
| E | = m are, respectively, the number of nodes and edges in G .
Given an edge e = { u, v } , nodes u and v are said to be adjacent or neighboring each other.
The neighborhood N ( v ) of v is the set of nodes adjacent to it, and the degree d ( v ) of v is deﬁned as the number of its neighbors, i.e. | N ( v ) | .
An attributed graph is a tuple ( G, φ, ψ ) where φ : V → R h V and ψ : V × V → R h E are functions that assign a vector of features to each node and to each edge, respectively of size h V and h E .
If e / ∈ E , then ψ ( e ) = 0 .
A k -plex is a subset of nodes S ⊆ V such that each node in S has at least | S | − k
adjacent nodes in
S : for all v ∈ S , we have | N ( v ) ∩ S | ≥ | S | − k .
This deﬁnition is quite ﬂexible as for k =
we get the classical clique and for larger values of k we obtain a relaxed and broader family of (possibly larger) subgraphs of G .
A k -plex cover of G is a family of subsets S of V such that each set S ∈ S is a k -plex and their union is ∪ S ∈S S = V .
Graph pooling with k -plexes
KP LEX P OOL computes a k -plex cover S =
(cid:8) S 1 , . . .
, S c (cid:9) of the input graph ( G, φ, ψ ) , for a given k , and returns a coarsened graph ( G (cid:48) , φ (cid:48) , ψ (cid:48) ) , such that
V (cid:48)
=
V ( G (cid:48) ) =
(cid:8) v (cid:48)
[Equation]
. . . , v (cid:48) c [Equation] , and
[Equation]
where E (cid:0) G [ S i , S j ] (cid:1) =
E ( G ) ∩ ( S
i × S j ) .
Node v (cid:48)
i represents the coarsened version of S i , and edge { v (cid:48)
i , v (cid:48)
j (cid:9) exists iff there is at least one edge in G linking a node of S i with a node of S j .
The feature functions φ (cid:48) : V (cid:48)
→ R h V and ψ (cid:48) : V (cid:48)
× V (cid:48)
→ R h E aggregate, respectively, features belonging to the same k -plex S i and features of edges linking two different S i and S j .
In other words, they are deﬁned in such a way to provide a suitable relabeling for nodes and edges in the coarsened graph: φ
[Equation]
and
[Equation]
where β and γ are arbitrary aggregation functions deﬁned over multisets of feature vectors [26].
Differently from other partitioning-based graph coarsening methods [7, 19, 17, 24], in our approach a node may belong to multiple k -plexes.
This is also a key difference between C LIQUE P OOL [16] and
KP LEX P OOL with k = 1 (i.e., performing a clique cover ), where the former model forces a partition between nodes potentially destroying structural relationships in the communities. k -Plex cover algorithm
We propose an algorithm, whose pseudocode is shown in Algorithmsand 2, that ﬁnds a cover containing large k -plexes that have small intersection.
The rationale for this choice is driven by the sought-after effect on graph pooling mechanisms in graph neural networks.
On one hand, we seek to condense into a single community-node those neighboring nodes which are likely to share the same context and, hence, very similar embeddings.
On the other hand, we would like the pooled graph to preserve diversity for nodes belonging to different communities, i.e. avoiding trivial aggregations which would induce heavy connectivity between the communities.
Our algorithm is inspired to the clique covering framework in [5, 6], and leverages on heuristics that speciﬁes the order on which nodes are considered for k -plex inclusion.
Algorithm 1 receives in input this order by means of two priority functions f, g on V that are deﬁned to provide large k -plexes with small pair-wise intersections.
In practice, we ﬁxed f, g to prioritise nodes with lower degree (for f ) and more neighbors in the k -plex (for g ).
Algorithm 1 begins by iterating over the available nodes in the candidate set U , which is initialized with the whole set of nodes of the input graph.
At each iteration, it selects the next candidate v by extracting the node in U with higher priority f ( v ) .
The node v will be then used as a starting node for retrieving the next k -plex S ⊆ V by Algorithm 2 ( S will eventually include v ).
The output
1: S ← ∅ 2: U ← V
G ) 3: while U = ∅ do 4: v ← argmax u ∈ U f u ) 5: S ← F IND KP LEX ( G k g v ) 6:
S ← S
(cid:8) S (cid:9) ; U
U \ S 7:
Suitably priority f .
input A graph G , an integer k ≥ 1 , a priority function g , and a pivot node v .
output A k -plex S , s.t. v ∈ S .
1: S ← { v } ; C ← N ( v ) 2: while C (cid:54) = ∅ do 3: u ← argmax w ∈ C g ( w ) 4:
S ← S ∪ { u } ; C ← C \ { u } 5: for w ∈ S do 6: if | S \ N ( w ) | =
k then C ← C ∩ N ( w ) 7: for w ∈ C do 8: if | S \ N ( w ) | =
k then C ← C \ { w } 9: for w ∈ N ( u ) do 10: if | S \ N ( w ) | <
k then C ← C ∪ { w } 11:
Suitably update priority g .
elements of S will then be removed from the set U of candidates, and S will be included in the output cover S .
Note that the nodes in the k -plexes are removed from U but not from the graph, hence a successive execution of Algorithm 2 may contain previously removed nodes.
The algorithm stops when eventually all the nodes are removed from U , and then returns cover S .
Algorithm 2, instead, constructs a k -plex S starting from a given node v , which is the only element available at startup.
It initializes a candidate set C of nodes that could be part of the k -plex, this time relying on N ( v ) .
Again, Algorithm 2 iterates over the nodes in C following the ordering deﬁned by the priority g , and adds them to S .
The main loop has the following invariants
[Equation]
where Equation (1) states that every node u in S needs to have at least | S | − k adjacent nodes in S and Equation (2) states that, when u has exactly | S | − k adjacent nodes in S , we cannot have nodes in C that are not adjacent to u (as it would break Equation (1) if selected).
As a result, any node from C can be added to S .
Both invariants are satisﬁed at the ﬁrst iteration as all the candidates in
C = N ( v ) are adjacent to v , which is the only element in S .
At each iteration, Algorithm 2 preserves
Equation (1) by selecting a node u from C according to priority g .
It needs to preserve Equation (2) as S changes because of the addition of u .
The ﬁrst two for loops remove the nodes from C that no longer can be added to S .
The third loop adds to C the nodes adjacent to u which can be added to S .
After that, g is updated.
Computational cost It can be easily shown that Algorithm 2 takes O ( m ) time, since we can efﬁciently store and update g with standard data structures.
Indeed, as each node v is added to
S or C and removed from C at most once, and each update costs O ( d ( v )) , the amortized cost is
O ( (cid:80) v d ( v ))
=
O ( m ) time.
The time cost of Algorithm 1 is bounded by O ( mn ) , as it performs
O ( n ) updates of S , U , and f , and O ( n ) calls to Algorithm 2, which requires O ( m ) time.
Experiments and Discussion
Experimental setting We tested KP LEX P OOL against related methods from literature on four molecular graph datasets, namely DD [8], NCI-1 [23], ENZYMES [20], and PROTEINS [4], and ﬁve social network datasets, COLLAB, IMDB-BINARY, IMDB-MULTI, REDDIT-BINARY, and
REDDIT-MULTI-5K [27].
All datasets have been retrieved from the TU-Dortmund collection [13].
For fairness, each pooling method has been tested by plugging it into the same standardized architecture (B ASELINE ), comprising (cid:96) ∈ { 2 , 3 } convolutional blocks, followed by two dense layers, the latter interleaved by dropout with probability 0 .
Every convolutional block is formed by two GNN layers (GCN [14] or G RAPH S AGE [12]) with Jumping Knowledge [25, CONCAT variant] followed by a dense layer.
After every convolutional block we have a global sum-pooling, and the concatenation of their resulting vector is batch-normalized and fed to the ﬁnal dense block.
Every layer, GNN or dense, has h ∈ { 64 , 128 } units and a ReLU activation function [11].
For every other model, a pooling
[Table] [Table]
layer is placed after the ﬁrst (cid:96) − 1 convolutional blocks and its output feed to the next block.
For
KP LEX P OOL we set k ∈ { 1 , 2 , 4 , 8 } , while we used r ∈ { 1 / 4 , 1 / 2 , 3 / 4 } as node-reduction factor for T OP K-, SAG-, and D IFF P OOL .
For KP LEX P OOL and G RACLUS we used both sum and max (concatenated) as node aggregation ( β ) and sum for edge aggregation ( γ ).
All models have been implemented using PyG [10].
Our experimental approach followed the standardized reproducible setting in [9] (stratiﬁed 10-fold with an inner grid-search on a 10% validation split per fold).
Discussion Tables 1 and 2 show the mean accuracy and standard deviation on test data (outer fold).
On chemical datasets, KP LEX P OOL yields competitive results on all datasets, with higher performances than other related methods on all benchmarks but ENZYMES.
On social benchmarks,
KP LEX P OOL performs better than parametric pooling models on all datasets, when considering the same experimental conditions.
D IFF P OOL is out-of-resources on REDDIT data, but KP LEX P OOL is still competitive also with respect to D IFF P OOL results from [9].
On REDDIT-5K, only the topological pooling of G RACLUS yields to higher accuracy.
Its community-seeking bias seems certainly very adequate for the processing of social graphs, where adaptive pooling methods do not seem capable of leveraging their parameters to produce more informative graph reductions.
Perhaps surprisingly, KP LEX P OOL achieves excellent performances also on molecular data, where we would have expected adaptive models to have an edge, conﬁrming our initial intuition about the ﬂexibility and generality of k -plex cover communities.
These results can be well appreciated in the last column of Tables 1 and 2, where we report the average rank of each model separately on chemical and social datasets.
We also report that KP LEX P OOL has the best overall average rank at 2 .
33 , followed by
G RACLUS on par with D IFF P OOL at 2 . 78 .
KP LEX P OOL has state-of-the-art performances in 5 out of 9 graph classiﬁcation benchmarks.
KP LEXP OOL is shown to be the best performing method, on average, when confronted with related pooling mechanisms from literature.
It does so through a fully topological approach that does not leverage task information for community building.
None of the related models, including the adaptive ones, seem to have the same ability to cope effectively with structures of radically different nature (molecules and social networks).
Apart from predictive performance, KP LEX P OOL has a very practical advantage in terms of computational cost, when compared to adaptive models such as D IFF P OOL .
For instance, its graph reduction can be pre-computed once for the whole dataset and re-used throughout the whole model selection and validation phase, as it does not depend on adaptive node embedding.
This aspect is clear from the empirical results, in which D IFF P OOL is shown to fail to complete training within the 2 weeks limit (or to exceed the available GPU memory) on datasets comprising larger graphs.