High-Order Pooling for Graph Neural Networks with Tensor Decomposition
Abstract
Graph Neural Networks (GNNs) are attracting growing attention due to their effectiveness and ﬂexibility in modeling a variety of graph-structured data.
Exiting GNN architectures usually adopt simple pooling operations ( e.g., sum, average, max) when aggregating messages from a local neighborhood for updating node representation or pooling node representations from the entire graph to compute the graph representation.
Though simple and effective, these linear operations do not model high-order non-linear interactions among nodes.
We propose the Tensorized Graph
Neural Network (tGNN), a highly expressive GNN architecture relying on tensor decomposition to model high-order non-linear node interactions.
tGNN leverages the symmetric CP decomposition to efﬁciently parameterize permutation-invariant multilinear maps for modeling node interactions.
Theoretical and empirical analysis on both node and graph classiﬁcation tasks show the superiority of tGNN over competitive baselines.
In particular, tGNN achieves the most solid results on two OGB node classiﬁcation datasets and one OGB graph classiﬁcation dataset.
Introduction
Graph neural networks (GNNs) generalize traditional neural network architectures for data in the
Euclidean domain to data in non-Euclidean domains [26, 45, 34, 35].
As graphs are very general and ﬂexible data structures and are ubiquitous in the real world, GNNs are now widely used in a variety of domains and applications such as social network analysis [20], recommender systems [49], graph reasoning [55] and drug discovery [42].
Indeed, many GNN architectures ( e.g., GCN [26], GAT [45], MPNN [16]) have been proposed.
The essential idea of all these architectures is to iteratively update node representations by aggregating the information from their neighbors through multiple rounds of neural message passing.
The ﬁnal node representations can be used for downstream tasks such as node classiﬁcation or link prediction.
For graph classiﬁcation, an additional readout layer is used to combine all the node representations to calculate the entire graph representation.
In general, an effective aggregation (or pooling) function is required to aggregate the information at the level of both local neighborhoods and the entire graph.
In practice, some simple aggregation functions are usually used such as sum, mean, and max.
Though simple and effective in some applications, the expressiveness of these functions is limited as they only model linear combinations of node features, which can limit their effectiveness in some cases.
A recent work, principled neighborhood aggregation (PNA) [13], aims to design a more ﬂexible aggregation function by combining multiple simple aggregation functions, each of which is associated with a learnable weight.
However, the practical capacity of PNA is still limited by simply combining multiple simple aggregation functions.
A more expressive solution would be to model high-order non-linear interactions when aggregating node features.
However, explicitly modeling high-order non-linear interactions among nodes is very expensive, with both the time and memory complexity being exponential in the size of the neighborhood.
This raises the question of whether there exists an aggregation function which can model high-order non-linear interactions among nodes while remaining computationally efﬁcient.
In this paper, we propose such an approach based on symmetric tensor decomposition.
We design an aggregation function over a set of node representations for graph neural networks, which is permutation-invariant and is capable of modeling non-linear high-order multiplicative interactions among nodes.
We leverage the symmetric CANDECOMP/PARAFAC decomposition (CP) [22, 29] to design an efﬁcient parameterization of permutation-invariant multilinear maps over a set of node representations.
Theoretically, we show that the CP layer can compute any permutation-invariant multilinear polynomial, including the classical sum and mean aggregation functions.
We also show that the CP layer is universally strictly more expressive than sum and mean pooling: with probability one, any function computed by a random CP layer cannot be computed using sum and mean pooling.
We propose the CP-layer as an expressive mean of performing the aggregation and update functions in GNN.
We call the resulting model a tensorized GNN (tGNN).
We evaluate tGNN on both node and graph classiﬁcation tasks.
Experimental results on real-world large-scale datasets show that our proposed architecture outperforms or can compete with existing state-of-the-art approaches and traditional pooling techniques.
Notably, our proposed method is more effective and expressive than existing GNN architectures and pooling methods on two citation networks, two OGB node datasets, and one OGB graph dataset.
Summary of Contributions We propose a new aggregation layer, the CP layer, for pooling and readout functions in GNNs.
This new layer leverages the symmetric CP decomposition to efﬁciently parameterize polynomial maps, thus taking into account high-order multiplicative interactions between node features.
We theoretically show that the CP layer can compute any permutation-invariant multilinear polynomial including sum and mean pooling.
Using the CP layer as a drop-in replacement for sum pooling in classical GNN architectures, our approach achieves more effective and expressive results than existing GNN architectures and pooling methods on several benchmark graph datasets.
Preliminaries 2.1 Notation
We use bold font letters for vectors ( e.g., v ), capital letters ( e.g., M , T ) for matrices and tensors respectively, and regular letters for nodes ( e.g., v ).
Let G =
( V, E ) be a graph, where V is the node set and E is the edge set with self-loop.
We use N ( v ) to denote the neighborhood set of node v , i.e.,
N ( v ) =
{ u : e vu 2 E } .
A node feature is a vector x 2 R F deﬁned on V , where x v is deﬁned on the node v .
We use ⌦ to denote the Kronecker product,  to denote the outer product, and  to denote the Hadamard ( i.e., component-wise) product between vectors, matrices, and tensors.
For any integer k , we use the notation [ k ] = { 1 , · · · , k } .
2.2 Tensors
We introduce basic notions of tensor algebra, more details can be found in [29].
A k -th order tensor
T 2 R N 1 ⇥ N 2 ⇥ ... ⇥
N k can simply be seen as a multidimensional array.
The mode- i ﬁbers of T are the vectors obtained by ﬁxing all indices except the i -th one: T n 1 ,n 2 ,...,n i  1 , : ,n i +1 ,...,n k 2 R N i .
The i -th mode matricization of a tensor is the matrix having its mode- i ﬁbers as columns and is denoted by T ( i ) , e.g., T (1) 2 R N 1 ⇥ N 2 ··· N k .
We use T ⇥ i v 2 R N 1 ⇥ ··· ⇥ N i  1 ⇥ N i +1 ⇥ ··· ⇥ N k to denote the mode- i product between a tensor T 2 R N 1 ⇥ ··· ⇥ N k and a vector v 2 R N i , which is deﬁned by ( T ⇥ i v ) n 1 ,...,n i  [Equation] n [Equation]
[Equation]
n
[Equation]
P [Equation] i n i
[Equation]
n [Equation] [Equation] k
[Equation]
n i . The following useful identity relates the mode- i product with the Kronecker product:
[Equation]
2.3 CANDECOMP/PARAFAC Decomposition
We refer to C ANDECOMP/ P ARAFAC decomposition of a tensor as CP decomposition [24, 22].
A Rank R CP decomposition factorizes a k  th order tensor T 2 R N 1 ⇥ ... ⇥ N k into the sum of R one tensors as
= P
Rr =1 v 1 r  v 2 r  · · ·  v kr 1 r 2 R N 1 , v 2 r 2 R N 2 , ..., v kr 2 R N
The decomposition vectors, v : r for r = 1 , ..., R , are equal in length, thus can be naturally gathered into factor matrices M 1 =
[ v 11 , ..., v 1 R ] 2 R N 1 ⇥ R , ..., M k
= [ v k 1 , ..., v kR ] 2 R N k ⇥ R .
Using the factor matrices, we denote the CP decomposition of T as
[Equation]
The k -th order tensor T is cubical if all its modes have the same size, i.e., N 1 = N
= ...
= N k := N .
A tensor T is symmetric if it is cubical and is invariant under permutation of its indices:
[Equation]
for any permutation  : [ k ] !
[ k ] .
A rank R symmetric CP decomposition of a symmetric tensor
T is a decomposition of the form T =
[[ M , · · · , M ]] with M 2 R N ⇥ R .
It is well known that any symmetric tensor admits a symmetric CP decomposition [12], we illustrate a rank R symmetric CP decomposition in Fig. 1.
We say that a tensor T is partially symmetric if it is symmetric in a subset of its modes [28].
For example, a 3 -rd order tensor T 2 R N 1 ⇥ N 1 ⇥ N 3 is partially symmetric w.r.t.
modes 1 and 2 if it has symmetric frontal slices; i.e., T : , : ,k is a symmetric matrix for all k 2 [ N 3 ] .
We prove the fact that any partially symmetric tensor admits a partially symmetric CP decomposition in Lemmain Appendix A, e.g., if T 2 R N 1 ⇥ N 1 ⇥ N 3 is partially symmetric w.r.t.
modes 1 and 2, there exist
M 2 R N 1 ⇥ R and W 2 R N 3 ⇥ R such that T =
[[ M , M , W ]] .
2.4 Graph Neural Networks and Pooling Functions
Given a graph G =
( V, E ) , a graph neural network always aggregates information in a neighborhood to give node-level representations.
During each message-passing iteration, the embedding h v corresponding to node v 2 V is generated by aggregating features from N ( v ) [18].
Formally, at the l -th layer of a graph neural network,
[Equation]
where AGGREGATE ( l ) ( · ) and UPDATE ( l ) ( · ) are differentiable functions, the former being permutation-invariant.
In words, AGGREGATE ( l ) ( · ) ﬁrst aggregates information from N ( v ) , then
UPDATE ( l ) ( · ) combines the aggregated message and previous node embedding h ( l  1) v to give a new embedding.
The node representation h ( L ) v of node v from the last GNN layer L can be used for predicting relevant properties of the node.
For graph classiﬁcation, an additional READOUT ( · ) function aggregates node representations from the ﬁnal layer to obtain a graph representation h G of graph G as,
[Equation]
where READOUT ( · ) can be a simple permutation-invariant function ( e.g., sum, mean, etc. ) or a more sophisticated graph pooling function [50, 48].
In the general design of GNNs, it is important to have an effective pooling function to aggregate messages from local neighborhoods and update node representations (see Eq.
equation 2), and to combine node representations to compute a representation at the graph level (see Eq.
equation 3).
In our paper, high-order refers to multi-dimensional feature products between nodes in a neighborhood.
[38, 43] are previous works on high-dimensional feature products, and introduce how to use tensor methods to achieve those products.
They have stated the importance of having high-order products in physics and geometries.
For example, if we have x 1 and x 2 , a simple GNN layer can result in a vector of [ x 1 , x 2 , x 1 + x 2 ] , but our high-order layer result in a vector of [ x 1 , x 2 , x 1 + x 2 , x 1 x 2 ] with another term x 1 x 2 .
And if we have we have x 1 , x 2 , and x 3 , a simple low-order GNN layer will result in [ x 1 , x 2 , x 3 , x 1 + x 2 , x 1 + x 3 , x 2 + x 3 , x 1 + x 2 + x 3 ] while the high-order layer will result in a vector of [ x 1 , x 2 , x 3 ,
x 1 + x 2 , x 1 + x 3 , x 2 + x 3 , x 1 + x 2 + x 3 , x 1 x 2 , x 1 x 3 , x 2 x 3 , x 1 x 2 x 3 ] with extra 4 terms at a higher feature dimension.
In general content of GNNs, high-order will normally refer to the ability of capturing long-range dependencies, the interactions between long-range nodes [48, 2, 46], but ours refers to the multi-dimensional node feature products.
However, the CP layer is able to capture stacking long-range dependencies by stacking layers like other simple GNNs [26, 45].
One may raise a question on why we need to compute multi-dimensional feature products.
The idea is fairly easy if one thinks the multi-dimensional feature space.
Take a molecular graph dataset as an example, and blood pressure could be one feature.
Many studies analyze the impact of blood pressure on one’s health.
Blood pressure is the feature in this case.
So, one could try to predict life expectancy using blood pressure as a feature, but you can imagine that this might not be sufﬁcient.
You really need more information.
So you could add age, exercise frequency, and body fat to see if you can predict a person’s life expectancy.
Those additional measurements are also features and now every individual has a multi-dimensional feature vector consisting of these measurements.
However, a simple GNN with one-dimensional feature interactions may not capture the correlation between age, exercise frequency, and body fat.
There could be some correlation between the three factors to predict life expectancy.
A simple GNN that does age+exercise frequency+body fat is not able to capture the correlation between those factors.
Still, our multi-dimensional feature product can capture age ⇥ exercise frequency, age ⇥ body fat, exercise frequency ⇥ body fat, age ⇥ exercise frequency ⇥ body fat, all the correlations to predict life expectancy.
So in molecular graph case, high-dimensional feature products are importantly needed to make a prediction as most features are correlated (like in atom graphs, atom charge, and atom mass), and our model shows some signiﬁcant improvements on molecular datasets in Sec.6.
And the high-dimensional feature products might also be important for other networks and applications [38, 43].
Tensorized Graph Neural Network
In this section, we introduce the CP-layer and tensorized GNNs (tGNN).
For convenience, we let { x 1 , x 2 , ..., x k } denote features of a node v and its 1 -hop neighbors N ( v ) such that |{ v } [ N ( v ) |
=
k .
4.1 Motivation and Method
We leverage the symmetric CP decomposition to design an efﬁcient parameterization of permutationinvariant multilinear maps for aggregation operations in graph neural networks, Tensorized Graph
Neural Network (tGNN), resulting in a more expressive high-order node interaction scheme.
We visualize the CP pooling layer and compare it with sum pooling in Fig.
Let T 2 R N ⇥ N ⇥ ··· ⇥ N ⇥ M of order k + 1 be a tensor which is partially symmetric w.r.t.
its ﬁrst k modes.
We can parameterize T using a rank R partially symmetric CP decomposition (see
Section 2.3): T =
[[ W , · · · , W , M ]] where W 2 R N ⇥ R and M 2 R M ⇥ R .
Such a tensor naturally deﬁnes a map from ( R N ) k to R M using contractions over the ﬁrst k modes:
[Equation]
This map satisﬁes two very important properties for GNNs: it is permutation-invariant (due to the partial symmetry of T ) and its number of parameters is independent of k (due to the partially symmetric CP parameterization).
Thus, using only two parameter matrices of ﬁxed size, the map in
Eq.
equation 4 can be applied to sets of N -dimensional vectors of arbitrary cardinality.
In particular, we will show that it can be leveraged to replace both the AGGREGATE and UPDATE functions in GNNs.
CP Layer
W
Input Features x y z
W W
W
Product Pooling
Layer x’ y’ z’
W x’
W y’
W z’
Activation
M
Linear Transformation via M
Linear Transformation via W =
Output Representation of 27 Elements
If Linear Activation = = 4-th Order Symmetric Tensor T } y } z } x
There are several way to interpret the map in Eq.
equation 4.
First, from Eq.
equation 1 we have
[Equation]
where T ( k +1) 2 R M ⇥ N k is the mode- ( k + 1) matricization of T .
This shows that each element of the output f ( x 1 , · · · , x k ) is a linear combinations of terms of the form ( x 1 ) i 1 ( x 2 ) i 2 · · · ( x k )
i k
( k -th order multiplicative interactions between the components of the vectors x 1 , · · · , x k ).
That is, f is a multivariate polynomial map of order k involving only k -th order interactions.
By using homogeneous coordinates, i.e., appending an entry equal to one to each of the input tensors x i , the map f becomes a more general polynomial map taking into account all multiplicative interactions between the x i up to the k -th order:
[Equation]
where T is now of size ( N +1) ⇥ · · · ⇥ ( N +1) ⇥
M and can still be parameterized using the partially symmetric CP decomposition T =
[[ W , · · · , W , M ]] with W 2 R ( N +1) ⇥ R and M 2 R M ⇥ R .
With this parameterization, one can check that
[Equation]
where  denotes the component-wise product between vectors.
The map f can thus be seen as the composition of a linear layer with weight W , a multiplicative pooling layer, and another linear map
M .
Since it is permutation-invariant and can be applied to any number of input vectors, this map can be used as both the aggregation, update, and readout functions of a GNN using non-linear activation functions, which leads us to introduce the novel CP layer for GNN.
Deﬁnition 1.
(CP layer)
Given parameter matrices M 2 R d ⇥ R and W 2 R F +1 ⇥ R and activation functions  ,  0 , a rank R CP layer computes the function f CP : [ i  1 ( R F ) i ! R d deﬁned by
[Equation]
for any k  1 and any x 1 , · · · , x k 2 R F .
The rank R of a CP layer is a hyperparameter controlling the trade-off between parameter efﬁciency and expressiveness.
Note that the CP layer computes AGGREGATE and UPDATE (see Eq. 2) in one step.
One can think of the component-wise product of the W > [ x i 1] > as AGGREGATE, while the UPDATE corresponds to the two non-linear activation functions and linear transformation M .
We observed in our experiments that the non-linearity  is crucial to avoid numerical instabilities during training caused by repeated products of W .
In practice, we use Tanh for  and ReLU for  0 .
Fig.
2 graphically explains the computational process of a CP layer, comparing it with a classical sum pooling operation.
We intuitively see in this ﬁgure that the CP layer is able to capture high order multiplicative interactions that are not modeled by simple aggregation functions such as the sum or the mean.
In the next section, we theoretically formalize this intuition.
Complexity Analysis
The sum, mean and max poolings result in O ( F in ( N + F out )) time complexity, while CP pooling is O ( R ( NF in + F out ) ) , where N denotes the number of nodes, F in is the input feature dimension, F out is out feature dimension, and R is the CP decomposition rank.
In Sec.
6.3, we experimentally compare tGNN and CP pooling with various GNNs and pooling techniques to show the model efﬁciency with limited computation and time budgets.
4.2 Theoretical Analysis
We now analyze the expressive power of CP layers.
In order to characterize the set of functions that can be computed by CP layers, we ﬁrst introduce the notion of multilinear polynomial.
A multilinear polynomial is a special kind of vector-valued multivariate polynomial in which no variables appears with a power of 2 or higher.
More formally, we have the following deﬁnition.
Deﬁnition 2.
A function g : R k !
R is called a univariate multilinear polynomial if it can be written as
[Equation]
where each ⌧ i 1 i 2 ··· i k 2 R .
The degree of a univariate multilinear polynomial is the maximum number of distinct variables occurring in any of the non-zero monomials ⌧ i
[Equation]
i [Equation] n [Equation]
A function f : ( R d ) k !
R p is called a multilinear polynomial map if there exist univariate multilinear polynomials
g i,j 1 , ··· ,j n for j 1 , · · · , j k 2 [ d ] and i 2 [ p ] such that f
[Equation]
x 1 , · · · , x [Equation] for all x 1 , · · · , x k 2 R d and all i 2 [ p ] .
The degree of f is the highest degree of the multilinear polynomials g i,j 1 , ··· ,j k .
The following theorem shows that CP layers can compute any permutation-invariant multilinear polynomial map.
We also visually represent the expressive power of CP layers in Fig.
3, showing that the class of functions computed by CP layer subsumes multilinear polynomials (including sum and mean aggregation functions).
Theorem 1.
The function computed by a CP layer (Eq.
equation 1) is permutation-invariant.
In addition, any permutation-invariant multilinear polynomial f : ( R F ) k !
R d can be computed by a CP layer (with a linear activation function).
Note also that in Fig.
3 the CP layer is strictly more expressive than permutation invariant multilinear polynomials due to the non-linear activation functions in Def. 1.
Since the classical sum and mean pooling aggregation functions are degree 1 multilinear polynomial maps, it readily follows from the previous theorem that the CP layer is more expressive than these standards aggregation functions.
However, it is natural to ask how many parameters a CP layer needs to compute sums and means.
We answer this question in the following theorem.
Theorem 2.
A CP layer of rank F · k can compute the sum and mean aggregation functions over k vectors in R F .
Consequently, for any k  1 and any GNN N using mean or sum pooling with feature and embedding dimensions bounded by F , there exists a GNN with CP layers of rank
F · k computing the same function as N over all graphs of uniform degree k .
It follows from this theorem that a CP layer with 2 F 2 k can compute sum and mean aggregation over sets of k vectors.
While Theorem 2 shows that any function using sum and mean aggregation can be computed by a CP layer, the next theorem shows that the converse is not true, i.e., the CP layer is a strictly more expressive aggregator than using the mean or sum.
Theorem 3.
With probability one, any function f CP : ( R F ) k !
R d computed by a CP layer (of any rank) whose parameters are drawn randomly (from a distribution which is continuous w.r.t.
the
Lebesgue measure) cannot be computed by a function of the form
[Equation]
where M 2 R d ⇥ R , W 2 R F ⇥ R and  ,  0 are component-wise activation function.
This theorem not only shows that there exist functions computed by CP layers that cannot be computed using sum pooling, but that this is the case for almost all functions that can be computed by (even rank-one) CP layers.
From an expressive power viewpoint, we showed that a CP layer is able to leverage both low and highorder multiplicative interactions.
However, from a learning perspective, it is clear that the CP layer has a natural inductive bias towards capturing high-order interactions.
We are not enforcing any sparsity in the tensor parameterizing the polynomial, thus the number and magnitude of weights corresponding to high-order terms will dominate the result (intuitively, learning a low order polynomial would imply setting most of these weights to zero).
In order to counterbalance this bias, we complement the CP layer with simple but efﬁcient linear low-order interactions (reminiscent of the idea behind residual networks [21]) when using it in tGNN:
[Equation]
where the ﬁrst term corresponds to the CP layer and the second one to a standard sum pooling layer (with  ,  0 and  00 being activation functions).
Related Work and Discussion
We now discuss relevant work on the parameterization of tensors on graph neural networks.
In general, our work relates to three areas of deep learning: (1) aggregation functions, (2) universal approximator for set aggregation, (3) high-order pooling, and (4) tensor methods for deep learning.
GNN & Aggregation Scheme [26] successfully deﬁne convolutions on graph-structured data by averaging node information in a neighborhood.
[48] prove the incomplete expressivity of mean aggregation to distinguish nodes, and further propose to use sum aggregation to differentiate nodes with similar properties.
[13] further generalize this idea and show that mean aggregation can be a particular case of sum aggregation with a linear multiplier, and further propose an architecture with multiple aggregation channels to adaptively learn low-order information.
Most GNNs use low-order aggregation schemes for learning node representations.
To the best of our knowledge, tGNN is the ﬁrst GNN architecture that adopts high-order multiplicative interactions in aggregation.
Universal Approximator & Permutation-invariant NN Universal approximators for set aggregation functions have been previously proposed and studied.
[51] show that sum pooling is enough provided that it is combined with two universal approximators.
[48] further discuss the limitations of non-injective set function.
[25] propose a generalization of transformers to permutation-invariant sets.
Most universal approximation results for GNN relies on combining simple aggregation functions (e.g. sum, mean) with universal approximators for the feature and output maps.
In contrast, the CP layer achieves the same goal of being an universal approximator but using a different mean: explicit computation of multilinear polynomials (in an effective manner using the CP parameterization).
While the approaches might be expressively similar, they may differ from a learning aspect, i.e., how easy it is for these models to learn a function from data.
High-order Pooling [2] formulate the pooling problem as a multiset encoding problem with auxiliary information about the graph structure, and propose an attention-based pooling layer that captures the interaction between nodes according to their structural dependencies.
[46] apply secondorder statistic methods because the use of second-order statistics takes advantage of the Riemannian geometry of the space of symmetric positive deﬁnite matrices.
However, high-order in CP pooling refers to high-dimensional multiplicative feature products, which is different than previous literature.
Tensor Methods Tensor methods allow one to deﬁne meaningful geometries to build more expressive models.
Authors in [52, 30] deﬁne geometries of tensor networks on complex or hypercomplex manifolds, their models encompass greater freedom in the choice of the product between the algebra elements.
[7] develop a general framework for both probabilistic and neural models for tree-structured data with a tensor-based aggregation function.
[6] leverages permutation-invariant CP-based aggregation function to capture high-order interactions in NLP tasks.
Using multiplicative interactions as a powerful source of non-linearities in neural network models have been studied previously for convolutional [11, 10, 32] and recurrent networks [41, 44, 47].
The connection between such multiplicative interactions with tensor networks have been leveraged both from theoretical and practical perspectives.
The CP layer can be seen as a graph generalization of the convolutional and recurrent arithmetic circuits considered in [10] and [31], respectively.
Experiments on Real-World Datasets
In this section, we evaluate Tensorized Graph Neural Net on real-world node- and graph-level datasets.
We introduce experiment setup in 6.1, compare tGNN with the state-of-the-arts models in 6.2, and conduct ablation study on model performance and efﬁciency in 6.3.
The hyperparameter and computing resources are attached in Appendix K. Dataset information can be found in Appendix J.
6.1 Experiment Setup
In this work, we conduct experiments on three citation networks ( Cora , Citeseer , Pubmed ) and three OGB datasets ( PRODUCTS , ARXIV , PROTEINS ) [23] for node-level tasks, one OGB dataset ( MolHIV ) [23] and three benchmarking datasets ( ZINC , CIFAR10 , MNIST ) [14] for graph-level tasks.
Training Procedure For three citation networks ( Cora , Citeseer , Pubmed ), we run experimentstimes on each dataset with 60%/20%/20% random splits used in [9], and report results in Tab. 1.
For data splits of OGB node and graph datasets, we follow [23], run experiments 5 times on each dataset (due to training cost), and report results in Tab. 1, 2.
For benchmarking datasets, we run experiments 5 times on each dataset with data split used in [14], and report results in Tab. 2.
To avoid numerical instability and ﬂoating point exception in tGNN training, we sample 5 neighbors for each node.
For graph datasets, we do not sample because the training is already in batch thus numerical instability can be avoided, and we apply the CP pooling at both node-level aggregation and graph-level readout.
Model Comparison tGNN has two hyperparameters, hidden unit and decomposition rank, we ﬁx hidden unit and explore decomposition rank.
For citation networks, we compare 2-layer GNNs with 32 hidden units.
And for OGB and benchmarking datasets, we use 32 hidden units for tGNN, and the results for all other methods are reported from the leaderboards and corresponding references.
Particularly, tGNN and CP pooling are more effective and expressive than existing pooling techniques for GNNs on two citation networks, two OGB node datasets, and one OGB graph dataset in Tab. 1, 2.
6.2 Real-world Datasets
In this section, we present tGNN performance on node- and graph-level tasks.
We compare tGNN with several classic baseline models under the same training setting.
For three citation networks, we compare tGNN with several baselines including GCN [26], GAT [45], GraphSAGE [19], H 2 GCN [54], GPRGNN [9], APPNP [27] and MixHop [1]; for three OGB node datasets, we compare tGNN with MLP, Node2vec [17], GCN [26], GraphSAGE [19] and DeeperGCN [33].
And for graph-level tasks, we compare tGNN with several baselines including MLP, GCN [26], GIN [48], DiffPool [50],
GAT [45], MoNet [37], GatedGCN [4], PNA [13], PHMGNN [30] and DGN [3].
The model choice is because we propose a new pooling method and want to mainly compare tGNN with other poolings in standard GNN architectures.
Current leading models on OGB leaderboards [23] adopt transformer, equivariant, ﬁngerprint, C&S, or others, which are more complex than standard GNNs.
We visualize the performance boost and comparisons with GNNs and pooling techniques in Tab.
From Tab. 1, we can observe that tGNN outperforms all classic baselines on Cora , Pubmed , PRODUCTS and ARXIV , and have slight improvements on the other datasets but underperforms GCN on
Citeseer and DeeperGCN on PROTEINS .
On the citation networks, tGNN outperforms others onout of 3 datasets.
Moreover, on the OGB node datasets, even when tGNN is not ranked ﬁrst, it is still very competitive (top 3 for all datasets).
We believe it is reasonable and expected that tGNN does not outperform all methods on all datasets.
But overall tGNN shows very competitive performance and
[Table]
deliver signiﬁcant improvement on challenging graph benchmarks compared to popular commonly used pooling methods (with comparable computational cost)
[Table]
in comparison with GNN architectures.
[Table]
[Figure]
In Tab. 2, we present tGNN performance on graph property prediction tasks.
tGNN achieves stateof-the-arts results on MolHIV , and have slight improvements on other three Benchmarking graph datasets.
Overall tGNN achieves more effective and accurate results on 5 out of 10 datasets comparing with existing pooling techniques, which suggests that high-order CP pooling can leverage a GNN to generalize better node embeddings and graph representations.
6.3 Ablation and Efﬁciency Study
In the ablation study, we ﬁrst investigate the effectiveness of having the high-order non-linear CP pooling and adding the linear low-order transformation in Tab. 3, then investigate the relations of the model performance, efﬁciency, and tensor decomposition rank in Fig. 4.
Moreover, we compare tGNN with different GNN architectures and aggregation functions to show the efﬁciency in Tab.
5,by showing the number of model parameters, computation time, and accuracy.
In Tab. 3, we test each component, high-order non-linear CP pooling, low-order linear sum pooling, and two pooling techniques combined, separately.
We ﬁx 2-layer GNNs with 32 hidden unit anddecomposition rank, and run experiments 10 times on Cora and Pubmed with 60%/20%/20% random splits used in [9], run ZINC 5 times with 10,000/1,000/1,000 graph split used in [14].
From the results, we can see that adding the linear low-order interactions helps put essential weights on them.
Ablation results show that high-order CP pooling has the advantage over low-order linear pooling for generating expressive node and graph representations, moreover, tGNN is more expressive with the combination of high-order pooling and low-order aggregation.
This illustrates the necessity of learning high-order components and low-order interactions simultaneously in tGNN.
Computational Aspect In Fig.
4, we compare model performance with computation costs.
We ﬁx a 2-layer tGNN with 32 hidden channels with 8,16,...,2048,4096 rank, and 2-layer baselines with 8,16,...,2048,4096 hidden dim.
We run 10 times on each citation network with the same 60%/20%/20% random splits for train/validation/test and draw the relations of average test accuracy, rank/hidden dim, and computation costs.
From the ﬁgure, we can see that the model performance can be improved with higher ranks ( i.e., Tensor T is more accurately computed as the rank R gets larger), but training time is also increased, thus it is a trade-off between classiﬁcation accuracy and computation efﬁciency.
And in comparison with baselines, tGNN still has marginal improvements with higher ranks while the baselines stop improving with larger hidden dimensions.
In Sec. 4.1, we theoretically discuss the time complexity of tGNN.
In Appendix H, we experimentally assess model efﬁciency by comparing tGNN with GCN [26], GAT [45], GCN2 [8], and mean, max poolings on Cora on a CPU over 10 runtimes, and compare the number of model parameters, training epochs per second, and accuracy.
The experiments show that tGNN is more competitive in terms of running time and better accuracy with a ﬁxed number of parameters and the same time budget.
Conclusion and Future Work
In this paper, we theoretically develop a high-order permutation-invariant multilinear map for node aggregation and graph pooling via tensor parameterization.
We show its powerful ability to compute any permutation-invariant multilinear polynomial including sum and mean pooling functions.
Experiments demonstrate that tGNN is more effective and accurate on 5 out of 10 datasets, showcasing the relevance of tensor methods for high-order graph neural network models.
For future work, one interesting direction is to augment various GNN architectures with non-linear high-order CP layers.
Most of the existing GNN models adopt low-order aggregation and pooling functions, it would be interesting to equip current state-of-the-art models with high-order pooling functions for a potential performance boost.
Another interesting direction is to enhance tGNN with an adaptive channel mixing mechanism.
In tGNN with high-order and low-order pooling functions, one node receives two channels of information combined in a linear way.
The linear combination may not be sufﬁcient to balance and extract high-order components and low-order components, so it would be interesting to design an adaptive channel mixing tGNN model for learning from different node-wise components.
Acknowledgements
This research was supported by the Canadian Institute for Advanced Research (CIFAR AI chair program), the Natural Sciences and Engineering Research Council of Canada (Discovery program,
RGPIN-2019-05949).
This research was enabled in part by support provided by Calcul Québec (www.calculquebec.ca) and the Digital Research Alliance of Canada (alliance.can.ca).
The authors acknowledge the material support of NVIDIA in the form of computational resources.