High-Order Pooling for Graph Neural Networks with Tensor Decomposition
Abstract
Graph Neural Networks (GNNs) are attracting growing attention due to their effectiveness and flexibility in modeling a variety of graph-structured data.
Exiting GNN architectures usually adopt simple pooling operations ( e.g., sum, average, max) when aggregating messages from a local neighborhood for updating node representation or pooling node representations from the entire graph to compute the graph representation.
Though simple and effective, these linear operations do not model high-order non-linear interactions among nodes.
We propose the Tensorized Graph
Neural Network (tGNN), a highly expressive GNN architecture relying on tensor decomposition to model high-order non-linear node interactions.
tGNN leverages the symmetric CP decomposition to efficiently parameterize permutation-invariant multilinear maps for modeling node interactions.
Theoretical and empirical analysis on both node and graph classification tasks show the superiority of tGNN over competitive baselines.
In particular, tGNN achieves the most solid results on two OGB node classification datasets and one OGB graph classification dataset.
Introduction
Graph neural networks (GNNs) generalize traditional neural network architectures for data in the
Euclidean domain to data in non-Euclidean domains [26, 44, 34].
As graphs are very general and flexible data structures and are ubiquitous in the real world, GNNs are now widely used in a variety of domains and applications such as social network analysis [20], recommender systems [48], graph reasoning [54] and drug discovery [41].
Indeed, many GNN architectures ( e g., GCN [26], GAT [44], MPNN [16]) have been proposed.
The essential idea of all these architectures is to iteratively update node representations by aggregating the information from their neighbors through multiple rounds of neural message passing.
The final node representations can be used for downstream tasks such as node classification or link prediction.
For graph classification, an additional readout layer is used to combine all the node representations to calculate the entire graph representation.
In general, an effective aggregation (or pooling) function is required to aggregate the information at the level of both local neighborhoods and the entire graph.
In practice, some simple aggregation functions are usually used such as sum, mean, and max.
Though simple and effective in some applications, the expressiveness of these functions is limited as they only model linear combinations of node features, which can limit their effectiveness in some cases.
A recent work, principled neighborhood aggregation (PNA) [13], aims to design a more flexible aggregation function by combining multiple simple aggregation functions, each of which is associated with a learnable weight.
However, the practical capacity of PNA is still limited by simply combining multiple simple aggregation functions.
A more expressive solution would be to model high-order non-linear interactions when aggregating node features.
However, explicitly modeling high-order non-linear interactions among nodes is very expensive, with both the time and memory complexity being exponential in the size of the neighborhood.
This raises the question of whether there exists an aggregation function which can model high-order non-linear interactions among nodes while remaining computationally efficient.
In this paper, we propose such an approach based on symmetric tensor decomposition.
We design an aggregation function over a set of node representations for graph neural networks, which is permutation-invariant and is capable of modeling non-linear high-order multiplicative interactions among nodes.
We leverage the symmetric CANDECOMP/PARAFAC decomposition (CP) [22, 29] to design an efficient parameterization of permutation-invariant multilinear maps over a set of node representations.
Theoretically, we show that the CP layer can compute any permutation-invariant multilinear polynomial, including the classical sum and mean aggregation functions.
We also show that the CP layer is universally strictly more expressive than sum and mean pooling: with probability one, any function computed by a random CP layer cannot be computed using sum and mean pooling.
We propose the CP-layer as an expressive mean of performing the aggregation and update functions in GNN.
We call the resulting model a tensorized GNN (tGNN).
We evaluate tGNN on both node and graph classification tasks.
Experimental results on real-world large-scale datasets show that our proposed architecture outperforms or can compete with existing state-of-the-art approaches and traditional pooling techniques.
Notably, our proposed method is more effective and expressive than existing GNN architectures and pooling methods on two citation networks, two OGB node datasets, and one OGB graph dataset.
Summary of Contributions We propose a new aggregation layer, the CP layer, for pooling and readout functions in GNNs.
This new layer leverages the symmetric CP decomposition to efficiently parameterize polynomial maps, thus taking into account high-order multiplicative interactions between node features.
We theoretically show that the CP layer can compute any permutation-invariant multilinear polynomial including sum and mean pooling.
Using the CP layer as a drop-in replacement for sum pooling in classical GNN architectures, our approach achieves more effective and expressive results than existing GNN architectures and pooling methods on several benchmark graph datasets.
Preliminaries 2.1 Notation
We use bold font letters for vectors ( e.g., v ), capital letters ( e.g., M , T ) for matrices and tensors respectively, and regular letters for nodes ( e.g., v ).
Let G =
( V, E ) be a graph, where V is the node set and E is the edge set with self-loop.
We use N ( v ) to denote the neighborhood set of node v , i.e.,
N ( v ) =
{ u : e vu ∈ E } .
A node feature is a vector x ∈ R F defined on V , where x v is defined on the node v .
We use ⊗ to denote the Kronecker product, ◦ to denote the outer product, and ⊙ to denote the Hadamard ( i.e., component-wise) product between vectors, matrices, and tensors.
For any integer k , we use the notation [ k ] = { 1 , · · · , k } .
2.2 Tensors
We introduce basic notions of tensor algebra, more details can be found in [29].
A k -th order tensor
T ∈ R N 1 × N 2 × ...
× N k can simply be seen as a multidimensional array.
The mode- i fibers of T are the vectors obtained by fixing all indices except the i -th one: T n 1 ,n 2 ,...,n i − 1 , : ,n i +1 ,...,n k ∈ R N i .
The i -th mode matricization of a tensor is the matrix having its mode- i fibers as columns and is denoted by T ( i ) , e.g., T (1) ∈ R N 1 × N 2 ··· N k .
We use T × i v ∈ R N 1 ×···× N i − 1 × N i +1 ×···× N k to denote the mode- i product between a tensor T ∈ R N 1 ×···× N k and a vector v ∈ R N i , which is defined by ( T × i v ) n 1 ,...,n i − 1 ,n i +
[Equation]
,...,n k = (cid:80) N i n [Equation] =1 T n 1 ,...,n k v n i .
The following useful identity relates the mode- i product with the Kronecker product:
[Equation]
2.3 CANDECOMP/PARAFAC Decomposition
We refer to C ANDECOMP/ P ARAFAC decomposition of a tensor as CP decomposition [24, 22].
A Rank R CP decomposition factorizes a k − th order tensor T ∈ R N 1 × ...
× N k into the sum of R
[Equation]
rank one tensors as T =
(cid:80)
Rr =1 v 1 r ◦ v 2 r ◦ · · · ◦ v kr , where ◦ denotes the vector outer-product and v 1 r ∈ R N 1 , v 2 r ∈ R N 2 , ..., v kr ∈ R N k for every r = 1 , 2 , ..., R .
The decomposition vectors, v : r for r = 1 , ..., R , are equal in length, thus can be naturally gathered into factor matrices M 1 =
[ v 11 , ..., v 1 R ] ∈ R N 1 × R , ..., M k
= [ v k 1 , ..., v kR ] ∈ R N k × R .
Using the factor matrices, we denote the CP decomposition of T as
[Equation]
The k -th order tensor T is cubical if all its modes have the same size, i.e., N 1 = N
= ...
= N k := N .
A tensor T is symmetric if it is cubical and is invariant under permutation of its indices:
[Equation]
for any permutation ϕ :
[ k ] → [ k ] .
A rank R symmetric CP decomposition of a symmetric tensor
T is a decomposition of the form T =
[[ M , · · · , M ]] with M ∈ R N × R .
It is well known that any symmetric tensor admits a symmetric CP decomposition [12], we illustrate a rank R symmetric CP decomposition in Fig. 1.
We say that a tensor T is partially symmetric if it is symmetric in a subset of its modes [28].
For example, a 3 -rd order tensor T ∈ R N 1 × N 1 × N 3 is partially symmetric w.r.t.
modes 1 and 2 if it has symmetric frontal slices; i.e., T : , : ,k is a symmetric matrix for all k ∈ [ N 3 ] .
We prove the fact that any partially symmetric tensor admits a partially symmetric CP decomposition in Lemmain Appendix A, e.g., if T ∈ R N 1 × N 1 × N 3 is partially symmetric w.r.t.
modes 1 and 2, there exist
M ∈ R N 1 × R and W ∈ R N 3 × R such that T
=
[[ M , M , W ]] .
2.4 Graph Neural Networks and Pooling Functions
Given a graph G =
( V, E ) , a graph neural network always aggregates information in a neighborhood to give node-level representations.
During each message-passing iteration, the embedding h v corresponding to node v ∈ V is generated by aggregating features from N ( v ) [18].
Formally, at the l -th layer of a graph neural network,
[Equation]
where AGGREGATE ( l ) ( · ) and UPDATE ( l ) ( · ) are differentiable functions, the former being permutation-invariant.
In words, AGGREGATE ( l ) ( · ) first aggregates information from N ( v ) , then
UPDATE ( l ) ( · ) combines the aggregated message and previous node embedding h ( l − 1) v to give a new embedding.
The node representation h ( L ) v of node v from the last GNN layer L can be used for predicting relevant properties of the node.
For graph classification, an additional READOUT ( · ) function aggregates node representations from the final layer to obtain a graph representation h G of graph G as,
[Equation]
where READOUT ( · ) can be a simple permutation-invariant function ( e.g., sum, mean, etc. ) or a more sophisticated graph pooling function [49, 47].
In the general design of GNNs, it is important to have an effective pooling function to aggregate messages from local neighborhoods and update node representations (see Eq.
(2)), and to combine node representations to compute a representation at the graph level (see Eq.
Meaning of High-Order of tGNN
In our paper, high-order refers to multi-dimensional feature products between nodes in a neighborhood. [37, 42] are previous works on high-dimensional feature products, and introduce how to use tensor methods to achieve those products. They have stated the importance of having high-order products in physics and geometries. For example, if we have x 1 and x 2 , a simple GNN layer can result in a vector of [ x 1 , x 2 , x 1 + x 2 ] , but our high-order layer result in a vector of [ x 1 , x 2 , x 1 + x 2 , x 1 x 2 ] with another term x 1 x 2 .
And if we have we have x 1 , x 2 , and x 3 , a simple low-order GNN layer will result in [ x 1 , x 2 , x 3 , x 1 + x 2 , x 1 + x 3 , x 2 + x 3 , x 1 + x 2 + x 3 ] while the high-order layer will result in a vector of [ x 1 , x 2 , x 3 , x 1 + x 2 , x 1 + x 3 , x 2 + x 3 , x 1 + x 2 + x 3 , x 1 x 2 , x 1 x 3 , x 2 x 3 , x 1 x 2 x 3 ] with extra 4 terms at a higher feature dimension. In general content of GNNs, high-order will normally refer to the ability of capturing long-range dependencies, the interactions between long-range nodes [47, 2, 45], but ours refers to the multi-dimensional node feature products. However, the CP layer is able to capture stacking long-range dependencies by stacking layers like other simple GNNs [26, 44].
One may raise a question on why we need to compute multi-dimensional feature products.
The idea is fairly easy if one thinks the multi-dimensional feature space. Take a molecular graph dataset as an example, and blood pressure could be one feature.
Many studies analyze the impact of blood pressure on one’s health.
Blood pressure is the feature in this case. So, one could try to predict life expectancy using blood pressure as a feature, but you can imagine that this might not be sufficient. You really need more information. So you could add age, exercise frequency, and body fat to see if you can predict a person’s life expectancy.
Those additional measurements are also features and now every individual has a multi-dimensional feature vector consisting of these measurements.
However, a simple GNN with one-dimensional feature interactions may not capture the correlation between age, exercise frequency, and body fat. There could be some correlation between the three factors to predict life expectancy.
A simple GNN that does age+exercise frequency+body fat is not able to capture the correlation between those factors. Still, our multi-dimensional feature product can capture age × exercise frequency, age × body fat, exercise frequency × body fat, age × exercise frequency × body fat, all the correlations to predict life expectancy. So in molecular graph case, high-dimensional feature products are importantly needed to make a prediction as most features are correlated (like in atom graphs, atom charge, and atom mass), and our model shows some significant improvements on molecular datasets in Sec.6. And the high-dimensional feature products might also be important for other networks and applications [37, 42].
Tensorized Graph Neural Network
In this section, we introduce the CP-layer and tensorized GNNs (tGNN).
For convenience, we let { x 1 , x 2 , ..., x k } denote features of a node v and its 1 -hop neighbors N ( v ) such that |{ v }∪ N ( v ) |
=
k .
4.1 Motivation and Method
We leverage the symmetric CP decomposition to design an efficient parameterization of permutationinvariant multilinear maps for aggregation operations in graph neural networks, Tensorized Graph
Neural Network (tGNN), resulting in a more expressive high-order node interaction scheme.
We visualize the CP pooling layer and compare it with sum pooling in Fig.
Let T ∈ R N × N ×···× N × M of order k + 1 be a tensor which is partially symmetric w.r.t.
its first k modes.
We can parameterize T using a rank R partially symmetric CP decomposition (see
Section 2.3): T =
[[ W , · · · , W , M ]] where W ∈ R N × R and M ∈ R M × R .
Such a tensor naturally defines a map from ( R N ) k to R M using contractions over the first k modes:
[Equation]
This map satisfies two very important properties for GNNs: it is permutation-invariant (due to the partial symmetry of T ) and its number of parameters is independent of k (due to the partially symmetric CP parameterization).
Thus, using only two parameter matrices of fixed size, the map in
Eq.
(4) can be applied to sets of N -dimensional vectors of arbitrary cardinality.
In particular, we will show that it can be leveraged to replace both the AGGREGATE and UPDATE functions in GNNs. CP Layer W
Input Features x y z W W W
Product Pooling
Layer x’ y’ z’
W x’
W y’
W z’
Activation
M
Linear Transformation via M
Linear Transformation via W =
Output Representation of 27 Elements
If Linear Activation = = 4-th Order Symmetric Tensor T } y } z } x only considers 1st order
There are several way to interpret the map in Eq.
First, from Eq.
(1) we have
[Equation]
where T ( k +1) ∈ R M × N k is the mode- ( k + 1) matricization of T .
This shows that each element of the output f ( x 1 , · · · , x k ) is a linear combinations of terms of the form ( x 1 ) i 1 ( x 2 ) i 2 · · · ( x k )
i k
( k -th order multiplicative interactions between the components of the vectors x 1 , · · · , x k ).
That is, f is a multivariate polynomial map of order k involving only k -th order interactions.
By using homogeneous coordinates, i.e., appending an entry equal to one to each of the input tensors x i , the map f becomes a more general polynomial map taking into account all multiplicative interactions between the x i up to the k -th order:
[Equation]
where T is now of size ( N +1)
×· · ·× ( N +1)
× M and can still be parameterized using the partially symmetric CP decomposition T =
[[ W , · · · , W , M ]] with W ∈ R ( N +1) × R and M ∈ R M × R .
With this parameterization, one can check that
[Equation]
where ⊙ denotes the component-wise product between vectors.
The map f can thus be seen as the composition of a linear layer with weight W , a multiplicative pooling layer, and another linear map
M .
Since it is permutation-invariant and can be applied to any number of input vectors, this map can be used as both the aggregation, update, and readout functions of a GNN using non-linear activation functions, which leads us to introduce the novel CP layer for GNN.
Definition 1.
(CP layer)
Given parameter matrices M ∈ R d × R and W ∈ R F +1 × R and activation functions σ, σ ′ , a rank R CP layer computes the function f CP : ∪ i ≥ 1 ( R F ) i → R d defined by
[Equation]
for any k ≥ 1 and any x 1 , · · · , x k ∈ R F .
The rank R of a CP layer is a hyperparameter controlling the trade-off between parameter efficiency and expressiveness.
Note that the CP layer computes AGGREGATE and UPDATE (see Eq. 2) in one step.
One can think of the component-wise product of the W ⊤ [ x i 1] ⊤ as AGGREGATE, while the UPDATE corresponds to the two non-linear activation functions and linear transformation M .
We observed in our experiments that the non-linearity σ is crucial to avoid numerical instabilities during training caused by repeated products of W .
In practice, we use Tanh for σ and ReLU for σ ′ .
Fig.
2 graphically explains the computational process of a CP layer, comparing it with a classical sum pooling operation.
We intuitively see in this figure that the CP layer is able to capture high order multiplicative interactions that are not modeled by simple aggregation functions such as the sum or the mean.
In the next section, we theoretically formalize this intuition.
Complexity Analysis
The sum, mean and max poolings result in O ( F in ( N + F out )) time complexity, while CP pooling is O ( R ( NF in + F out ) ) , where N denotes the number of nodes, F in is the input feature dimension, F out is out feature dimension, and R is the CP decomposition rank.
In Sec.
6.3, we experimentally compare tGNN and CP pooling with various GNNs and pooling techniques to show the model efficiency with limited computation and time budgets.
4.2 Theoretical Analysis
We now analyze the expressive power of CP layers.
In order to characterize the set of functions that can be computed by CP layers, we first introduce the notion of multilinear polynomial.
A multilinear polynomial is a special kind of vector-valued multivariate polynomial in which no variables appears with a power of 2 or higher.
More formally, we have the following definition.
Definition 2.
A function g : R k
→ R is called a univariate multilinear polynomial if it can be written as
[Equation]
where each τ i 1 i 2 ··· i k ∈ R .
The degree of a univariate multilinear polynomial is the maximum number of distinct variables occurring in any of the non-zero monomials τ
[Equation] i 2 2 · · · a i k k .
A function f : ( R d ) k
→ R p is called a multilinear polynomial map if there exist univariate multilinear polynomials
g i,j 1 , ··· ,j n for j 1 , · · · , j k ∈ [ d ] and i ∈
[ p ] such that f ( x 1 , · · · , x k ) i =
d (cid:88) j 1 , ··· ,j k = g
i,j 1 , ··· ,j k (( x 1 ) j 1 , · · · , ( x k ) j k ) for all x 1 , · · · , x k ∈ R d and all i ∈ [ p ] .
The degree of f is the highest degree of the multilinear polynomials g i,j 1 , ··· ,j k . space function space} ⊇ {permutation space} ⊇ {sum aggregation
The following theorem shows that CP layers can compute any permutation-invariant multilinear polynomial map.
We also visually represent the expressive power of CP layers in Fig.
3, showing that the class of functions computed by CP layer subsumes multilinear polynomials (including sum and mean aggregation functions).
Theorem 1.
The function computed by a CP layer (Eq.
(1) ) is permutation-invariant.
In addition, any permutation-invariant multilinear polynomial f : ( R F ) k
→ R d can be computed by a CP layer (with a linear activation function).
Note also that in Fig.
3 the CP layer is strictly more expressive than permutation invariant multilinear polynomials due to the non-linear activation functions in Def. 1.
Since the classical sum and mean pooling aggregation functions are degree 1 multilinear polynomial maps, it readily follows from the previous theorem that the CP layer is more expressive than these standards aggregation functions.
However, it is natural to ask how many parameters a CP layer needs to compute sums and means.
We answer this question in the following theorem.
Theorem 2.
A CP layer of rank F · k can compute the sum and mean aggregation functions over k vectors in R F .
Consequently, for any k ≥ 1 and any GNN N using mean or sum pooling with feature and embedding dimensions bounded by F , there exists a GNN with CP layers of rank F · k computing the same function as N over all graphs of uniform degree k .
It follows from this theorem that a CP layer with 2 F 2 k can compute sum and mean aggregation over sets of k vectors.
While Theorem 2 shows that any function using sum and mean aggregation can be computed by a CP layer, the next theorem shows that the converse is not true, i.e., the CP layer is a strictly more expressive aggregator than using the mean or sum.
Theorem 3.
With probability one, any function f CP : ( R F ) k
→ R d computed by a CP layer (of any rank) whose parameters are drawn randomly (from a distribution which is continuous w.r.t.
the
Lebesgue measure) cannot be computed by a function of the form
[Equation]
where M ∈ R d × R , W ∈ R F × R and σ , σ ′ are component-wise activation function.
This theorem not only shows that there exist functions computed by CP layers that cannot be computed using sum pooling, but that this is the case for almost all functions that can be computed by (even rank-one) CP layers.
From an expressive power viewpoint, we showed that a CP layer is able to leverage both low and highorder multiplicative interactions.
However, from a learning perspective, it is clear that the CP layer has a natural inductive bias towards capturing high-order interactions.
We are not enforcing any sparsity in the tensor parameterizing the polynomial, thus the number and magnitude of weights corresponding to high-order terms will dominate the result (intuitively, learning a low order polynomial would imply setting most of these weights to zero).
In order to counterbalance this bias, we complement the CP layer with simple but efficient linear low-order interactions (reminiscent of the idea behind residual networks [21]) when using it in tGNN:
[Equation]
⊙ · · · ⊙ W ⊤(cid:20) x k (cid:21)(cid:19)(cid:19)(cid:19) + σ ′′
W ⊤ 2 x 1 + ... + W ⊤ 2 x k ) (5) where the first term corresponds to the CP layer and the second one to a standard sum pooling layer (with σ , σ ′ and σ ′′ being activation functions).
Related Work and Discussion
We now discuss relevant work on the parameterization of tensors on graph neural networks.
In general, our work relates to three areas of deep learning: (1) aggregation functions, (2) universal approximator for set aggregation, (3) high-order pooling, and (4) tensor methods for deep learning.
GNN & Aggregation Scheme [26] successfully define convolutions on graph-structured data by averaging node information in a neighborhood.
[47] prove the incomplete expressivity of mean aggregation to distinguish nodes, and further propose to use sum aggregation to differentiate nodes with similar properties.
[13] further generalize this idea and show that mean aggregation can be a particular case of sum aggregation with a linear multiplier, and further propose an architecture with multiple aggregation channels to adaptively learn low-order information.
Most GNNs use low-order aggregation schemes for learning node representations.
To the best of our knowledge, tGNN is the first GNN architecture that adopts high-order multiplicative interactions in aggregation.
Universal Approximator & Permutation-invariant NN Universal approximators for set aggregation functions have been previously proposed and studied.
[50] show that sum pooling is enough provided that it is combined with two universal approximators.
[47] further discuss the limitations of non-injective set function.
[25] propose a generalization of transformers to permutation-invariant sets.
Most universal approximation results for GNN relies on combining simple aggregation functions (e.g. sum, mean) with universal approximators for the feature and output maps.
In contrast, the CP layer achieves the same goal of being an universal approximator but using a different mean: explicit computation of multilinear polynomials (in an effective manner using the CP parameterization).
While the approaches might be expressively similar, they may differ from a learning aspect, i.e., how easy it is for these models to learn a function from data.
High-order Pooling [2] formulate the pooling problem as a multiset encoding problem with auxiliary information about the graph structure, and propose an attention-based pooling layer that captures the interaction between nodes according to their structural dependencies.
[45] apply secondorder statistic methods because the use of second-order statistics takes advantage of the Riemannian geometry of the space of symmetric positive definite matrices.
However, high-order in CP pooling refers to high-dimensional multiplicative feature products, which is different than previous literature.
Tensor Methods Tensor methods allow one to define meaningful geometries to build more expressive models.
Authors in [51, 30] define geometries of tensor networks on complex or hypercomplex manifolds, their models encompass greater freedom in the choice of the product between the algebra elements.
[7] develop a general framework for both probabilistic and neural models for tree-structured data with a tensor-based aggregation function.
[6] leverages permutation-invariant CP-based aggregation function to capture high-order interactions in NLP tasks.
Using multiplicative interactions as a powerful source of non-linearities in neural network models have been studied previously for convolutional [11, 10, 32] and recurrent networks [40, 43, 46].
The connection between such multiplicative interactions with tensor networks have been leveraged both from theoretical and practical perspectives.
The CP layer can be seen as a graph generalization of the convolutional and recurrent arithmetic circuits considered in [10] and [31], respectively.
Experiments on Real-World Datasets
In this section, we evaluate Tensorized Graph Neural Net on real-world node- and graph-level datasets.
We introduce experiment setup in 6.1, compare tGNN with the state-of-the-arts models in 6.2, and conduct ablation study on model performance and efficiency in 6.3.
The hyperparameter and computing resources are attached in Appendix K. Dataset information can be found in Appendix J.
6.1 Experiment Setup
In this work, we conduct experiments on three citation networks ( Cora , Citeseer , Pubmed ) and three OGB datasets ( PRODUCTS , ARXIV , PROTEINS ) [23] for node-level tasks, one OGB dataset ( MolHIV ) [23] and three benchmarking datasets ( ZINC , CIFAR10 , MNIST ) [14] for graph-level tasks.
Training Procedure For three citation networks ( Cora , Citeseer , Pubmed ), we run experimentstimes on each dataset with 60%/20%/20% random splits used in [9], and report results in Tab. 1.
For data splits of OGB node and graph datasets, we follow [23], run experiments 5 times on each dataset (due to training cost), and report results in Tab. 1, 2.
For benchmarking datasets, we run experiments 5 times on each dataset with data split used in [14], and report results in Tab. 2.
To avoid numerical instability and floating point exception in tGNN training, we sample 5 neighbors for each node.
For graph datasets, we do not sample because the training is already in batch thus numerical instability can be avoided, and we apply the CP pooling at both node-level aggregation and graph-level readout.
Model Comparison tGNN has two hyperparameters, hidden unit and decomposition rank, we fix hidden unit and explore decomposition rank.
For citation networks, we compare 2-layer GNNs with 32 hidden units.
And for OGB and benchmarking datasets, we use 32 hidden units for tGNN, and the results for all other methods are reported from the leaderboards and corresponding references.
Particularly, tGNN and CP pooling are more effective and expressive than existing pooling techniques for GNNs on two citation networks, two OGB node datasets, and one OGB graph dataset in Tab. 1, 2.
6.2 Real-world Datasets
In this section, we present tGNN performance on node- and graph-level tasks.
We compare tGNN with several classic baseline models under the same training setting.
For three citation networks, we compare tGNN with several baselines including GCN [26], GAT [44], GraphSAGE [19], H 2 GCN [53], GPRGNN [9], APPNP [27] and MixHop [1]; for three OGB node datasets, we compare tGNN with MLP, Node2vec [17], GCN [26], GraphSAGE [19] and DeeperGCN [33].
And for graph-level tasks, we compare tGNN with several baselines including MLP, GCN [26], GIN [47], DiffPool [49],
GAT [44], MoNet [36], GatedGCN [4], PNA [13], PHMGNN [30] and DGN [3].
The model choice is because we propose a new pooling method and want to mainly compare tGNN with other poolings in standard GNN architectures.
Current leading models on OGB leaderboards [23] adopt transformer, equivariant, fingerprint, C&S, or others, which are more complex than standard GNNs.
We visualize the performance boost and comparisons with GNNs and pooling techniques in Tab.
From Tab. 1, we can observe that tGNN outperforms all classic baselines on Cora , Pubmed , PRODUCTS and ARXIV , and have slight improvements on the other datasets but underperforms GCN on
Citeseer and DeeperGCN on PROTEINS .
On the citation networks, tGNN outperforms others onout of 3 datasets.
Moreover, on the OGB node datasets, even when tGNN is not ranked first, it is still very competitive (top 3 for all datasets).
We believe it is reasonable and expected that tGNN does not outperform all methods on all datasets.
But overall tGNN shows very competitive performance and
[Table]
deliver significant improvement on challenging graph benchmarks compared to popular commonly used pooling methods (with comparable computational cost)
[Table]
in comparison with GNN architectures.
[Table]
[Table]
In Tab. 2, we present tGNN performance on graph property prediction tasks.
tGNN achieves stateof-the-arts results on MolHIV , and have slight improvements on other three Benchmarking graph datasets.
Overall tGNN achieves more effective and accurate results on 5 out of 10 datasets comparing with existing pooling techniques, which suggests that high-order CP pooling can leverage a GNN to generalize better node embeddings and graph representations.
6.3 Ablation and Efficiency Study
In the ablation study, we first investigate the effectiveness of having the high-order non-linear CP pooling and adding the linear low-order transformation in Tab. 3, then investigate the relations of the model performance, efficiency, and tensor decomposition rank in Fig. 4.
Moreover, we compare tGNN with different GNN architectures and aggregation functions to show the efficiency in Tab. 5,by showing the number of model parameters, computation time, and accuracy.
In Tab. 3, we test each component, high-order non-linear CP pooling, low-order linear sum pooling, and two pooling techniques combined, separately.
We fix 2-layer GNNs with 32 hidden unit anddecomposition rank, and run experiments 10 times on Cora and Pubmed with 60%/20%/20% random splits used in [9], run ZINC 5 times with 10,000/1,000/1,000 graph split used in [14].
From the results, we can see that adding the linear low-order interactions helps put essential weights on them.
Ablation results show that high-order CP pooling has the advantage over low-order linear pooling for generating expressive node and graph representations, moreover, tGNN is more expressive with the combination of high-order pooling and low-order aggregation.
This illustrates the necessity of learning high-order components and low-order interactions simultaneously in tGNN.
Computational Aspect In Fig.
4, we compare model performance with computation costs.
We fix a 2-layer tGNN with 32 hidden channels with 8,16,...,2048,4096 rank, and 2-layer baselines with 8,16,...,2048,4096 hidden dim.
We run 10 times on each citation network with the same 60%/20%/20% random splits for train/validation/test and draw the relations of average test accuracy, rank/hidden dim, and computation costs.
From the figure, we can see that the model performance can be improved with higher ranks ( i.e., Tensor T is more accurately computed as the rank R gets larger), but training time is also increased, thus it is a trade-off between classification accuracy and computation efficiency.
And in comparison with baselines, tGNN still has marginal improvements with higher ranks while the baselines stop improving with larger hidden dimensions.
In Sec. 4.1, we theoretically discuss the time complexity of tGNN.
In Appendix H, we experimentally assess model efficiency by comparing tGNN with GCN [26], GAT [44], GCN2 [8], and mean, max poolings on Cora on a CPU over 10 runtimes, and compare the number of model parameters, training epochs per second, and accuracy.
The experiments show that tGNN is more competitive in terms of running time and better accuracy with a fixed number of parameters and the same time budget.
Conclusion and Future Work
In this paper, we theoretically develop a high-order permutation-invariant multilinear map for node aggregation and graph pooling via tensor parameterization.
We show its powerful ability to compute any permutation-invariant multilinear polynomial including sum and mean pooling functions.
Experiments demonstrate that tGNN is more effective and accurate on 5 out of 10 datasets, showcasing the relevance of tensor methods for high-order graph neural network models.
For future work, one interesting direction is to augment various GNN architectures with non-linear high-order CP layers.
Most of the existing GNN models adopt low-order aggregation and pooling functions, it would be interesting to equip current state-of-the-art models with high-order pooling functions for a potential performance boost.
Another interesting direction is to enhance tGNN with an adaptive channel mixing mechanism.
In tGNN with high-order and low-order pooling functions, one node receives two channels of information combined in a linear way.
The linear combination may not be sufficient to balance and extract high-order components and low-order components, so it would be interesting to design an adaptive channel mixing tGNN model for learning from different node-wise components. Acknowledgements
This research was supported by the Canadian Institute for Advanced Research (CIFAR AI chair program) and the Natural Sciences and Engineering Research Council of Canada (Discovery program,
RGPIN-2019-05949).
A Proof of Theorem
In order to prove this theorem, we first need the following lemma showing the existence of partially symmetric CP decompositions.
Lemma 1.
Any partially symmetric tensor admits a partially symmetric CP decomposition.
Proof.
We show the results for 3-rd order tensors that are partially symmetric w.r.t.
their two first modes.
The proof can be straightforwardly extended to tensors of arbitrary order that are partially symmetric w.r.t.
any subset of modes.
Let T ∈ R m × m × n be partially symmetric w.r.t.
modes 1 and 2 .
We have that T : , : ,i is a symmetric tensor for each i ∈ [ n ] .
By Lemma 4.2 in [12], each tensor T : , : ,i admits a symmetric CP decomposition:
[Equation]
where A ( i ) ∈
R m × R i and R i is the symmetric CP rank of T : , : ,i .
By defining R = (cid:80) ni =1 R i and A =
[ A (1) A (2) · · · A ( n ) ]
∈ R m × R , one can easily check that T admits the partially symmetric CP decomposition T
=
[[ A , A , ∆ ]] where ∆ ∈ R m × R is defined by ∆ i,r = (cid:26)if R 1 + · · · + R i − 1 < r ≤ R 1 + · · · + R i otherwise.
We can now prove Theorem 1.
Theorem.
The function computed by a CP layer (Eq.
(1) ) is permutation-invariant.
In addition, any permutation-invariant multilinear polynomial f : ( R F ) k
→ R d can be computed by a CP layer (with linear activation function).
Proof.
The fact that the function computed by a CP layer is permutation-invariant directly follows from the definition of the CP layer and the fact that the Hadamard product is commutative.
We now show the second part of the theorem.
Let f : ( R F ) k
→ R d be a permutation-invariant multilinear polynomial map.
Then, there exists permutation-invariant univariate multilinear polynomials g i,j 1 , ··· ,j k for j 1 , · · · , j k ∈
[ F ] and i ∈ [ d ] such that
[Equation]
Moreover, by definition, each such polynomial satisfies
[Equation]
for some scalars τ ( i,j 1 , ··· ,j k )
i 1 , ··· ,i k .
Putting those two expressions together, we get
[Equation]
We can then group together the coefficients τ ( i,j 1 , ··· ,j k ) i 1 , ··· ,i k corresponding to the same monomials ( x 1 ) i 1 j 1 · · · ( x k ) i k j k .
E.g., the coefficients τ ( i,j 1 , ··· ,j k ) 1 , 0 , ··· ,all correspond to the same monomial ( x 1 ) jfor any values of j 2 , j 3 , · · · , j k .
Similarly, all the coefficients τ ( i,j 1 , ··· ,j k ) 0 , 1 , ··· ,all correspond to the same monomial ( x 2 ) j 2 · · · ( x k ) j k for any values of j 1 .
By grouping and summing together these coefficients into a tensor T ∈ R ( F +1) ×···× ( F +1) × d , where T j 1 , ··· ,j k ,i is equal to the sum (cid:88) j ℓ ∈ [ F ]: j ℓ = F +1 ∧ i ℓ = τ ( i,j 1 , ··· ,j k ) i 1 , ··· ,i k , we obtain
[Equation]
Since f is permutation-invariant, the tensor T is partially symmetric w.r.t.
its first k modes.
Thus, by
Lemma 1, there exist matrices A ∈ R R × F and B ∈ R d × R such that
[Equation]
with σ being the identity function, which concludes the proof.
B Proof of Theorem
Theorem.
A CP layer or rank F · k can compute the sum and mean aggregation functions for k vectors in R F .
Consequently, for any k ≥ 1 and any GNN N using mean or sum pooling with feature and embedding dimensions bounded by F , there exists a GNN with CP layers of rank F · k computing the same function as N over all graphs of uniform degree k .
Proof.
We will show that the function f : (cid:0) R F (cid:1) k
→ R d defined by
[Equation]
where α ∈ R , can be computed by a CP layer of rank Fk , which will show the first part of the theorem.
The second part of the theorem directly follows by letting α = 1 for the sum aggregation and α = 1 /k for the mean aggregation.
Let ˜ e 1 , · · · , ˜ e F +1 be the canonical basis of R F +1 , and let e 1 , · · · , e F be the canonical basis of R F .
We define the tensor T ∈ R ( F +1) ×···× ( F +1)
×
d by
T =
d (cid:88)
j = k
[Equation]
where ◦ denotes the outer (or tensor) product between vectors.
We start by showing that contracting k vectors in homogeneous coordinates along the first k modes of T results in the sum of those vectors weighted by α .
For any vectors x 1 , · · · , x k ∈ R F
[Equation]
we have
[Equation]
To show that f can be computed by a CP layer of rank Fk , it thus remains to show that T admits a partially symmetric CP decomposition of rank Fk .
This follows from Corollary 4.3 in [52], which states that any k th order tensor A of CP rank less than k has symmetric CP rank bounded by k .
Indeed, consider the tensors
[Equation]
for j ∈ [ d ] .
They are all k -th order tensor of CP rank bounded by k .
Thus, by Corollary 4.3 in [52], they all admit a symmetric CP decomposition of rank at most k , from which it directly follows that the tensor T =
(cid:80) Fj = A ( j ) ◦
e j admits a partially symmetric CP decomposition of rank at most
Fk .
C Proof of Theorem
Theorem.
With probability one, any function f CP : ( R F ) k
→ R d computed by a CP layer (of any rank) whose parameters are drawn randomly (from a continuous distribution) cannot be computed by a function of the form
[Equation]
where M ∈ R d × R , W ∈ R F × R and σ , σ ′ are component-wise activation function.
Proof.
Let f CP : ( R F ) k
→ R d be the function computed by a random CP layer.
I.e.,
[Equation]
where the entries of the matrices B ∈ R ( N +1) × R and A ∈ R M × R are identically and independently drawn from a distribution which is continuous w.r.t.
the Lebesgue measure.
It is well know that since the entries of the two parameter matrices are drawn from a continuous distribution, all the entries of A and B are non-zero and distinct with probability one.
It follows that all the entries of the vector f CP ( x 1 , · · · , x k ) are k -th order multilinear polynomials of the entries of the input vectors x 1 , · · · , x k , which, with probability one, have non trivial high-order interactions that cannot be computed by a linear map.
In particular, with probability one, the map f CP cannot be computed by any map of the form g sum : x 1 , · · · , x k
(cid:55)→ σ ′ (cid:16) M (cid:16) σ (cid:16)(cid:80) ki =1 W ⊤ x i (cid:17)(cid:17)(cid:17) since the sum pooling aggregates the inputs in a way that prevents modeling independent higher order multiplicative interactions, despite the non-linear activation functions.
D Rank and Performance
Generally speaking, the low-rank decomposition methods will sacrifice some expressivity for some computation costs, so as we increase the rank, we will re-gain some expressivity but lose some computation costs.
As talked about in our paper, it is a trade-off between expressivity (model performance) and computation costs.
This idea can be found in previous works [38, 39, 5] where they test on computer vision datasets (with visualization of recovering images from noises, a high-rank model can always better recover an image than a low-rank model does).
In this work, we use the low-rank decomposition technique which can potentially reduce the number of learnable parameters by a lower number of ranks.
The low-rank method saves the computation costs but the use of low-rank decomposition for learnable parameters will sacrifice some expressivity in computation.
If we increase the rank, the learnable parameters can be principally recovered, in this case, we no longer sacrifice its expressivity but computation costs (as the Figures shown in the ablation study).
In computer vision, low-rank decomposition methods were introduced to replace a fully-connected layer.
And in their papers [38, 39, 5] (in both theory and experiments), they show that a model is better with higher ranks because higher ranks can lead to higher expressivity than low ranks do.
They conclude and show that the model performance is positively correlated to the tensor decomposition rank.
In [38, 39, 5], they experimentally show that higher ranks can result in lower test errors as the model becomes more expressive (plotting a curve of decomposition rank or model parameter vs. error).
Moreover, in [38, 39], they visualize the relationship between the model performance and model rank.
They aim to recover the original pictures from noises with different tensor ranks (from low to high), and the recovered pictures are more clear with high ranks than with low ranks.
To conclude our findings in the ablation study and previous works on low-rank decomposition, high-rank models can usually have better performance and stronger expressivity than low-rank models, and higher ranks can usually lead to better performance (in terms of high accuracy or low error) than low ranks do.
E Low and High Terms
The reason we introduce the inductive bias is that we want to balance the importance of low- and high-order terms.
We design an experiment to show which interactions play the main role.
We have two learnable attention weights a low , a high , one before high-order terms and the other one before low-order terms, node representation H = a high H high + a low H low with a low + a high = 1 .
If high-order interactions dominate a high will be larger, else if low-order interactions dominate a low will be larger.
At the early model learning stage, a high is a lot higher ( ∼ 0 . 9 ) than a low ( ∼ 0 . 1 ), which show high-order terms dominate at the beginning, and a high , a low will converge to a similar value ( ∼ 0 . 5 ) as the model improves, which show that low- and high-order terms have the similar importance.
F GCN Performance
We do see in our Tables, GCN can beat popular baselines.
It is not really weird, if you can do a full hyperparameter search (either GCN built on PyG or Torch), you’ll find out that GCN can outperform the most popular GNN architectures (after GCN gets proposed).
And the papers [35, 34] find out the same experimental phenomenon.
The question, that GCN can outperform a lot GNNs on many (heterophilic or homophilic) datasets after thorough hyperparameter tuning, was discussed in the community before but never raised attention.
G Additional Graph Classification
[2] formulate the graph pooling problem as a multiset encoding problem with auxiliary information about the graph structure, and propose an attention-based pooling layer that captures the interaction between nodes according to their structural dependencies.
[45] apply second-order statistic methods because the use of second-order statistics takes advantage of the Riemannian geometry of the space of symmetric positive definite matrices.
Their formulation adapts the bilinear pooling, and the bilinear mapping is capable of capturing second-order statistics and topology information.
We test and compare tGNN with [2, 45] following the setting of [15].
[Table]
The table records graph classification results on test sets for 5 graph-level tasks under the setting of [15].
We can see that our model makes improvements on 4 out of 5 datasets (big improvements on
MUTAG, PROTEINS [Table] and COLLAB, and small improvements on IMDB-B) [Table]
H Efficiency Study
We study efficiency by comparing tGNN with models on Cora on a CPU over 10 runtimes, and compare the number of model paramterts, number of training epochs per second, and accuracy.
Sampling means we sample ’3’ neighbors for each node or we use ’Full’ neighborhood.
Notice that the average node degree of Cora is 3.9, which means if a node has a number of neighbors less than 10, some of its neighbor nodes will get resampled until it hits 10. [Table] From Table
[Table]
we can see that the model performance is not heavily affected by the number of sampled neighbor nodes because the average node degree is not that high on the majority of network datasets [Table] and ’3’ or ’5’ would be sufficient [Table]
[Table]
I Emperical Study on Classical Poolings
We study and compare CP pooling with classical pooling methods on Cora on a CPU over 10 runtimes, and compare the number of model paramterts, number of training epochs per second, and accuracy.
Sampling means we sample ’3’ neighbors for each node or we use ’Full’ neighborhood.
[Table]
A more detailed statistics of real-world datasets.
[Table]
[Table] [Table]
[Table]
PROTEINS
[Table] [Table]
K Hyperparameter
For experiments on real-world datasets, we use NVIDIA P100 Pascal as our GPU computation resource.
And the searching hyperparameter includes the learning rate, weight decay, dropout, decomposition rank R .
[Table] [Table]
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[No] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[No] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[No]