High-Order Pooling for Graph Neural Networks with Tensor Decomposition
Abstract
Graph Neural Networks (GNNs) are attracting growing attention due to their effectiveness and flexibility in modeling a variety of graph-structured data.
Exiting GNNarchitectures usually adopt simple pooling operations ( e.g., sum, average, max)when aggregating messages from a local neighborhood for updating node representation or pooling node representations from the entire graph to compute the graphrepresentation.
Though simple and effective, these linear operations do not modelhigh-order non-linear interactions among nodes.
We propose the Tensorized Graph
Neural Network (tGNN), a highly expressive GNN architecture relying on tensordecomposition to model high-order non-linear node interactions.
tGNN leveragesthe symmetric CP decomposition to efficiently parameterize permutation-invariantmultilinear maps for modeling node interactions.
Theoretical and empiricalanalysis on both node and graph classification tasks show the superiority of tGNNover competitive baselines.
In particular, tGNN achieves state-of-the-art results ontwo OGB node classification datasets and one OGB graph classification dataset.
Introduction
Graph neural networks (GNNs) generalize traditional neural network architectures for data in the
Euclidean domain to data in non-Euclidean domains [26, 43, 33].
As graphs are very general andflexible data structures and are ubiquitous in the real world, GNNs are now widely used in a varietyof domains and applications such as social network analysis [20], recommender systems [47], graphreasoning [53] and drug discovery [40].
Indeed, many GNN architectures ( e g., GCN [26], GAT [43], MPNN [16]) have been proposed.
Theessential idea of all these architectures is to iteratively update node representations by aggregatingthe information from their neighbors through multiple rounds of neural message passing.
The finalnode representations can be used for downstream tasks such as node classification or link prediction.
For graph classification, an additional readout layer is used to combine all the node representations tocalculate the entire graph representation.
In general, an effective aggregation (or pooling) function isrequired to aggregate the information at the level of both local neighborhoods and the entire graph.
Inpractice, some simple aggregation functions are usually used such as sum, mean, and max.
Thoughsimple and effective in some applications, the expressiveness of these functions is limited as theyonly model linear combinations of node features, which can limit their effectiveness in some cases.
A recent work, principled neighborhood aggregation (PNA) [13], aims to design a more flexible aggregation function by combining multiple simple aggregation functions, each of which is associatedwith a learnable weight.
However, the practical capacity of PNA is still limited by simply combining multiple simple aggregation functions.
A more expressive solution would be to model high-order non-linear interactions when aggregating node features.
However, explicitly modeling high-order non-linear interactions among nodes is very expensive, with both the time and memory complexity being exponential in the size of the neighborhood.
This raises the question of whether there existsan aggregation function which can model high-order non-linear interactions among nodes whileremaining computationally efficient.
In this paper, we propose such an approach based on symmetric tensor decomposition.
We designan aggregation function over a set of node representations for graph neural networks, which ispermutation-invariant and is capable of modeling non-linear high-order multiplicative interactionsamong nodes.
We leverage the symmetric CANDECOMP/PARAFAC decomposition (CP) [22, 29]to design an efficient parameterization of permutation-invariant multilinear maps over a set of noderepresentations.
Theoretically, we show that the CP layer can compute any permutation-invariantmultilinear polynomial, including the classical sum and mean aggregation functions.
We also showthat the CP layer is universally strictly more expressive than sum and mean pooling: with probabilityone, any function computed by a random CP layer cannot be computed using sum and mean pooling.
We propose the CP-layer as an expressive mean of performing the aggregation and update functionsin GNN.
We call the resulting model a tensorized GNN (tGNN).
We evaluate tGNN on both nodeand graph classification tasks.
Experimental results on real-world large-scale datasets show thatour proposed architecture outperforms or can compete with existing state-of-the-art approaches andtraditional pooling techniques.
Notably, our proposed method is more effective and expressive thanexisting GNN architectures and pooling methods on two citation networks, two OGB node datasets,and one OGB graph dataset.
Summary of Contributions We propose a new aggregation layer, the CP layer, for pooling andreadout functions in GNNs.
This new layer leverages the symmetric CP decomposition to efficientlyparameterize polynomial maps, thus taking into account high-order multiplicative interactions between node features.
We theoretically show that the CP layer can compute any permutation-invariantmultilinear polynomial including sum and mean pooling.
Using the CP layer as a drop-in replacementfor sum pooling in classical GNN architectures, our approach achieves more effective and expressiveresults than existing GNN architectures and pooling methods on several benchmark graph datasets.
Preliminaries2.1 Notation
We use bold font letters for vectors ( e.g., v ), capital letters ( e.g., M , T ) for matrices and tensorsrespectively, and regular letters for nodes ( e.g., v ).
Let G =
( V, E ) be a graph, where V is the nodeset and E is the edge set with self-loop.
We use N ( v ) to denote the neighborhood set of node v , i.e.,
N ( v ) =
{ u : e vu ∈ E } .
A node feature is a vector x ∈ R F defined on V , where x v is defined on thenode v .
We use ⊗ to denote the Kronecker product, ◦ to denote the outer product, and ⊙ to denotethe Hadamard ( i.e., component-wise) product between vectors, matrices, and tensors.
For any integerk , we use the notation [ k ] = { 1 , · · · , k } .
2.2 Tensors
We introduce basic notions of tensor algebra, more details can be found in [29].
A k -th order tensor
T ∈ R N 1 × N 2 × ...
× N k can simply be seen as a multidimensional array.
The mode- i fibers of Tare the vectors obtained by fixing all indices except the i -th one: T n 1 ,n 2 ,...,n i − 1 , : ,n i +1 ,...,n k ∈ R N i .
The i -th mode matricization of a tensor is the matrix having its mode- i fibers as columns and isdenoted by T ( i ) , e.g., T (1) ∈ R N 1 × N 2 ··· N k .
We use T × i v ∈ R N 1 ×···× N i − 1 × N i +1 ×···× N k to denotethe mode- i product between a tensor T ∈ R N 1 ×···× N k and a vector v ∈ R N i , which is definedby ( T × i v ) n 1 ,...,n i −
[Equation]
n i .
The following useful identity relates the 79 mode- i product with the Kronecker product:
[Equation]
2.3 CANDECOMP/PARAFAC Decomposition
We refer to C ANDECOMP/ P ARAFAC decomposition of a tensor as CP decomposition [24, 22].
A Rank R CP decomposition factorizes a k − th order tensor T ∈ R N 1 × ...
× N k into the sum of R
[Equation]
rank one tensors as T =
(cid:80)
Rr =1 v 1 r ◦ v 2 r ◦ · · · ◦ v kr , where ◦ denotes the vector outer-product andv 1 r ∈ R N 1 , v 2 r ∈ R N 2 , ..., v kr ∈ R N k for every r = 1 , 2 , ..., R .
The decomposition vectors, v : r for r = 1 , ..., R , are equal in length, thus can be naturally gathered into factor matrices M 1 =
[ v 11 , ..., v 1 R ] ∈ R N 1 × R , ..., M k
= [ v k 1 , ..., v kR ] ∈ R N k × R .
Using the factor matrices, we denote the CP decomposition of T as
[Equation]
The k -th order tensor T is cubical if all its modes have the same size, i.e., N 1 = N
= ...
= N k := N .
A tensor T is symmetric if it is cubical and is invariant under permutation of its indices:
[Equation]
for any permutation ϕ :
[ k ] → [ k ] .
A rank R symmetric CP decomposition of a symmetric tensor
T is a decomposition of the form T =
[[ M , · · · , M ]] with M ∈ R N × R .
It is well known that anysymmetric tensor admits a symmetric CP decomposition [12], we illustrate a rank R symmetric CPdecomposition in Fig. 1.
We say that a tensor T is partially symmetric if it is symmetric in a subset of its modes [28].
Forexample, a 3 -rd order tensor T ∈ R N 1 × N 1 × N 3 is partially symmetric w.r.t.
modes 1 and 2 if ithas symmetric frontal slices; i.e., T : , : ,k is a symmetric matrix for all k ∈ [ N 3 ] .
We prove the factthat any partially symmetric tensor admits a partially symmetric CP decomposition in Lemma 1in Appendix A, e.g., if T ∈ R N 1 × N 1 × N 3 is partially symmetric w.r.t.
modes 1 and 2, there exist
M ∈ R N 1 × R and W ∈ R N 3 × R such that T
=
[[ M , M , W ]] .
2.4 Graph Neural Networks and Pooling Functions
Given a graph G =
( V, E ) , a graph neural network always aggregates information in a neighborhoodto give node-level representations.
During each message-passing iteration, the embedding h vcorresponding to node v ∈ V is generated by aggregating features from N ( v ) [18].
Formally, at thel -th layer of a graph neural network,
[Equation]
where AGGREGATE ( l ) ( · ) and UPDATE ( l ) ( · ) are differentiable functions, the former beingpermutation-invariant.
In words, AGGREGATE ( l ) ( · ) first aggregates information from N ( v ) , then
UPDATE ( l ) ( · ) combines the aggregated message and previous node embedding h ( l − 1) v to give a newembedding.
The node representation h ( L ) v of node v from the last GNN layer L can be used for predicting relevantproperties of the node.
For graph classification, an additional READOUT ( · ) function aggregatesnode representations from the final layer to obtain a graph representation h G of graph G as,
[Equation]
where READOUT ( · ) can be a simple permutation-invariant function ( e.g., sum, mean, etc. ) or a moresophisticated graph pooling function [48, 46].
In the general design of GNNs, it is important to have an effective pooling function to aggregatemessages from local neighborhoods and update node representations (see Eq.
(2)), and to combinenode representations to compute a representation at the graph level (see Eq.
CP Layer
W
Input Features x y z
W W
W
Product Pooling
Layer x’ y’ z’
W x’
W y’
W z’
Activation
M
Linear Transformation via M
Linear Transformation via W =
Output Representation of 27 Elements
If Linear Activation = = 4-th Order Symmetric Tensor T } y } z } x
Tensorized Graph Neural Network
In this section, we introduce the CP-layer and tensorized GNNs (tGNN).
For convenience, we let{ x 1 , x 2 , ..., x k } denote features of a node v and its 1 -hop neighbors N ( v ) such that |{ v }∪ N ( v ) |
=
k .
3.1 Motivation and Method
We leverage the symmetric CP decomposition to design an efficient parameterization of permutationinvariant multilinear maps for aggregation operations in graph neural networks, Tensorized Graph
Neural Network (tGNN), resulting in a more expressive high-order node interaction scheme.
Wevisualize the CP pooling layer and compare it with sum pooling in Fig.
Let T ∈ R N × N ×···× N × M of order k + 1 be a tensor which is partially symmetric w.r.t.
its firstk modes.
We can parameterize T using a rank R partially symmetric CP decomposition (see
Section 2.3): T =
[[ W , · · · , W , M ]] where W ∈ R N × R and M ∈ R M × R .
Such a tensor naturallydefines a map from ( R N ) k to R M using contractions over the first k modes:
[Equation]
This map satisfies two very important properties for GNNs: it is permutation-invariant (due tothe partial symmetry of T ) and its number of parameters is independent of k (due to the partiallysymmetric CP parameterization).
Thus, using only two parameter matrices of fixed size, the map in
Eq.
(4) can be applied to sets of N -dimensional vectors of arbitrary cardinality.
In particular, we willshow that it can be leveraged to replace both the AGGREGATE and UPDATE functions in GNNs.
There are several way to interpret the map in Eq.
First, from Eq.
(1) we have
[Equation]
where T ( k +1) ∈ R M × N k is the mode- ( k + 1) matricization of T .
This shows that each element ofthe output f ( x 1 , · · · , x k ) is a linear combinations of terms of the form ( x 1 ) i 1 ( x 2 ) i 2 · · · ( x k )
i k
( k -thorder multiplicative interactions between the components of the vectors x 1 , · · · , x k ).
That is, f is amultivariate polynomial map of order k involving only k -th order interactions.
By using homogeneouscoordinates, i.e., appending an entry equal to one to each of the input tensors x i , the map f becomesa more general polynomial map taking into account all multiplicative interactions between the x i upto the k -th order:
[Table]
where T is now of size ( N +1)
×· · ·× ( N +1)
× M and can still be parameterized using the partially symmetric CP decomposition T =
[[ W , · · · , W , M ]] with W ∈ R ( N +1) × R and M ∈ R M × R .
With this parameterization, one can check that
[Equation]
where ⊙ denotes the component-wise product between vectors.
The map f can thus be seen as thecomposition of a linear layer with weight W , a multiplicative pooling layer, and another linear map
M .
Since it is permutation-invariant and can be applied to any number of input vectors, this map canbe used as both the aggregation, update, and readout functions of a GNN using non-linear activationfunctions, which leads us to introduce the novel CP layer for GNN.
Definition 1.
(CP layer)
Given parameter matrices M ∈ R d × R and W ∈ R F +1 × R and activationfunctions σ, σ ′ , a rank R CP layer computes the function f CP : ∪ i ≥ 1 ( R F ) i → R d defined by
[Equation]
for any k ≥ 1 and any x 1 , · · · , x k ∈ R F .
The rank R of a CP layer is a hyperparameter controlling the trade-off between parameter efficiencyand expressiveness.
Note that the CP layer computes AGGREGATE and UPDATE (see Eq. 2) inone step.
One can think of the component-wise product of the W ⊤ [ x i 1] ⊤ as AGGREGATE, whilethe UPDATE corresponds to the two non-linear activation functions and linear transformation M .
We observed in our experiments that the non-linearity σ is crucial to avoid numerical instabilitiesduring training caused by repeated products of W .
In practice, we use Tanh for σ and ReLU for σ ′ .
Fig.
2 graphically explains the computational process of a CP layer, comparing it with a classicalsum pooling operation.
We intuitively see in this figure that the CP layer is able to capture high ordermultiplicative interactions that are not modeled by simple aggregation functions such as the sum orthe mean.
In the next section, we theoretically formalize this intuition.
Complexity Analysis
The sum, mean and max poolings result in O ( F in ( N + F out )) time complexity, while CP pooling is O ( R ( NF in + F out ) ) , where N denotes the number of nodes, F in is the inputfeature dimension, F out is out feature dimension, and R is the CP decomposition rank.
In Sec.
5.3,we experimentally compare tGNN and CP pooling with various GNNs and pooling techniques toshow the model efficiency with limited computation and time budgets.
3.2 Theoretical Analysis
We now analyze the expressive power of CP layers.
In order to characterize the set of functions thatcan be computed by CP layers, we first introduce the notion of multilinear polynomial.
A multilinearpolynomial is a special kind of vector-valued multivariate polynomial in which no variables appearswith a power of 2 or higher.
More formally, we have the following definition.
Definition 2.
A function g : R k
→ R is called a univariate multilinear polynomial if it can be written as
[Equation]
where each τ i 1 i 2 ··· i k ∈ R .
The degree of a univariate multilinear polynomial is the maximum numberof distinct variables occurring in any of the non-zero monomials τ
i 1 i 2 ··· i n a i 1 1 a i 2 2 · · · a i k k .
A function f : ( R d ) k
→ R p is called a multilinear polynomial map if there exist univariate multilinear polynomials
g i,j 1 , ··· ,j n for j 1 , · · · , j k ∈ [ d ] and i ∈
[ p ] such that
[Equation]
for all x 1 , · · · , x k ∈ R d and all i ∈ [ p ] .
The degree of f is the highest degree of the multilinearpolynomials g i,j 1 , ··· ,j k . functionspace function space} ⊇ {permutation-invariant multilinearpolynomial space} ⊇ {sum and mean aggregation functions}.
The following theorem shows that CP layers can compute anypermutation-invariant multilinear polynomial map.
We also visually represent the expressive power of CP layers in Fig.
3, showing that the classof functions computed by CP layer subsumes multilinear polynomials(including sum and mean aggregation functions).
Theorem 1.
The function computed by a CP layer (Eq.
(1) ) ispermutation-invariant.
In addition, any permutation-invariant multilinearpolynomial f : ( R F ) k
→ R d can be computed by a CP layer (with alinear activation function).
Note also that in Fig.
3 the CP layer is strictly more expressive thanpermutation invariant multilinear polynomials due to the non-linear activation functions in Def. 1.
Since the classical sum and mean poolingaggregation functions are degree 1 multilinear polynomial maps, it readilyfollows from the previous theorem that the CP layer is more expressivethan these standards aggregation functions.
However, it is natural to askhow many parameters a CP layer needs to compute sums and means.
Weanswer this question in the following theorem.
Theorem 2.
A CP layer of rank F · k can compute the sum and meanaggregation functions over k vectors in R F .
Consequently, for any k ≥ 1 and any GNN N using mean or sum pooling with feature and embeddingdimensions bounded by F , there exists a GNN with CP layers of rank F · k computing the samefunction as N over all graphs of uniform degree k .
It follows from this theorem that a CP layer with 2 F 2 k can compute sum and mean aggregation oversets of k vectors.
While Theorem 2 shows that any function using sum and mean aggregation can becomputed by a CP layer, the next theorem shows that the converse is not true, i.e., the CP layer is astrictly more expressive aggregator than using the mean or sum.
Theorem 3.
With probability one, any function f CP : ( R F ) k
→ R d computed by a CP layer (of any rank) whose parameters are drawn randomly (from a distribution which is continuous w.r.t.
the
Lebesgue measure) cannot be computed by a function of the form
[Equation]
where M ∈ R d × R , W ∈ R F × R and σ , σ ′ are component-wise activation function.
This theorem not only shows that there exist functions computed by CP layers that cannot be computedusing sum pooling, but that this is the case for almost all functions that can be computed by (evenrank-one) CP layers.
From an expressive power viewpoint, we showed that a CP layer is able to leverage both low and highorder multiplicative interactions.
However, from a learning perspective, it is clear that the CP layer hasa natural inductive bias towards capturing high-order interactions.
We are not enforcing any sparsityin the tensor parameterizing the polynomial, thus the number and magnitude of weights correspondingto high-order terms will dominate the result (intuitively, learning a low order polynomial would implysetting most of these weights to zero).
In order to counterbalance this bias, we complement the CPlayer with simple but efficient linear low-order interactions (reminiscent of the idea behind residualnetworks [21]) when using it in tGNN:
[Equation]
where the first term corresponds to the CP layer and the second one to a standard sum poolinglayer (with σ , σ ′ and σ ′′ being activation functions).
Related Work and Discussion
We now discuss relevant work on the parameterization of tensors on graph neural networks.
Ingeneral, our work relates to three areas of deep learning: (1) aggregation functions, (2) universalapproximator for set aggregation, (3) high-order pooling, and (4) tensor methods for deep learning.
GNN & Aggregation Scheme [26] successfully define convolutions on graph-structured databy averaging node information in a neighborhood.
[46] prove the incomplete expressivity of meanaggregation to distinguish nodes, and further propose to use sum aggregation to differentiate nodeswith similar properties.
[13] further generalize this idea and show that mean aggregation can bea particular case of sum aggregation with a linear multiplier, and further propose an architecture withmultiple aggregation channels to adaptively learn low-order information.
Most GNNs use low-orderaggregation schemes for learning node representations.
To the best of our knowledge, tGNN is thefirst GNN architecture that adopts high-order multiplicative interactions in aggregation.
Universal Approximator & Permutation-invariant NN Universal approximators for set aggregation functions have been previously proposed and studied.
[49] show that sum pooling is enoughprovided that it is combined with two universal approximators.
[46] further discuss the limitations ofnon-injective set function.
[25] propose a generalization of transformers to permutation-invariant sets.
Most universal approximation results for GNN relies on combining simple aggregation functions(e.g.
sum, mean) with universal approximators for the feature and output maps.
In contrast, the CPlayer achieves the same goal of being an universal approximator but using a different mean: explicitcomputation of multilinear polynomials (in an effective manner using the CP parameterization).
While the approaches might be expressively similar, they may differ from a learning aspect, i.e., howeasy it is for these models to learn a function from data.
High-order Pooling [2] formulate the pooling problem as a multiset encoding problem withauxiliary information about the graph structure, and propose an attention-based pooling layer thatcaptures the interaction between nodes according to their structural dependencies.
[44] apply secondorder statistic methods because the use of second-order statistics takes advantage of the Riemanniangeometry of the space of symmetric positive definite matrices.
However, our CP pooling can not onlycapture long-range dependencies but also result in high-dimensional multiplicative feature products.
Tensor Methods Tensor methods allow one to define meaningful geometries to build more expressive models.
Authors in [50, 30] define geometries of tensor networks on complex or hypercomplexmanifolds, their models encompass greater freedom in the choice of the product between the algebraelements.
[7] develop a general framework for both probabilistic and neural models for tree-structureddata with a tensor-based aggregation function.
[6] leverages permutation-invariant CP-based aggregation function to capture high-order interactions in NLP tasks.
Using multiplicative interactionsas a powerful source of non-linearities in neural network models have been studied previously forconvolutional [11, 10, 31] and recurrent networks [39, 42, 45].
The connection between such multiplicative interactions with tensor networks have been leveraged both from theoretical and practicalperspectives.
The CP layer can be seen as a graph generalization of the convolutional and recurrentarithmetic circuits considered in [10] and [39], respectively.
Experiments on Real-World Datasets
In this section, we evaluate Tensorized Graph Neural Net on real-world node- and graph-leveldatasets.
We introduce experiment setup in 5.1, compare tGNN with the state-of-the-arts models in5.2, and conduct ablation study on model performance and efficiency in 5.3.
The hyperparameter andcomputing resources are attached in Appendix L. Dataset information can be found in Appendix K.
5.1 Experiment Setup
In this work, we conduct experiments on three citation networks ( Cora , Citeseer , Pubmed ) andthree OGB datasets ( PRODUCTS , ARXIV , PROTEINS ) [23] for node-level tasks, one OGB dataset( MolHIV ) [23] and three benchmarking datasets ( ZINC , CIFAR10 , MNIST ) [14] for graph-level tasks.
Training Procedure For three citation networks ( Cora , Citeseer , Pubmed ), we run experiments 10times on each dataset with 60%/20%/20% random splits used in [9], and report results in Tab. 1.
Fordata splits of OGB node and graph datasets, we follow [23], run experiments 5 times on each dataset(due to training cost), and report results in Tab. 1, 2.
For benchmarking datasets, we run experiments5 times on each dataset with data split used in [14], and report results in Tab. 2.
To avoid numericalinstability and floating point exception in tGNN training, we sample 5 neighbors for each node.
Forgraph datasets, we do not sample because the training is already in batch thus numerical instabilitycan be avoided, and we apply the CP pooling at both node-level aggregation and graph-level readout.
Model Comparison tGNN has two hyperparameters, hidden unit and decomposition rank, we fixhidden unit and explore decomposition rank.
For citation networks, we compare 2-layer GNNs with32 hidden units.
And for OGB and benchmarking datasets, we use 32 hidden units for tGNN, and theresults for all other methods are reported from the leaderboards and corresponding references.
Particularly, tGNN and CP pooling are more effective and expressive than existing pooling techniquesfor GNNs on two citation networks, two OGB node datasets, and one OGB graph dataset in Tab. 1, 2.
5.2 Real-world Datasets
In this section, we present tGNN performance on node- and graph-level tasks.
We compare tGNNwith several classic baseline models under the same training setting.
For three citation networks, wecompare tGNN with several baselines including GCN [26], GAT [43], GraphSAGE [19], H 2 GCN[52], GPRGNN [9], APPNP [27] and MixHop [1]; for three OGB node datasets, we compare tGNNwith MLP, Node2vec [17], GCN [26], GraphSAGE [19] and DeeperGCN [32].
And for graph-leveltasks, we compare tGNN with several baselines including MLP, GCN [26], GIN [46], DiffPool [48],
GAT [43], MoNet [35], GatedGCN [4], PNA [13], PHMGNN [30] and DGN [3].
The model choiceis because we propose a new pooling method and want to mainly compare tGNN with other poolingsin standard GNN architectures.
Current leading models on OGB leaderboards [23] adopt transformer,equivariant, fingerprint, C&S, or others, which are more complex than standard GNNs.
We visualizethe performance boost and comparisons with GNNs and pooling techniques in Tab. 1,2.
[Table]
283 From Tab. 1, we can observe that tGNN outperforms all classic baselines on Cora , Pubmed , PRODUCTS and ARXIV , and have slight improvements on the other datasets but underperforms GCN on
Citeseer and DeeperGCN on PROTEINS .
On the citation networks, tGNN outperforms others on 2out of 3 datasets.
Moreover, on the OGB node datasets, even when tGNN is not ranked first, it is stillvery competitive (top 3 for all datasets).
We believe it is reasonable and expected that tGNN does notoutperform all methods on all datasets.
But overall tGNN shows very competitive performance anddeliver significant improvement on challenging graph benchmarks compared to popular commonlyused pooling methods (with comparable computational cost).
in comparison with GNN architectures.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
In Tab. 2, we present tGNN performance on graph property prediction tasks.
tGNN achieves stateof-the-arts results on MolHIV , and have slight improvements on other three Benchmarking graphdatasets.
Overall tGNN achieves more effective and accurate results on 5 out of 10 datasets comparingwith existing pooling techniques, which suggests that high-order CP pooling can leverage a GNN togeneralize better node embeddings and graph representations.
5.3 Ablation and Efficiency Study
In the ablation study, we first investigate the effectiveness of having the high-order non-linear CPpooling and adding the linear low-order transformation in Tab. 3, then investigate the relations ofthe model performance, efficiency, and tensor decomposition rank in Fig. 4.
Moreover, we comparetGNN with different GNN architectures and aggregation functions to show the efficiency in Tab. 5, 7by showing the number of model parameters, computation time, and accuracy.
In Tab. 3, we test each component, high-order non-linear CP pooling, low-order linear sum pooling,and two pooling techniques combined, separately.
We fix 2-layer GNNs with 32 hidden unit and 64decomposition rank, and run experiments 10 times on Cora and Pubmed with 60%/20%/20% randomsplits used in [9], run ZINC 5 times with 10,000/1,000/1,000 graph split used in [14].
From the results, we can see that adding the linear low-order interactions helps put essential weightson them.
Ablation results show that high-order CP pooling has the advantage over low-order linearpooling for generating expressive node and graph representations, moreover, tGNN is more expressivewith the combination of high-order pooling and low-order aggregation.
This illustrates the necessityof learning high-order components and low-order interactions simultaneously in tGNN.
Computational Aspect In Fig.
4, we compare model performance with computation costs.
Wefix a 2-layer tGNN with 32 hidden channels with 8,16,...,2048,4096 rank, and 2-layer baselineswith 8,16,...,2048,4096 hidden dim.
We run 10 times on each citation network with the same60%/20%/20% random splits for train/validation/test and draw the relations of average test accuracy,rank/hidden dim, and computation costs.
From the figure, we can see that the model performancecan be improved with higher ranks ( i.e., Tensor T is more accurately computed as the rank R getslarger), but training time is also increased, thus it is a trade-off between classification accuracy andcomputation efficiency.
And in comparison with baselines, tGNN still has marginal improvementswith higher ranks while the baselines stop improving with larger hidden dimensions.
In Sec. 3.1, we theoretically discuss the time complexity of tGNN.
In Appendix I, we experimentallyassess model efficiency by comparing tGNN with GCN [26], GAT [43], GCN2 [8], and mean, maxpoolings on Cora on a CPU over 10 runtimes, and compare the number of model parameters, trainingepochs per second, and accuracy.
The experiments show that tGNN is more competitive in terms ofrunning time and better accuracy with a fixed number of parameters and the same time budget.
Conclusion and Future Work
In this paper, we theoretically develop a high-order permutation-invariant multilinear map for nodeaggregation and graph pooling via tensor parameterization.
We show its powerful ability to computeany permutation-invariant multilinear polynomial including sum and mean pooling functions.
Experiments demonstrate that tGNN is more effective and accurate on 5 out of 10 datasets, showcasing therelevance of tensor methods for high-order graph neural network models.
For future work, one interesting direction is to augment various GNN architectures with non-linearhigh-order CP layers.
Most of the existing GNN models adopt low-order aggregation and poolingfunctions, it would be interesting to equip current state-of-the-art models with high-order poolingfunctions for a potential performance boost.
Another interesting direction is to enhance tGNN withan adaptive channel mixing mechanism.
In tGNN with high-order and low-order pooling functions,one node receives two channels of information combined in a linear way.
The linear combinationmay not be sufficient to balance and extract high-order components and low-order components, so itwould be interesting to design an adaptive channel mixing tGNN model for learning from differentnode-wise components.
A Proof of Theorem 1
In order to prove this theorem, we first need the following lemma showing the existence of partiallysymmetric CP decompositions.
Lemma 1.
Any partially symmetric tensor admits a partially symmetric CP decomposition.
Proof.
We show the results for 3-rd order tensors that are partially symmetric w.r.t.
their two firstmodes.
The proof can be straightforwardly extended to tensors of arbitrary order that are partiallysymmetric w.r.t.
any subset of modes.
Let T ∈ R m × m × n be partially symmetric w.r.t.
modes 1 and 2 .
We have that T : , : ,i is a symmetric tensor for each i ∈ [ n ] .
By Lemma 4.2 in [12], each tensor T : , : ,i admits a symmetric CP decomposition:
[Equation]
where A ( i ) ∈
R m × R i and R i is the symmetric CP rank of T : , : ,i .
By defining R = (cid:80) ni =1 R i and A =
[ A (1) A (2) · · · A ( n ) ]
∈ R m × R , one can easily check that T admits the partially symmetric CP decomposition T
=
[[ A , A , ∆ ]] where ∆ ∈ R m × R is defined by ∆ i,r = (cid:26)if R 1 + · · · + R i − 1 < r ≤ R 1 + · · · + R i otherwise.
We can now prove Theorem 1.
Theorem.
The function computed by a CP layer (Eq.
(1) ) is permutation-invariant.
In addition, anypermutation-invariant multilinear polynomial f : ( R F ) k
→ R d can be computed by a CP layer (withlinear activation function).
Proof.
The fact that the function computed by a CP layer is permutation-invariant directly followsfrom the definition of the CP layer and the fact that the Hadamard product is commutative.
We now show the second part of the theorem.
Let f : ( R F ) k
→ R d be a permutation-invariant multilinear polynomial map.
Then, there exists permutation-invariant univariate multilinear polynomials g i,j 1 , ··· ,j k for j 1 , · · · , j k ∈
[ F ] and i ∈ [ d ] such that
[Equation]
Moreover, by definition, each such polynomial satisfies
[Equation]
for some scalars τ ( i,j 1 , ··· ,j k )
i 1 , ··· ,i k .
Putting those two expressions together, we get
[Equation]
We can then group together the coefficients τ ( i,j 1 , ··· ,j k ) i 1 , ··· ,i k corresponding to the same monomials ( x 1 ) i 1 j 1 · · · ( x k ) i k j k .
E.g., the coefficients τ ( i,j 1 , ··· ,j k ) 1 , 0 , ··· ,all correspond to the same monomial ( x 1 ) jfor any values of j 2 , j 3 , · · · , j k .
Similarly, all the coefficients τ ( i,j 1 , ··· ,j k ) 0 , 1 , ··· ,all correspond to the same monomial ( x 2 ) j 2 · · · ( x k ) j k for any values of j 1 .
By grouping and summing together these coefficients into a tensor T ∈ R ( F +1) ×···× ( F +1) × d , where T j 1 , ··· ,j k ,i is equal to the sum (cid:88)
j
[Equation]
∈
[Equation]
]: j
[Equation]
i ℓ
[Equation]
τ
[Equation]
i
[Equation]
,i k , we obtain
[Equation]
Since f is permutation-invariant, the tensor T is partially symmetric w.r.t.
its first k modes.
Thus, by
Lemma 1, there exist matrices A ∈ R R × F and B ∈ R d × R such that
[Equation]
with σ being the identity function, which concludes the proof.
B Proof of Theorem 2
Theorem.
A CP layer or rank F · k can compute the sum and mean aggregation functions for kvectors in R F .
Consequently, for any k ≥ 1 and any GNN N using mean or sum pooling with feature and embeddingdimensions bounded by F , there exists a GNN with CP layers of rank F · k computing the samefunction as N over all graphs of uniform degree k .
Proof.
We will show that the function f : (cid:0) R F (cid:1) k
→ R d defined by
[Equation]
where α ∈ R , can be computed by a CP layer of rank Fk , which will show the first part of thetheorem.
The second part of the theorem directly follows by letting α = 1 for the sum
aggregationand α
= 1 /k for the mean aggregation.
Let ˜ e 1 , · · · , ˜ e F +1 be the canonical basis of R F +1 , and let e 1 , · · · , e F be the canonical basis of R F .
We define the tensor T ∈ R ( F +1) ×···×
[Equation]
F +1)
×
d by
[Equation]
where ◦ denotes the outer (or tensor) product between vectors
[Equation]
We start by showing that contracting kvectors in homogeneous coordinates along the first k modes of T results in the sum of those vectorsweighted by α .
For any vectors x 1 , · · · , x k ∈ R F
[Equation]
we have
[Equation]
To show that f can be computed by a CP layer of rank Fk , it thus remains to show that T admits a partially symmetric CP decomposition of rank Fk .
This follows from Corollary 4.3 in [51], which states that any k th order tensor A of CP rank less than k has symmetric CP rank bounded by k .
Indeed, consider the tensors
[Equation]
for j ∈ [ d ] .
They are all k -th order tensor of CP rank bounded by k .
Thus, by Corollary 4.3 in [51],they all admit a symmetric CP decomposition of rank at most k , from which it directly follows thatthe tensor T
= (cid:80) Fj =1 A ( j ) ◦
e j admits a partially symmetric CP decomposition of rank at most
Fk .
C Proof of Theorem
Theorem.
With probability one, any function f CP : ( R F ) k
→ R d computed by a CP layer (of any rank) whose parameters are drawn randomly (from a continuous distribution) cannot be computed by a function of the form
[Equation]
where M ∈ R d × R , W ∈ R F × R and σ , σ ′ are component-wise activation function.
Proof.
Let f CP : ( R F ) k
→ R d be the function computed by a random CP layer.
I.e.,
[Equation]
where the entries of the matrices B ∈ R ( N +1) × R and A ∈ R M × R are identically and independentlydrawn from a distribution which is continuous w.r.t.
the Lebesgue measure.
It is well know thatsince the entries of the two parameter matrices are drawn from a continuous distribution, all theentries of A and B are non-zero and distinct with probability one.
It follows that all the entriesof the vector f CP ( x 1 , · · · , x k ) are k -th order multilinear polynomials of the entries of the inputvectors x 1 , · · · , x k , which, with probability one, have non trivial high-order interactions that cannotbe computed by a linear map.
In particular, with probability one, the map f CP cannot be computedby any map of the form g sum : x 1 , · · · , x k
(cid:55)→ σ ′ (cid:16) M (cid:16) σ (cid:16)(cid:80) ki =1 W ⊤ x i (cid:17)(cid:17)(cid:17) since the sum poolingaggregates the inputs in a way that prevents modeling independent higher order multiplicativeinteractions, despite the non-linear activation functions.
D Meaning of High-Order of tGNN
In general content of GNNs, high-order will normally mean the ability of capturing long-rangedependencies, the interactions between long-range nodes.
However, in our paper, high-order meansmulti-dimensional feature products.
Like other simple GNNs, the CP layer is able to capture stackinglong-range dependencies by stacking layers.
For example, if we have x 1 and x 2 , a simple GNN layercan result in a vector of [ x 1 , x 2 , x 1 + x 2 ] , but our high-order layer result in a vector of [ x 1 , x 2 , x 1 +x 2 , x 1 x 2 ] with another term x 1 x 2 .
And if we have we have x 1 , x 2 , and x 3 , a simple low-order GNNlayer will result in [ x 1 , x 2 , x 3 , x 1 + x 2 , x 1 + x 3 , x 2 + x 3 , x 1 + x 2 + x 3 ] while the high-order layer willresult in a vector of [ x 1 , x 2 , x 3 , x 1 + x 2 , x 1 + x 3 , x 2 + x 3 , x 1 + x 2 + x 3 , x 1 x 2 , x 1 x 3 , x 2 x 3 , x 1 x 2 x 3 ]with extra 4 terms at a higher feature dimension.
One may raise a question on why we need to compute multi-dimensional feature products.
The ideais fairly easy if one thinks the multi-dimensional feature space.
Take a molecular graph dataset as anexample, and blood pressure could be one feature.
Many studies analyze the impact of blood pressureon one’s health.
Blood pressure is the feature in this case.
So, one could try to predict life span usingblood pressure as a feature, but you can imagine that this might not work very well.
You really needmore information.
So you could add cholesterol level, resting heart rate, and body-mass index to seeif you can predict a person’s life span.
Those additional measurements are also features and now every individual has a multi-dimensional feature vector consisting of these measurements.
However, asimple GNN with one-dimensional feature interactions may not capture the correlation between bloodpressure and cholesterol level.
There could be some correlation between blood pressure, cholesterollevel, and body-mass index to predict life span.
A simple GNN that does blood pressure+cholesterollevel+body-mass index is not able to capture the correlation between those factors.
Still, our multidimensional feature product can capture blood pressurecholesterol level, blood pressurebody-massindex, cholesterol levelbody-mass index, blood pressurecholesterol levelbody-mass index all thecorrelations to predict life span.
So I believe for molecular graphs, high-dimensional feature productsare importantly needed to make a prediction as most features are correlated (like in atom graphs,atom charge, and atom mass), and our model shows some significant improvements on moleculardatasets.
But such correlation might not be that important on citation networks or social networks,so the improvements are marginal.
Therefore, for molecular graphs, it is not a waste to computehigh-dimensional feature products while sacrificing computational costs.[36, 41] are previous works on high-dimensional feature products, and introduce how to use tensormethods to achieve those products.
They have stated the importance of having high-order products inphysics and geometries.
E Rank and Performance
Generally speaking, the low-rank decomposition methods will sacrifice some expressivity for somecomputation costs, so as we increase the rank, we will re-gain some expressivity but lose somecomputation costs.
As talked about in our paper, it is a trade-off between expressivity (modelperformance) and computation costs.
This idea can be found in previous works [37, 38, 5] where theytest on computer vision datasets (with visualization of recovering images from noises, a high-rankmodel can always better recover an image than a low-rank model does).
In this work, we use the low-rank decomposition technique which can potentially reduce the numberof learnable parameters by a lower number of ranks.
The low-rank method saves the computationcosts but the use of low-rank decomposition for learnable parameters will sacrifice some expressivityin computation.
If we increase the rank, the learnable parameters can be principally recovered, inthis case, we no longer sacrifice its expressivity but computation costs (as the Figures shown in theablation study).
In computer vision, low-rank decomposition methods were introduced to replace a fully-connectedlayer.
And in their papers [37, 38, 5] (in both theory and experiments), they show that a model isbetter with higher ranks because higher ranks can lead to higher expressivity than low ranks do.
Theyconclude and show that the model performance is positively correlated to the tensor decompositionrank.
In [37, 38, 5], they experimentally show that higher ranks can result in lower test errors asthe model becomes more expressive (plotting a curve of decomposition rank or model parametervs.
error).
Moreover, in [37, 38], they visualize the relationship between the model performanceand model rank.
They aim to recover the original pictures from noises with different tensor ranks(from low to high), and the recovered pictures are more clear with high ranks than with low ranks.
Toconclude our findings in the ablation study and previous works on low-rank decomposition, high-rankmodels can usually have better performance and stronger expressivity than low-rank models, andhigher ranks can usually lead to better performance (in terms of high accuracy or low error) than lowranks do.
F Low and High Terms
The reason we introduce the inductive bias is that we want to balance the importance of low- andhigh-order terms.
We design an experiment to show which interactions play the main role.
We havetwo learnable attention weights a low , a high , one before high-order terms and the other one beforelow-order terms, node representation H = a high H high + a low H low with a low + a high = 1 .
Ifhigh-order interactions dominate a high will be larger, else if low-order interactions dominate a lowwill be larger.
At the early model learning stage, a high is a lot higher ( ∼ 0 . 9 ) than a low ( ∼ 0 . 1 ),which show high-order terms dominate at the beginning, and a high , a low will converge to a similarvalue ( ∼ 0 . 5 ) as the model improves, which show that low- and high-order terms have the similarimportance.
G GCN Performance
We do see in our Tables, GCN can beat popular baselines.
It is not really weird, if you can do a fullhyperparameter search (either GCN built on PyG or Torch), you’ll find out that GCN can outperformthe most popular GNN architectures (after GCN gets proposed).
And the papers [34, 33] find outthe same experimental phenomenon.
The question, that GCN can outperform a lot GNNs on many(heterophilic or homophilic) datasets after thorough hyperparameter tuning, was discussed in thecommunity before but never raised attention.
H Additional Graph Classification
[2] formulate the graph pooling problem as a multiset encoding problem with auxiliary information
[Table]
about the graph structure, and propose an attention-based pooling layer that captures the interaction [Table] between nodes according to their structural dependencies.
[44] apply second-order statistic methodsbecause the use of second-order statistics takes advantage of the Riemannian geometry of the space ofsymmetric positive definite matrices.
Their formulation adapts the bilinear pooling, and the bilinearmapping is capable of capturing second-order statistics and topology information.
We test and compare tGNN with [2, 44] following the setting of [15].
[Table]
The table records graph classification results on test sets for 5 graph-level tasks under the setting of[15].
We can see that our model makes improvements on 4 out of 5 datasets (big improvements on
MUTAG, PROTEINS, and COLLAB, and small improvements on IMDB-B).
I Efficiency Study
We study efficiency by comparing tGNN with models on Cora on a CPU over 10 runtimes, andcompare the number of model paramterts, number of training epochs per second, and accuracy.
Sampling means we sample ’3’ neighbors for each node or we use ’Full’ neighborhood.
Notice thatthe average node degree of Cora is 3.9, which means if a node has a number of neighbors less than10, some of its neighbor nodes will get resampled until it hits 10.
From Table 6, we can see that the model performance is not heavily affected by the number ofsampled neighbor nodes because the average node degree is not that high on the majority of networkdatasets, and ’3’ or ’5’ would be sufficient.
J Emperical Study on Classical Poolings
We study and compare CP pooling with classical pooling methods on Cora on a CPU over 10 runtimes,and compare the number of model paramterts, number of training epochs per second, and accuracy.
Sampling means we sample ’3’ neighbors for each node or we use ’Full’ neighborhood.
[Table]
[Table]
[Table]
K
Dataset
A more detailed of real
[Table] [Table]
L Hyperparameter
For experiments on real-world datasets, we use NVIDIA P100 Pascal as our GPU computationresource.
And the searching hyperparameter includes the learning rate, weight decay, dropout, decompositionrank R .
[Table] [Table]
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes](c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[No](b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[No]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[No]