G RASSMANN G RAPH E MBEDDING
A BSTRACT
Geometric deep learning that employs the geometric and topological features of data has attracted increasing attention in deep neural networks.
Learning the intrinsic structure property of data is a crucial step for dimensionality reduction and effective feature extraction.
This paper develops Grassmann graph embedding , which combines graph convolutions to capture the main components within graphs’ hidden representations.
Each set of featured graph nodes is mapped to a point on a Grassmann matrix manifold through Singular Value Decomposition, which is then embedded into a symmetric matrix space that approximates denoised second-order feature information.
The view of treating nodes as a set could inspire many potential applications.
In particular, we propose Grassmann (global graph) pooling that can connect with any graph convolution for graph neural networks.
The Grassmann pooling achieves state-of-the-art performance on a variety of graph prediction benchmarks.
I NTRODUCTION
Graphs are prevalent objects in machine learning and data science.
Graph representation learning has recently attracted increasing attention with graph neural networks (GNNs) emerging as one of the most prominent avenues in deep learning (Bronstein et al., 2017; Hamilton, 2020).
One critical ingredient of graph representation like GNNs is to effectively distill key features of graph data.
Graph convolution (Bruna et al., 2014) to GNN, or more generally, graph neural message passing (Gilmer et al., 2017), provides an efﬁcient way to extract core information of a graph.
In node prediction tasks, graph convolution is an excellent learner that aggregates graph features and uses node relation to predict node labels.
On the contrary, graph-level learning tasks are usually deﬁned on graph data with varying size and structure, in which case dimensionality reduction (for graph size) is often an indispensable necessity.
To this end, a graph pooling scheme is critical to obtaining scaled-down graphs or graph-level embeddings.
For example, T OP KP OOL (Cˇatˇalina et al., 2018) exploits a node selection criterion by a learned score and drops all but the top-ranked nodes;
SAGP OOL (Lee et al., 2019) calculates a score with an extra MLP in the T OP K framework to enhance the selection efﬁcacy; and A TTENTION P OOL (Li et al., 2016) deﬁnes a soft scoring mechanism that concatenates node state and annotation.
One basic property of a pooling or embedding operation is node permutation invariance, as the node order of an undirected graph must not incline the network to focus on speciﬁc information. The
[Figure] permutation invariance is intuitive in heuristic pooling operations like summation, averaging, and maximization, where the aggregation rules do not rely on the order information of nodes.
Indeed we can deﬁne such operations on subsets of a graph.
The view of treating nodes as a set in pooling can be seen in the recent work of Kolouri et al.
(2021), where a node set ﬁnds its Wasserstein Embedding within a linear optimal transport framework, and match each pair of nodes by a proper Wasserstein metric (M ´ emoli, 2011).
When graph nodes come with features, we may view the set of graph nodes from the perspective of manifold learning.
The set of feature vectors naturally generates a subspace as a new representation.
In other words, the node information of a graph can be abstracted into a lower-dimensional subspace, and a learning task over graphs of different size and structure can thus be converted into a learning task on a Grassmann manifold (Absil et al., 2008).
We name this process of mapping a graph onto the
Grassmann manifold as Grassmann (graph) embedding .
The acquired Grassmann embeddings from graphs can be pipelined into a Grassmann learning algorithm, such as Deep Grassmann Networks (Huang et al., 2018) and Grassmann clustering (Wang et al., 2014; 2017).
As a proof of concept, this work develops a simple manifold embedding strategy (Figure 1).
An input graph is ﬁrst passed through Graph Convolutional layers to extract the hidden representation of its node information, which is then embedded as a Grassmann point with a subspace dimension by Singular Value Decomposition (SVD).
All Grassmann points that represent corresponding graphs are on the same Grassmann manifold, and they can be projected to Euclidean space in the form of symmetric matrices (Absil et al., 2008), where all graphs acquire the same size representation.
The vectorized representations can be sent to a conventional learning model for any learning task such as graph classiﬁcation or clustering.
The architecture design is one practice of exploiting the Grassmann representation as a pooling or embedding component.
Based on this, we propose
Grassmann Graph Pooling (G R P OOL ), a feasible graph pooling mechanism for graph data in varying size and structure.
The strategy can also inspire other learning tasks, such as graph coarsening and graph clustering.
G RASSMANN GRAPH EMBEDDING
We can embed multiple graphs to one Grassmann manifold using their graph convolutional network representation.
For a given set of N graphs G = {G 1 , . . .
, G N } , a sequence of graph convolutional layers like GCN (Kipf & Welling, 2017) and
GIN (Xu et al., 2019) maps them to a set of hidden representations H
=
{ H 1 , . . .
, H N } where H i ∈ R n i × m is with respect to n i nodes in G i , and m denotes the number of hidden units in the last graph convolutional layer.
To conduct graphlevel learning tasks, one needs to ﬁnd a uni-dimensional graph representation that is independent of the node size.
We accomplish the mission by transforming each graph representation of H to a Grassmann point, which can later be projected back to its Euclidean form for regular output layers.
Grassmann manifold A Grassmann manifold M ( d, m ) is the space of all d -dimensional linear subspaces of R m for 0 ≤ d ≤ m .
A point on M ( d, m ) is a d -dimensional subspace of R m , which can be represented by any orthonormal basis U =
[ u 1 , u 2 , . . . ,
u d ] ∈ R m × d .
The chosen orthonormal basis is called a representative of its subspace.
The Grassmann manifold is an abstract [Table] quotient manifold that can be represented in many ways.
In this paper [Table] we embed the Grassmann manifold into the space of symmetric matrices Sym ( m ) by
[Equation]
This method is called Grassmann embedding , which we now introduce.
Rectiﬁed manifold embedding Suppose we have obtained a graph hidden representation H ∈ H ⊂ R n × m and its row-generated subspace span ( H (cid:62) ) , which as mentioned can be achieved by applying one or multiple layers of graph convolution like GCN.
We then go to ﬁnd a better Grassmann representation for the subspace to extract the most important features of the data.
Such representation is characterized by an orthogonal basis of the subspace, which can be rectiﬁed in several ways.
A typical method as demonstrated in Huang et al.
(2018) is QR decomposition of matrix H (cid:62) .
Our preliminary goal is to bring representation towards a unique subspace dimension k .
That is, we need to ﬁnd out the most representative basis U =
[ u 1 , u 2 , . . . ,
u d ] , thus U =
[ U ] , an equivalence class of U , is a point of an underlying Grassmann manifold M .
To this end, we employ singular value decomposition (SVD) for the hidden feature
[Equation]
where U ∈ R m × k with rank ( H (cid:62) )
=
k, 1 ≤ k ≤ min { m, n } is an orthonormal basis that spans the column space of H (cid:62) .
The diagonal S :=
Diag ([ σ 1 , . . .
, σ k ]) ∈
R k × k contains k singular values sorted in the descending order, where the l th singular value indicates the percentage importance of the l th basis vector.
The corresponding singular vectors constitute V =
[ v 1 , . .
. , v k ] ∈ R n × k .
The ﬁrst d -columns of U ( d ≤ k ) from SVD include the d most important components in the original space of H , and the subspace it spans naturally composes a Grassmann point U :=
[ U d ] (the equivalent class under the orthogonal group O ( d ) ) on the Grassmann manifold M ( d, M ) .
This embedding operation can be applied to any hidden representation from H , which intuitively regards a graph with a set of n featured nodes as a d -dimensional subspace of the embedding space R m .
Each graph feature is transformed to a hidden feature, and different graphs (possibly with varying node size) are taken as points on the (same) Grassmann manifold, where similar instances are also geodesically close to each other.
Such similarity provides a criterion for clustering or classiﬁcation.
At this point, we have ﬁnished embedding graphs to the Grassmann manifold where each graph is represented by a subspace of orthogonal basis U d ∈ R m × d .
Projection mapping With the above Grassmann embedding, we have a set of embedded Grassmann points {U 1 , . .
, U N } ⊂ M ( d, M ) with their relevant orthogonal matrix basis as representatives.
Any follow-up learning can then be built on {U 1 , . .
, U N } by the Grassmann geometry, see Absil et al.
As pointed out in (1), the entire manifold M ( d, M ) can be embedded (mapped) into the Euclidean space
Sym ( m ) .
This embedding naturally transforms graph features to their Euclidean representation and thus conventional deep learning can be applied. [Figure] [Figure]
D i m e n s i o n ( [Figure] Raw t [Figure]
{+50 Dimension 
+}
{+D i m e n s i o n (c) GCN–G R P OOL  25 Dimension 
+}
D i m e n s i o n (d) GIN–G R P OOL
To be precise, we project the graph latent representation to the symmetric positive deﬁnite (SPD) matrix Π( U d ) =
U d U (cid:62) d .
The resulting SPD matrix representation, an analog to a bilinear mapping, captures a second-order statistics that can better reﬂect regional features (Tuzel et al., 2006).
Moreover, the rectiﬁed representation U from (2) can reduce noise, and the approximated covariance matrix U d U (cid:62) d is thus more expressive.
This projected Euclidean representation can be used for various tasks.
For example, in graph property prediction, the Grassmann embedding can be vectorized and output as the readout for graph prediction.
In subspace clustering, Π( U d ) with respect to H is analogous to the afﬁnity matrix and it can be used by clustering algorithms for graph segmentation. Remark 1.
The SPD matrices from the projection mapping are points on the positive semideﬁnite (PSD) cone.
However, we do not leverage this special topology here when calculating the pair-wise distance.
We measure the distance between Grassmann points with Euclidean metrics, which only requires the vector representation of m ( m + 1) / 2 dimensions.
E XPERIMENTS FOR G RASSMANN POOLING
We evaluate the proposed G R P OOL on a set of graph classiﬁcation tasks with different graph sizes, volume, and density.
We also include a graph regression task with moderate data size.
We benchmark the performance on ﬁve binary classiﬁcation, one multi-class classiﬁcation and one regression tasks.
Except for ogbg-molhiv (Hu et al., 2020), all datasets are provided by TUDataset benchmarks (Morris et al., 2020) with default graph attributes.
The performance is compared against two hierarchical (T OP KP OOL (Gao & Ji, 2019; C ˇ at ˇ alina et al., 2018), SAGP OOL (Lee et al., 2019)) and four global pooling methods (A TTENTION P OOL (Li et al., 2016), S UM , A VG and M AX pooling) with two types of network architectures.
For completeness, we report prediction results of all models with GCN (Kipf & Welling, 2017) and
GIN (Xu et al., 2019) (under JKN ET structure (Xu et al., 2018)) convolutional layers.
We detail the model setup in Appendix C.
Table 1 compares the prediction performance of G R P OOL with baselines.
We report mean test accuracy for the ﬁrst ﬁve classiﬁcation tasks; ROC-AUC score for ogbg-molhiv classiﬁcation; and mean square error (MSE) for QM7 regression.
All the mean scores are averaged over 10 repetitions.
Our G R P OOL achieves the top score with generally a lower volatility on all tasks with primary feature matrices.
The advantage is more salient on the bottom half of the table, where we apply
JKN ET structure with GIN convolution for G R P OOL .
It provides a global pooling method and can express more information from its compressed graph representation.
The performance of G R P OOL on COLLAB and QM7 is less promising, which is likely due to the lack of non-structural feature information from the dataset.
In general, the abstracted subspaces provide more information compared to the baseline methods with common graph convolutional layers especially when there are feature matrices on graphs or nodes.
F URTHER INVESTIGATION ON G RASSMANN E MBEDDING
Real datasets could have a great number of graphs with a large variation on node sizes from a few to thousands.
Consequently, deﬁning an identical relatively small subspace dimension d for all graphs could potentially hurt the expressiveness of graph embedding.
Instead, we let d of a Grassmann point [ U d ] ∈ M ( m, d ) be determined by d = (cid:80) ki
=1 1 { σ i > r } .
The σ i corresponds to the singular values from (2), and r denotes the global threshold of the percentage importance.
However, all these
Grassmannian points {U} can be naturally mapped to the embedded Euclidean space Sym ( m ) of the manifold M ( m, d max ) , with d max the highest d over all Grassmann points.
The r is a hyperparameter of the G R P OOL .
However, the choice of r has little impact on the performance of the Grassmann embedding, as demonstrated by our sensitivity analysis.
The experiment is conducted on PROTEINS with both GCN and GIN network architectures from Section 3.
We report the mean test accuracy over 10 repetitions with seven different values of the threshold ratio r , ranging from 0 .
3 to 0 .
9 with step size 0 .
All other hyperparameters are tuned in the same way as Section 3.
Figure 2a shows that the mean test accuracy is stable over all values of the threshold ratio r .
This suggests that the hyperparameter r (within a wide range) has a negligible impact on the performance of our proposed G R P OOL , which indicates that a moderately high value of r (e.g., r > 0 . 5 ) is sufﬁcient to retain the essential information of a graph for Grassmann embedding.
The rest three plots in Figure 2 exploit two-dimensional t-distributed Stochastic Neighbor Embedding (t-SNE) of ﬂattened Euclidean graph embeddings on COLLAB , which is a 3-classiﬁcation task.
Each point denotes a graph representation after G R P OOL , and their color indicates the true labels.
We randomly sampled a sufﬁcient amount of 4 , 000 instances.
As the GIN-G R P OOL model employs JKN ET structure, we aggregate all four pooling results as an approximated graph representation.
Both Figures 2c and 2d suggest a clear clustering pattern of the pooled graphs.
D ISCUSSION AND C ONCLUSION
In this paper, we propose G R P OOL , a new graph pooling strategy that connects with the Grassmann geometry.
The underlying method treats the hidden feature subspace of graphs as Grassmann points, and make analysis on them.
The process of extracting subspace of graph features is a non-linear transformation that outputs the most valuable feature information, and the projection embedding makes second-order approximation on feature relationship. The result graph representation supports Euclidean metrics for loss design. Such strategy can be used for various scenarios.
In speciﬁc, we propose G R P OOL that pools graph hidden representations to identical vectors which we employ for label prediction.
In this way, non-linear analysis on the graph features is deﬁned, which is useful for working on complex or massive attributes.
Moreover, the way of treating regional graphs as
Grassmann points might motivate more connections of mutual learning schemes between the two non-Euclidean spaces of graph and manifold.
A S TABLE B ACK - PROPAGATION FOR SVD
As Singular Value Decomposition (SVD) is one of the essential parts of our proposed Grassmann graph embedding , we present in this section the derivation for the back-propagation for SVD.
This back-propagation is numerically stable, especially in the case when the input matrix has some extremely small singular values.
The following materials are mainly based on the note of Townsend 1 .
For a matrix A ∈ R m × n with rank k ≤ min( m, n ) , we decompose A = USV (cid:62) by using SVD, where U is a m × k matrix, V is in n × k , and S = diag( s 1 , s 2 , . . .
, s k ) is a k × k diagonal matrix which stores the singular values of A .
Here, we have the constraints that the matrices U and V are orthogonal.
We denote the objective function by f ( A ) whose argument is a square matrix A .
We aim to ﬁnd the gradient of the objective function: A := ∇ A f .
For simplicity, we employ the shorthand
A := ∇ A f for the derivation below.
By the property of matrix algebra (Searle & Khuri, 2017), we reach the relation
[Equation]
To calculate A , we need the following three differentials
[Equation]
where F ij = 1 s 2 j − s 2 i · 1 { i (cid:54) = j } which satisﬁes the identity F (cid:62) = − F .
The calculations of F and
S − 1 are often numerically unstable due to the possible near-zero singular values.
To circumvent this difﬁculty, we employ the following improved strategy, as similarly used by Huang & Van Gool (2017):
[Equation]
where (cid:15) is a small number and can usually be set to 10 − 12 .
In practice, we can replace S by the modiﬁed matrix S new in the back-propagation formulas.
We evaluate the three terms in (3) respectively, as follows.
By (5), we can write tr( S (cid:62) d S ) as
[Equation]
Using (4) for the term tr( U (cid:62) d U ) yields
[Equation]
The above expression is a sum of two terms.
By F (cid:62) = − F , these two terms can be simpliﬁed as
[Equation]
and
[Equation]
Thus
[Equation]
an equivalent expression for the term tr( U (cid:62) d U ) reads
[Equation]
[Table]
Similarly
[Equation]
the term tr( V (cid:62) d V ) has the following form after some simple algebraic manipulations
[Equation] [Equation]
Finally
[Equation]
we substitute (7)–(9) into (3) and then retrieve the expression for B to obtain the solution
[Equation]
V (cid:62) V − V (cid:62) V (cid:105)(cid:17) V (cid:62) + S − 1 V (cid:62) (cid:0) I n − V V (cid:62) (cid:1)(cid:105)
=
A .
B E SSENTIAL R EQUIREMENTS FOR G RAPH P OOLING
In this section, we check the two essential requirements of a valid graph pooling method, which are graph embedding with a uniﬁed size and permutation invariance .
Proposition 1.
G R P OOL always produces a graph embedding g ∈ R m ( m +1) for the node representation matrix H ∈ R n × m , regardless of the graph size n .
Proof.
Given the node representation matrix H ∈ R n × m of a graph G with n nodes and m features, the Grassmann graph embedding gives the output UU (cid:62) , where U ∈ R m × k with k = rank
( H (cid:62) ) .
Then, the output of G R P OOL is the ﬂattened representation of the upper triangular matrix of UU (
cid:62) ∈
R m × m , which is independent of the graph size n .
Proposition 2.
G R P OOL satisﬁes the requirement of permutation invariance so that it produces the same Grassmann graph embedding under row permutations of the input node representation matrix.
Proof.
Suppose H 1 is the node representation matrix of a graph and let H 2 =
P H 1 , where P is a permutation matrix.
Then,
[Equation]
The Grassmann point for both H 1 and H 2 can be accessed by the same matrix U .
Hence, the proposed graph embedding method is permutation invariant.
C E XPERIMENT C. M ORE ON MODEL TRAINING
We supplement below a summary of the datasets for graph property prediction tasks in Section 3.
The ‘R’ in the
[Figure]
Hyperparameter
[Table]
In the ablation, we consider two graph convolutions to be connected with pooling methods:
GCN (Kipf & Welling, 2017) and GIN (Xu et al., 2019).
For TU Dataset s, the ﬁrst set connects two GCN convolutional layers with one pooling layer except for NCI1 , where we add a third convolutional layer.
We also study the performance of the second GNN architecture with four GIN convolutional layers and JKN ET structure (Xu et al., 2018).
For the ogbg-molhiv , both models use four convolutional layers.
The hidden units of the one-layer MLP for baselines are set identical to that of the convolutional layer.
For G R P OOL , it is set to 64 for the ﬁrst layer and 16 for the second.
Below we provide the searching space of the key hyperparameters that are ﬁne-tuned with grid search, including learning rate, weight decay, hidden unit numbers of convolutional layers, and dropout ratio in MLP layers.
Unless stated, models on OGB applies the same searching space.
Other hyperparameters, if not speciﬁed, are set to default values.
For model training and evaluation, we split each dataset into 80% training, 10% validation, and 10% test samples.
The stopping criterion is deﬁned as the validation loss stops improving for 20 consecutive epochs, or the maximum 200 epochs is reached.
In Table 1, except for those on COLLAB , the results of baselines with GCN on TU Dataset s are retrieved from Zheng et al.
C. L EARNING C URVE AND C ONVERGENCE
The training and validation curves for loss and accuracy are shown in Figure 3.
The results are retrieved from a (random) single run on PROTEINS .
The hyperparameters and network architectures follow exactly the same routine as Section 3.
We only display results from one repetition due to the employment of early stopping criteria, where each independent run would stop at an individual epoch.
The minor volatility after epoch 5 is generally brought about by stochastic gradients.
All four visualizations validate an efﬁcient convergence of our G R P OOL , where the loss curve stabilizes quickly after a few epochs.
The training process is slightly longer for GIN convolution, which is partly due to the more sophisticated network architecture for the model to ﬁt.