O N R E C O V E R A B I L I T Y O F G R A P H N E U R A L N E TW O R K R E P R E S E N TAT I O N S
A B S T R AC T
Despite their growing popularity, graph neural networks (GNNs) still have multiple unsolved problems, including finding more expressive aggregation methods, propagation of information to distant nodes, and training on large-scale graphs.
Understanding and solving such problems require developing analytic tools and techniques.
In this work, we propose the notion of recoverability , which is tightly related to information aggregation in GNNs, and based on this concept, develop the method for GNN embedding analysis.
Through extensive experimental results on various datasets and different GNN architectures, we demonstrate that estimated recoverability correlates with aggregation method expressivity and graph sparsification quality.
The code to reproduce our experiments is available at https://github.com/Anonymous1252022/Recoverability .
I N T RO D U C T I O N
Over the last decade, deep learning allowed researchers to tackle multiple hard tasks, previously considered intractable.
For example, convolutional neural networks (CNNs) have been successfully applied to computer vision problems such as image classification (He et al., 2016), object detection (Ren et al., 2015), and semantic segmentation (Ronneberger et al., 2015).
Nevertheless, while CNNs are successful in processing pixels, multiple other modalities such as 3D meshes (Wu et al., 2019), social networks (Ribeiro et al., 2017), brain connections (Shapson-Coe et al., 2021) and many others, require the processing of irregular data.
As a result, in recent years, researchers have been looking into ways to exploit deep learning methods, such as CNNs, in order to work with graph structured data.
Deep learning on graphs and, in particular, graph neural networks (GNNs, Gori et al., 2005;
Scarselli et al., 2008; Kipf & Welling, 2017), based on message passing (Gilmer et al., 2017), has become a very popular tool for machine learning with graphs.
Despite their exceptional ability to learn graph-based data representations, GNNs still suffer from some important problems.
One issue is related to aggregation method expressiveness.
Kipf & Welling (2017) proposed graph convolution networks (GCNs), which aggregate information from neighboring nodes but cannot distinguish between them, Hu et al.
(2020) integrated a self-attention mechanism (Vaswani et al., 2017) into aggregation.
Another problem is lack of ability to propagate information between distant nodes in the graph.
Li et al.
(2018) conjectured that this phenomenon is a result of over-smoothing , Alon & Yahav (2021) offered another explanation – over-squashing .
Finally, training on large-scale graphs is a significant obstacle in integrating GNNs in real-life problems.
The diameter (maximal distance between two nodes) is often small (3–5) even for large graphs.
Thus, GCNs with only 3–5 layers must aggregate information from the whole graph to calculate embedding of a single node, which requires a large amount of memory and compute.
Graph sparsification by dropping a subset of edges (Srinivasa et al., 2020; Rathee et al., 2021) can diminish the problem.
In this work, we introduce the notion of recoverability and demonstrate its tight relationship to information aggregation in GNNs.
Recoverability provides an alternative insight into the aforementioned problems and can serve as a good tool for understanding their roots and finding better solutions for them.
First, we provide a qualitative definition of recoverability and its empirical counterpart, recoverability loss.
We then use reproducing kernel Hilbert space (RKHS) embedding for estimating recoverability loss in an efficient and differentiable way.
Finally, we show how to use recoverability for measuring the quality of embedding method.
M E T H O D
We characterize neural graph embedding as having three parameters: an aggregation algorithm, a certain number of layers, and a graph sparsification algorithm.
Our goal is to compare different embeddings based on those parameters and see which one performs better on the given data.
To achieve this goal, we define a method for analyzing neural graph embeddings.
First, the problem of measuring the quality of a given embedding is reduced to the problem of determining how well one can recover one random variable from another.
Second, we define a new concepts of recoverability and recoverability loss.
Third, we demonstrate how to compute recoverability loss in a differentiable and efficient way.
Finally, we present an algorithm for measuring graph embedding quality using recoverability.
From an embedding to recoverability.
In the node property prediction task, given graph G =
( V, E ) equipped with node features X
= { x v } v ∈ V and labels Y =
{ y v } v ∈ V distributed according to some probability distribution ( x v , y v ) ∼ ( X, Y ) , one should predict the label of each node in the graph.
To solve this task we embed the graph with node features ( G, X ) into a latent space, acquiring some representation of each node, which we denote as h v ∼ H , and then apply a classifier to the embedding h v , trying to predict label y v .
The success of the above procedure is highly dependent on the ability to learn a map between random variables H and Y .
Since different embedding methods produce different random variables, we can reduce the problem of measuring the embedding method’s quality to measuring the ability to learn random variable Y from random variable H .
This leads to the natural question of how much information do we have in random variable H to recover Y .
We term the ability to recover Y from a given H recoverability .
Recoverability and recoverability loss.
Given two random variables H and Y , with the values in
U ⊂ R d and R , respectively, we say that Y is fully recoverable from H if there exists a continuous function f : U → R such that f ( H )
=
Y .
In the general case, however, we do not have such a relation between H and Y , which means that we cannot fully recover Y from H .
Nevertheless, H may still contain some information from which we can partially recover Y .
To measure the amount of such information in H , we define the set C H
= { f ( H )
| f : U → R is continuous function } i.e., the collection of all random variables that could be fully recovered from H , and then evaluate the distance between random variable Y and the set C H : ρ ( Y | H )
=
inf Z ∈C H d ( Y, Z ) , where d is some distance function.
We denote the value ρ ( Y | H ) as the recoverability loss.
When we are given two different random variables H 1 and H 2 (i.e., two different embeddings of ( G, X ) ) such that ρ ( Y | H 1 ) <
ρ ( Y | H 2 ) , Y is more recoverable from H 1 than from H 2 (which means that embedding method 1 is better than method 2).
This case is demonstrated in Fig.
C.3.
Recoverability loss estimation.
We denote the empirical estimation of ρ ( Y | H ) on a finite collection of samples { ( h n , y n ) } n ∈ [ N ] by ρ ∗ ( Y | H ) and use the distance induced form the L p -norm, where p ∈ [1 , ∞ ) .
Theorem 1 The empirical estimation of recoverability loss is given by ρ ∗ ( Y | H )
= 1 N 1 /p ∥
( I − Π) y ∥ p , where Π is an orthogonal projection to the
Im ( K ) and K ij = exp (cid:16) −
[Equation]
is a Gram matrix.
We provide the proof of the Theorem 1 in Appendix A.
An algorithm for embedding quality estimation.
In Algorithm 2, we first, apply an embedding method EM to ( G, { x n } n ∈ [ N ] ) to produce h n ∼ H , which is equivalent in time to a single forward pass of GNN on the full graph.
We then use { h n } n ∈ N to estimate ρ ( Y | H ) using Algorithm 1.
The time complexity of Algorithm 1 is O (cid:0) Nm 2 (cid:1) , where N is the number of nodes in the graph and m is the batch size of Algorithm 1.
This allows fast approximation of the recoverability loss, as compared to training a GNN model on a given dataset.
Algorithm 1 ρ ( Y | H ) estimation.
Input: { ( h n , y n ) } n ∈
[ N ]
Output: ρ ∗ ( Y | H ) for j th batch (cid:110) (˜ h
i , ˜ y i ) (cid:111) i ∈ [ m ] ⊆ { ( h n , y n ) } n ∈
[ N ] do
K kl ← exp (cid:18) −∥ ˜ h k − ˜ h
l ∥2 σ(cid:19)
Compute distance matrix.
U, Λ ← eig( K )
Compute eigendecomposition K
=
U − 1 Λ U of K .
Π ← (cid:80) λ i > 0 u i u Ti
Compute projection to im ( K ) .
ρ end for
Algorithm method quality.
Input EM Output
EM quality { h
← EM(( G, { x n } n ∈ [ N ] ))
Generate outcomes of H .
ρ
Use Algorithm 1 to estimate ρ .
E X P E R I M E N T S
In this section we provide extensive experimental results on real datasets and different GNN architectures, where we show various applications of the proposed method for graph embedding analysis.
The experimental settings are given in Appendix C.
The usefulness of edges.
The first experiment shows an interesting property of the datasets appearing in Table C.2, which could help elucidate why GNNs with a SAGEConv layer can be used efficiently for training on these.
We took three consecutive SAGEConv layers without learnable parameters as embedding method EM , i.e., only the aggregation parts (as shown in Appendix C), and applied it to the node features.
Let X denote the random variable of node features, H the node embedding (after EM application) and Y the node classes.
From Table 1 we can see that ρ ∗ ( Y | H ) <
ρ ∗ ( Y | X ) .
In other words, Y is more recoverable from H (node embedding) than from X (node features).
Additionally, we observe an interesting correlation between recoverability loss drop and homophily.
In the Reddit2 and ogbn-products datasets, which have a relatively high homophily, the recoverability loss drops relatively sharply, whereas in the Flickr dataset, which has a relatively low homphily, the recoverability loss changes relatively slightly.
Correlation between recoverability and aggregation method quality.
This experiment shows the correlation between recoverability loss and aggregation method quality on a given dataset, i.e., the test accuracy after the training on a GNN model defined in Appendix C.
The results are shown in
Figs. 1 and C.4 to C.7, where the y axis is the test accuracy of the trained GNN model on a given dataset, and the x axis is the inverse of the estimated recoverability loss.
For GraphConv, GCNConv,
SAGEConv and GINConv, the estimation of recoverability loss was done only on the aggregation part, according to Appendix C. For GATv2Conv, the recoverability loss was computed for the node embedding of the trained model.
From all plots it can be seen that as the recoverability loss drops, the test accuracy becomes higher.
Correlation between recoverability and the graph sparsification method quality.
In this experiment we show that the quality of the given sparsification method correlates with recoverability.
We use two simple sparsification methods.
In Random sparsification we randomly drop 90% of the edges from the graph.
In Max d sparsification we find the maximal value of d that satisfies (cid:80) v ∈ V min { d, d in ( v ) } ⩽ 0 .
1 | E | , where d in ( v ) is an input degree of v , and for each node v in the graph we randomly leave min { d, d in ( v ) } of input edges.
To evaluate quality of sparsification, we first apply it to the given dataset and then train a three-layer SAGEConv GNN mode (defined in
[Table]
[Table]
Appendix C) on sparsified data.
We also compute the recoverability loss of the aggregation part, as described in Appendix C.
The results are shown in Table 2.
We see that the Max d sparsification is better than Random , and correspondingly, the recoverability loss of Max d is lower than that of
Random .
B S Y N T H E T I C DATA E X P E R I M E N T S
If we extend the collection of continuous functions in C H (Section 2) to the measurable functions, and take as distance d the one induced from the L 2 norm on random variables, then the Z ∈
C H that minimizes ρ ( Y | H ) is almost everywhere equivalent to conditional expectation E
[ Y | H ] .
Consequently, conditional expectation E [ Y | H ] can be used for testing ρ ( Y | H ) values on synthetic data.
1-D: To demonstrate the ability of recoverability to capture the existence of a continuous map from one random variable to another, we use a simple 1D experiment.
Let X be a normally distributed random variable, and let Z and W be defined as follows:
[Equation]
Since f 1 is invertible and f 2 is not, we can fully recover X from Z but not from W .
Of course, we can fully recover Z and W from X , since we explicitly defined continuous maps f 1 and f 2 .
For this test we generated 1000 samples of X and estimated ρ ∗ with Algorithm 1.
The results are shown in Table B.1.
Theoretical recoverability loss for Table B.1:
[Equation]
100-D: Now, let X
and N be 100-dimensional random vectors, with independent normally distributed entries, and Y be defined as
[Equation]
where α is some parameter.
When α tends to zero, Y can be fully recovered from X , and when | α | is large, the noise N dominates the value of X , and consequently Y cannot be recovered from X .
This behavior is visualized in Fig.
B.1, where ρ ∗ ( Y | X ) , estimated on 1000 samples, is compared to its theoretical value.
Theoretical recoverability loss for Fig.
B.1:
[Equation]
[Table]
[Table]
C R E A L DATA E X P E R I M E N T S
Experimental setting: Dataset statistics, used in the following experiments, are summarized in
Table C.2.
Throughout the experiments we use five different embedding layers 2 :
• GraphConv (Morris et al., 2019):
[Equation]
where Θ 1 and Θ 2 are trainable parameters.
• GCNConv (Kipf & Welling,
[Equation]
[Equation]
where ˆ d i = 1 + (cid:80) j ∈ N ( i ) e ji , and Θ are trainable parameters.
• SAGEConv (Hamilton et al., 2017):
[Equation]
where W 1 and W 2 are trainable parameters.
• GINConv (Xu et al., 2019):
[Equation]
where h Θ is an MLP and ϵ could be trainable.
[Equation]
et
[Equation]
where
[Equation]
and Θ are trainable parameters
For experimental purposes, we separate the embedding part from the classifier part in a GNN model, as described in Section 2.
Therefore, for training, we use a GNN model consisting of two consecutive blocks, where the first block is an embedding built from k ∈ { 1 . . .
9 } layers of one of the considered types, and the second block is a classifier consisting of three fully connected layers.
Each layer except for the last one is followed by ReLU activation and dropout.
Aggregation method quality.
To evaluate the quality of the aggregation part of some embedding method, we fix learnable parameters as identities and apply the resulting layer to the graph multiple times, defining an embedding method EM , which can be evaluated using Algorithm 2.
For example, for SAGEConv (Hamilton et al., 2017), defined as follows:
[Equation]
we fix matrices W 1 and W 2 to identity.
Sparsification method quality.
Similarly, to evaluate sparsification, we apply sparsification SM to the graph G , producing a sparser graph ˜ G =
( V, ˜ E ) .
Then we use Algorithm 2 with ( ˜ G, { x n } n ∈ [ N ] ) , { y n } n ∈ [ N ] and embedding method EM as input, as shown in Fig.
C.2.
Propagation of information to distant nodes.
This experiment demonstrates the depth problem in GNNs and its correlation with the recoverability.
We took two GNN models with three and nine embedding layers of type SAGEConv each (the model is defined in Appendix C), and trained them on datasets from Table C.2.
The results are given in Table C.3.
For the Reddit2, ogbn-arxiv and ogbn-products datasets, we did not see a significant change in test accuracy after the training, whereas for the Flickr and PPI datasets, there was a degradation in accuracy for nine-layer architecture.
We collected the recoverability loss of each embedding layer in the trained nine-layer architecture.
The results are shown in Fig.
C.8.
For Reddit2, ogbn-arxiv and ogbn-products, we have a similar pattern of ρ behavior: it smoothly decreases, meaning that each consecutive layer aggregates more and more information to recover Y .
For Flickr and PPI, we see different behavior.
In Flickr, ρ increases, indicating that it does not aggregate any useful information from neighboring nodes to recover Y .
In PPI, ρ first decreases and starts to increase after three layers, which indicates that the over-smoothing or over-squashing problem is occurring.
[Equation]
• There is a numerical instability in gradient computation of eigendecomposition K =
U Λ U T .
The problem comes from equal eigenvalues in:
[Equation]
∂ Λ (cid:19)
[Equation]
(cid:41)
U T (D.38)
[Table]
where:
[Equation]
and ( · ) diag means all off-diagonal elements set to 0.
To overcome this issue, we changed ˜ K to be:
[Equation]
• When we perform eigendecomposition , we do not have explicit zero eigenvalues.
Thus instead of taking the first k non-zero eigenvalues , we simply clamp
them between 0 and