Extended Abstract Track
Surfing on the Neural Sheaf
Editors: Sophia Sanborn, Christian Shewmake, Simone Azeglio, Arianna Di Bernardo, Nina Miolane
Abstract
The deep connections between Partial Differential Equations (PDEs) and Graph Neural
Networks (GNNs) have recently generated a lot of interest in PDE-inspired architectures for learning on graphs.
However, despite being more interpretable and better understood via well-established tools from PDE analysis, the dynamics these models use are often too simple for complicated node classification tasks.
The recently proposed Neural Sheaf
Diffusion (NSD) models address this by making use of an additional geometric structure over the graph, called a sheaf, that can support a provably powerful class of diffusion equations.
In this work, we propose Neural Sheaf Propagation (NSP), a new PDE-based
Sheaf Neural Network induced by the wave equation on sheaves.
Unlike diffusion models that are characterised by a dissipation of energy, wave models conserve a certain energy, which can be beneficial for node classification tasks on heterophilic graphs.
In practice, we show that NSP obtains competitive results with NSD and outperforms many other existent models on several datasets.
1. Introduction
Since their introduction (Scarselli et al., 2008), Graph Neural Networks (GNNs) have shown outstanding results in a broad range of applications, from physics simulations (Shlomi et al., 2020) to protein folding (Jumper et al., 2021).
The key idea is to leverage the inductive bias induced by the topology of graph-structured data, represented by the connectivity structure, to perform graph representation learning tasks.
However, it has been shown that pioneering works on GNNs, like the Graph Convolutional Networks (GCNs) (Kipf and Welling, 2016) tend to suffer from oversmoothing Oono and Suzuki (2019) and perform poorly on heterophilic datasets Zhu et al.
(2020a).
In the former case, GCNs with many layers tend to have nodes that aggregate features from remote parts of the underlying graph, leading to the inability to exhibit local patterns over the underlying graph domain, which jeopardises the overall performance.
In the latter case, GCNs assume that nodes, that are connected, will share the same label, which is an inhibitive inductive bias in most of the real-world scenarios.
To overcome these limitations, Hansen and Gebhart (2020), Bodnar et al.
(2022) introduced generalisations of graph convolutional networks by equipping the underlying graph with the structure of a cellular sheaf Curry (2014) (see section 2 for a review).
Sheaves are used to construct convolution via the sheaf laplacian .
The richer harmonic space of such sheaf laplacians give better node separation powers.
Hansen and Gebhart (2020) introduced sheaf convolutional networks with a one dimensional sheaf, while Bodnar et al.
(2022) introduced Neural Sheaf Diffusion which allows higher dimensional sheaves in combination with learning the sheaf itself from the data Barbero et al.
; Bodnar et al.
In this work, we aim to exploit recent works on PDE-based GNNs Eliasof et al.
Chamberlain et al.
(2021) to introduce Neural Sheaf Propagation (NSP), a procedure used to induce the wave equation on sheaves.
Unlike sheaf diffusion models Bodnar et al.
Barbero et al.
(2022) or other ODE-based GNNs Di Giovanni et al.
; Chamberlain et al.
(2021), characterised by a dissipation of energy, wave models conserve energy.
This can be beneficial for node classification tasks on heterophilic graphs since the features will generally tend to be less smooth.
In Section 2, we review the required background on graph neural networks, cellular sheaf theory and neural sheaf diffusion.
Section 3 defines the Neural Sheaf Propagation (NSP), a procedure used to induce the wave equation on sheaves.
Finally, in Section 4, evaluate this technique on several datasets with varying homophily levels for the node classification task.
2. Background
We encourage the reader to consult Curry (2014);
Hansen and Ghrist (2019); Hansen (2020) for an in-depth overview of cellular sheaf theory and Bodnar et al. (2022) for a theoretical analysis regarding neural sheaf diffusion. Extended Abstract Track
Cellular Sheaves
Given an undirected graph G =
( V, E ), a cellular sheaf F over G consists of: (1) an assignment of a real vector space F ( v ) for each v ∈ V , (2) an assignment of a real vector space F ( e ) for each e ∈ E , and (3) a linear map F v ⊴ e : F ( v ) → F ( e ) whenever e is an edge containing v .
We refer to the vector spaces F ( v ) and F ( e ) as stalks and the linear maps between them as restriction maps .
For our purposes, elements of a vertex stalk F ( v ) correspond to feature vectors x v , while the edge stalks F ( e ) serve as ancillary spaces in which vertex features can mix.
The space C 0 ( G ; F ) of 0-cochains consists of collections of feature vectors x = ( x v ) v ∈ V .
The sheaf Laplacian L F : C 0 ( G, F ) → C 0 ( G, F ) is a linear operator defined on each stalk as L F ( x ) v = (cid:80) u,v ⊴ e F Tv ⊴ e ( F v ⊴ e x v − F u ⊴ e x u )
The sheaf Laplacian is a symmetric positive semi-definite block matrix.
The diagonal blocks are L F v,v = (cid:80) v ⊴ e F ⊤ v ⊴ e F v ⊴ e , while the offdiagonal blocks are L F v,u = −F ⊤ v ⊴ e F u ⊴ e .
The normalised sheaf Laplacian ∆ F is defined as ∆ F = D − 12 L F D − 12 where D is the block-diagonal of L F .
Neural Sheaf Diffusion
Let us consider a graph G =
( V, E ).
Consider also x ∈ C
( G, F ) to be an nd -dimensional vector obtained by stacking vertically
n individual d -dimensional feature vectors x v ∈ F ( v ) equipped to each node v ∈ V .
If each feature vector allows a number of channels equal to f , the result is a feature matrix X ∈ R ( nd ) ×
f such that each column of X is a vector that belongs in C 0 ( G, F ).
Sheaf diffusion is a process defined by the following differential equation:
[Equation]
evolving on ( G, F ).
In Bodnar et al.
(2022), the authors generated a neural network with layers based on (1).
More precisely, the model has layers of the form:
[Equation]
where σ is an activation function.
It is worth to remark that both the sheaf F ( t ) and the weights W t 1 , W t 2 in equation (2) vary in time, implying that the underlying ”geometry” is dynamic as well.
The sheaf F ( t ) is learned as a function of the data X ( t ).
3. Neural Sheaf Propagation
The interpretation of message passing convolution networks as diffusion processes opens the question of building models corresponding to other dynamical systems.
In this work we focus on the dynamics induced by a hyperbolic PDE, the wave equation.
We consider a time-dependent process X ( t ) ∈ C 0 ( G ; F ).
The wave equation is given by:
[Equation]
Unlike the diffusive dynamics of the heat equation, the wave equation conserves a certain total energy of the signal.
Namely, with the energy defined by E [Equation]
X
[Equation] we have the following result:
Proposition 1 The propagation through the PDE in (3) preserves the energy E F .
Extended Abstract Track
Proof The proof is standard.
Taking the derivative of the energy, we get: ˙
E F ( X ) = ˙
X T ¨ X + ˙ X T ∆ F ( t ) X
= ˙
X T ( ¨ X + ∆ F ( t ) X ) =
with the last equality coming from (3).
Motivated by this, we introduce a model we call Neural Sheaf Propagation .
It is a discretisation of a general hyperbolic version of (3), followed by a non-linearity.
Namely, we replace the Laplacian term by ∆ F ( t ) ( I ⊗ W t 1 ) X t W t 2 ), and add an activation term.
The discretisation is done using the leapfrog method.
Put together, the layers of the network have the structure:
[Equation]
4. Experiments
We evaluate the model proposed in this work on several different datasets, comparing its performance against several models present in the graph representation learning literature.
Similar to NSD, our model comes with a few variations depending on the type of restriction maps that it learns: diagonal, orthogonal and general matrices.
The results are shown in
Table 1.
The datasets are sorted based on their homophily coefficient, the fraction of edges which connect nodes of the same class label which ranges from 0 .
11 to 0 .
The higher the homophily coefficient, the more homophilic is the datasets.
The results are collected overfixed splits, where 48%, 32%, and 20% of nodes per class are used for training, validation, and testing, respectively.
The results shown in Table 1 are selected by taking the test accuracy corresponding to the highest validation accuracy.
5. Conclusion
In this work we proposed Neural Sheaf Propagation (NSP), a novel PDE-based architecture induced by the wave equation on (cellular) sheaves.
The purpose of this work was to leverage the recent development on Neural Sheaf Diffusion (NSD) from Bodnar et al.
(2022) to build a model to perform convolution operations that preserve the overall energy of the system rather than dispel it through the diffusion process.
Evaluation of the proposed method shows that this technique achieves competitive results in several node classification tasks and outperforms the other models in the most heterophilic setting.
However, the presented results are preliminary.
We plan to investigate the interplay between energy conservation and classification accuracy on heterophilic graphs more systematically in the future.
Acknowledgments
We are grateful to all those people involved in the 2022 London Geometry and Machine
Learning Summer School (LOGML), where this research project started.
In particular, we would like to thank the organising committee: Josh Southern, Michelle Li, Tim King, Sara
Veneziale, and Francesco Vigan`o.
Extended Abstract Track
[Table]
Appendix A. Details on experiments
Table 1 contains accuracy results for a wide range of models, along with ours, Conn-NSD, for node classification tasks.
The GNN models in Table 1 can be clustered in three main categories: (1) Classical: GCN Kipf and Welling (2016), GAT Velickovic et al.
GraphSAGE Hamilton et al.
(2017); (2) Models for heterophilic settings: GGCN Yan et al.
(2021), Geom-GCN Pei et al.
H2GCN Zhu et al.
(2020b)
, GPRGNN Chien et al.
(2020), FAGCN Bo et al.
(2021), MixHop Abu-El-Haija et al.
(3) Models which address over-smoothing: GCNII Chen et al.
(2020), PairNorm Zhao and Akoglu (2019).
To have a baseline for this learning task a Multi-Layer Perceptron (MLP) is also taken into account, whose result reported in the last row of Table 1.
The MLP is trained using only the node features and it used to quantify the relevance of the inductive bias provided by graph structure in performing the learning task.
Table 2 contains the ranges of hyperparameters used in the experiments.
[Table]