G RAPH A NISOTROPIC D IFFUSION
A BSTRACT
Traditional Graph Neural Networks (GNNs) rely on message passing, which amounts to permutation-invariant local aggregation of neighbour features.
Such a process is isotropic and there is no notion of ‘direction’ on the graph. We present a new GNN architecture called Graph Anisotropic Diffusion.
Our model alternates between linear diffusion, for which a closed-form solution is available, and local anisotropic filters to obtain efficient multi-hop anisotropic kernels.
We test our model on two common molecular property prediction benchmarks (ZINC and QM9) and show its competitive performance.
I NTRODUCTION
Graphs are a very common mathematical abstraction used to represent complex systems of relations and interactions.
However, they present several challenges for machine learning methods because they require algorithms to reason about their underlying structure as well as their node and edge features.
Graph Neural Networks (GNNs) are a popular architecture for learning on graph-structured data. GNNs have been successfully applied to problems in different domains such as social networks, recommendation systems (Ying et al., 2018;
Fan et al., 2019), chemistry (Gilmer et al., 2017; Sanchez-Lengeling et al., 2019), bioinformatics (Lv et al., 2021; Gainza et al., 2020), and drug discovery (Gaudelet et al., 2021).
The majority of GNNs are based on the message passing mechanism (Gilmer et al., 2017), in which the features of each node in the graph are updated by aggregating information from its neighbours.
However, recent studies have demonstrated several limitations of message passing, which include limited expressive power (due to equivalence to the Weisfeiler-Lehman graph isomorphism test)
(Xu et al., 2019; Corso et al., 2020; Bodnar et al., 2021), as well as the oversmoothing (Oono & Suzuki, 2020) and bottleneck phenomena (Alon & Yahav, 2021).
A recent line of works linked GNNs to diffusion processes on graphs.
GRAND (Chamberlain et al., 2021a) and BLEND (Chamberlain et al., 2021b) obtained GNNs from a discretisation of nonlinear and non-Euclidean diffusion PDEs.
A more general version of diffusion based on cellular sheaves constructed upon graphs was studied by Bodnar et al.
Klicpera et al. (2019) applied diffusion to the graph structure itself, as a form of graph rewiring that alleviates bottlenecks and improved learning in homophilic settings.
Topping et al.
(2022) related bottlenecks to graph curvature and used a discrete version of Ricci flow (which, in the continuous setting, can be conceptually interpreted as the ‘diffusion of the metric’) to improve oversquashing in GNNs. The lack of canonical ordering of graph nodes forces message passing to use permutation-invariant functions, meaning there is no natural ‘direction’ on graphs.
In geometric graphs and meshes where additional structure is available, learning schemes based on anisotropic diffusion have been studied (Andreux et al., 2014; Boscaini et al., 2016b;a; Sharp et al., 2021). Attempts to define ‘pseudodirections’ in GNNs using graph motifs were undertaken by Monti et al. More recently, (Beaini et al., 2021) proposed a scheme for adding directionality in message passing using the graph Laplacian eigenvectors.
[Figure] In this work, we build on the recent works of Sharp et al.
(2021) and Beaini et al.
(2021) to develop anisotropic diffusion on graphs as a basis for GNNs.
We propose a new way to perform isotropic diffusion on graphs with learnable kernel size (see Figure 1a) that can be applied efficiently by either solving a linear system or through spectral expansion.
We combine this diffusion with a local anisotropic filter (see Figure 1b) to obtain a global anisotropic kernel over the graph (see Figure 1c).
This technique is leveraged to propose a novel GNN architecture that we call Graph Anisotropic Diffusion (GAD).
Empirically, we show that GAD improves the performance of competitive GNNs on two popular molecular property prediction benchmarks: ZINC and QM9.
B ACKGROUND
Diffusion equation.
Diffusion describes the movement of some quantity from regions of high concentration to lower concentration over time.
A classical example of this phenomenon is the diffusion of heat on an iron rod – heat diffuses from warmer parts of the rod to colder ones. Given a domain Ω (e.g., a 1D iron rod), let x ( u, t ) : Ω × [0 , ∞ ) → R be a function representing some quantity (e.g., temperature) at position u ∈ Ω and time t ≥ 0 .
The diffusion process is governed by a partial differential equation (PDE) called the heat equation ,
[Equation]
where ∆ is the Laplacian operator associated with the domain.
1 Assuming initial condition x ( u, 0) , the solution of Equation 1 is given by the heat operator x ( u, t ) = exp( − t ∆) x ( u, 0) , where exp is the operator exponential.
Diffusion equation on surfaces. In computer graphics applications, it is common to work with discretisations of continuous domains (manifolds) such as meshes or point clouds.
Let us assume that the domain is sampled at n points, { u 1 , . . u n } ⊆ Ω . The sampled function can be represented as an n -dimensional vector x =
( x ( u 1 ) , . . . , x ( u n )) . The continuous Laplacian operator ∆ is discretised using finite elements methods giving rise to the (weak) Laplacian matrix L and the mass matrix M (both of size n × n ), such that the diffusion rate is approximated by M − 1 Lx .
Equationbecomes
[Equation] where x ( t ) represents the temperature on the nodes of the discretised domain at time t . On meshes, it is customary to use the cotangent scheme for Laplacian discretisation and lumped (diagonal) mass matrix, which can be interpreted as a weighted inner product.
[Figure] X
[Figure]
diffusionlayer h t ( x ) anisotropicfilters
B av & B dx + concat
MLP + addition Graph Anisotropic Diffusion
Implicit Timestep h t ( x ) =
( D + t L ) − 1 Dx
Spectral Expansion
[Figure]
G RAPH A NISOTROPIC D IFFUSION
In this section, we present our Graph Anisotropic Diffusion (GAD) method.
First, show how a closed-form solution to the graph diffusion equation.
Then, we describe the local anisotropic kernels used to provide some notion of direction.
Finally, we explain how these two components fit together in GAD.
Figure 2 provides a high-level overview of the GAD architecture layer, and Figures 1 and 1c provide intuition behind our construction.
3. L EARNABLE D IFFUSION
Let G =
( V , E ) be an undirected graph with |V| = n nodes and |E| =
e edges, A be the n × n adjacency matrix of G , and D is the node degree matrix.
Our method is based on the discretisation of the diffusion equation (Equation 1) analogous to mesh diffusion equation (Equation 2), where instead we use the graph Laplacian matrix L = D − A and the degree matrix as the mass matrix to diffuse the n × d input feature matrix X :
[Equation]
We use a channel-wise linear diffusion layer h t : R n → R n , diffusion each channel (column of X ) by applying Equation 3 for a time t that is also learnable per channel.
This enables us to go beyond traditional message passing mechanisms in GNNs.
Instead of a single propagation step or discrete random walk, we perform a continuous diffusion with control over the parameter t that determines the kernel size and can be learned end-to-end.
As a result, we can optimise the propagation of the features at each node from purely local to completely global by learning each channel’s appropriate
How to compute the diffusion?
Different numerical schemes are available to solve the linear diffusion equation (Equation 3).
To learn and optimise the parameter t via back-propagation, we should confine ourselves to differentiable schemes.
In this regard, two simple schemes are used in the experiments: implicit timestep and spectral expansion .
The implementation of both is simple using common scientific libraries.
Implicit timestep requires solving a large sparse linear system.
In contrast, the spectral expansion scheme has a reduced time complexity and it relies on dense arithmetic but requires some precomputation.
In the following we detail both options.
Direct implicit timestep.
Sharp et al.
(2021) used a single implicit Euler timestep to approximate the diffusion, which in our setting amounts to [Equation]
This approach requires solving a sparse linear system equations for each diffusion operation using Cholesky decomposition.
Spectral expansion scheme.
Another way to approximate the solution of the diffusion equation is using spectral (Fourier) expansion. On a graph, an orthogonal Fourier basis is given by the (generalised) eigenvectors of the normalised graph Laplacian, L ϕ i
= λ i
D
ϕ i .
The solution of the diffusion equation can be expressed through the spectral expansion of the heat operator,
[Equation]
where Φ k = ( ϕ 1 , . .
. , ϕ k ) is the n × k matrix of the first k eigenvectors and Λ k = diag( λ 1 , . . .
, λ k ) is the k × k diagonal matrix of the corresponding eigenvalues.
This is an approximation due to dropping higher-frequency components, and k plays the role of bandwidth. The matrices Φ k and Λ k can be pre-computed, as they do not depend on the features x nor the learnable parameter t .
Therefore, with a small approximation error, we obtain a significantly more efficient method than the implicit Euler approach.
3. A NISOTROPIC FILTERS
Beaini et al. (2021) proposed the use of anisotropic filters on graphs and defined them as aggregation matrices that allow directional flows between nodes in the graph. These matrices depend on the graph structure and are derived from a vector field F (represented as an n × n matrix) that associates each edge with a scalar weight describing the ‘flow’ from one node to its neighbours.
These weights correspond to the magnitude of the flow in a specific direction.
Following Beaini et al. (2021), we use the gradient of the first non-constant eigenvector of the normalised graph Laplacian (the Fiedler vector, which we denote here by ϕ ) to define the vector field [Equation] (6) where ⊙ is element-wise multiplication and ( ∇ ϕ ) ij = ϕ
i − ϕ
j is an n × n matrix. Beaini et al. (2021) employ this vector field to define two aggregation matrices B av and B dx as [Equation]
[Equation] ) where ˆ F denotes the vector field F after normalising each row by its L 1 -norm and | · | denotes the absolute value.
Message passing based on these aggregation schemes enables the messages to contain a notion of directionality — an indication of where a message comes from. B av can intuitively be understood to smooth the signal based on the directions specified by F . In contrast, the aggregation matrix B dx computes the discrete derivative of the signal. The visualisation of both
B av and B dx for a specific node in the graph is shown in Figure 1b. We use B av and B dx to define the anisotropic filters in our architecture. 3. G RAPH A NISOTROPIC D IFFUSION A RCHITECTURE We combine the diffusion layer in Section 3.1 and the aggregation matrices in Section 3.2 in one architecture which we call Graph Anisotropic Diffusion (GAD).
One block of GAD is shown in Figure 2. The input node features are passed through a diffusion layer h t . Next, we apply a nonlinearity to the output of the diffusion and then the aggregation matrices ( B av and B dx ).
Last, the output of the diffusion layer is concatenated with the output of these aggregation matrices and passed through an MLP.
We also use a skip connection in every GAD block, as shown in Figure 2.
By combining the linear diffusion and the local anisotropic filters, we can obtain a global anisotropic diffusion, as shown in Figure 1c.
R ELATED WORK
Computer graphics and geometry processing. In computer graphics, the heat kernel of the diffusion equation on a surface is related to its Gaussian curvature. This property was exploited by Sun et al.
; Bronstein & Kokkinos (2010) to construct heat kernel signature (HKS) shape descriptors.
Andreux et al.
(2014) introduced the use of the anisotropic Laplace-Beltrami operator to give a directional diffusion operation on discrete meshes.
Boscaini et al.
(2016b) also proposed a convolutional neural network architecture based on anisotropic diffusion kernels to learn an intrinsic dense correspondence between non-rigid shapes.
More recently, Sharp et al.
(2021) combined directional filters with diffusion for learning on meshes and point clouds.
Differential equations on graphs.
Diffusion processes and random walks on graphs have long been exploited, including the classical PageRank algorithm (Page et al., 1999) (steady state of diffusion on the web graph) that powered the early version of Google search engine ranking, or non-linear dimensionality reduction by diffusion maps proposed by Coifman & Lafon (2006).
In the graph ML literature, GNNs were formulated as ordinary differential equations (ODEs) by Avelar et al.
Poli et al.
(2021), continuous diffusion-type processes (Xhonneux et al., 2020; Gu et al., 2020), or physically-informed architectures (Sanchez-Gonzalez et al., 2019). On the other hand, solving the non-Euclidean diffusion equations using the Laplacian eigenvectors and eigenvalues was considered a crucial tool for some early techniques on graph neural networks (Henaff et al., 2015; Kipf & Welling, 2017; Levie et al., 2018).
GRAND (Chamberlain et al., 2021a) formulated attention-type GNNs as discretisations of the diffusion equation. BLEND (Chamberlain et al., 2021b) proposes a non-Euclidean extension to GRAND. The key difference of our work from GRAND and BLEND (Chamberlain et al., 2021a;b) is that we used closed-form solution to graph diffusion, interleaved with an anisotropic filter, rather than a numerical solver.
To evaluate our model, we tested it on two popular benchmarks: ZINC (Irwin et al., 2012) and coordinate-free QM9 (Ramakrishnan et al., 2014). The implementation using PyTorch (Paszke et al., 2019) and PyTorch Geometric (Fey & Lenssen, 2019) libraries is publicly available at https://github.com/Ahmed-A-A-Elhag/GAD.
Our results on each dataset are presented with a comparison with previous models in Tables 1 and 2.
(For QM9, we test our model on the first seven properties provided by the PyTorch Geometric (Fey & Lenssen, 2019) library).
Appendix A contains the implementation details and hyperparameter settings for each dataset.
GAD-i uses the implicit timestep approach and
GAD-s the spectral expansion scheme.
*Uses additional tailored features such as cycles.
[Table] [Table]
Given that DGN (Beaini et al., 2021) is the baseline GNN that our model was constructed upon, we consider this as the main comparison point to analyse the effectiveness of the diffusion process presented. On ZINC, the addition of the diffusion layer reduces the MAE by more than 17%
[Table]
and on QM9, it improves the performance in every single property. Note that while methods using the addition of tailored features or message passing structure achieve better performances on ZINC, these improvements are orthogonal to the ones proposed in this work and could be combined
[Table]
C ONCLUSION
We proposed Graph Anisotropic Diffusion (GAD), a novel method that consists of a new efficient way of performing isotropic diffusion on graphs combined with local anisotropic kernels.
This approach is used to obtain a global anisotropic diffusion.
Empirical results on popular molecular property prediction benchmarks confirm the improved predictive capacity of our method.
We hope this work spurs future research on alternative methods to obtain anisotropic diffusion on graphs and theoretical analyses of their expressive power.
A IMPLEMENTATION DETAILS AND H YPERPARAMETER S ETTING
Following Beaini et al.[-; in equations 6 and 7. Here k is the layer index, h k +
-]
(2021), in our implementation of the local filters use mean, max, and min aggregators on top of B av and B dx , followed by the degree scalers of PNA (Corso et al., 2020).
A. ZINC
ZINC is a molecule graphs dataset in which the task is to predict the penalised water-octanol partition coefficient of a certain molecule (formulated as a graph regression task).
We use the splits and evaluation criteria provided by Dwivedi et al.
(2020): 10K training, 1K validation, and 1K test, with a maximum of 100K parameters and mean absolute error (MAE) as evaluation metric.
We compare our model against various GNNs architectures: GCN (Kipf & Welling, 2017),
GIN (Xu et al., 2019), GraphSage (Hamilton et al., 2017), GAT (Velickovic et al., 2018), MoNet (Monti et al., 2017), GatedGCN (Bresson & Laurent, 2018), PNA (Corso et al., 2020), DGN (Beaini et al., 2021), HIMP (Fey et al., 2020), SMP (Vignac et al., 2020), and GSN (Bouritsas et al., 2021).
We ran each solving scheme (implicit timestep and spectral expansion) five times to estimate the mean performance and its standard deviation (Table 1).
We use the same hyperparameters setting in DGN (Beaini et al., 2021), and decrease the hidden dimensions to keep the same number of parameters. In the spectral expansion scheme, we use the number of eigenvectors k ∈ { 20 , 25 , 30 } .
A. QM
QM9 is a common molecule graph benchmark, which provides different quantum mechanical properties of small molecules.
we test our model on the first seven properties provided by the PyTorch Geometric (Fey & Lenssen, 2019) library (Table 2) keeping again constant the number of parameters and removing the 3D coordinates from the model’s inputs.
For the data split, we use around 110k training, 10k validation, and 10k test samples as provided by Brockschmidt (2020).
The evaluation metric is mean absolute error (MAE).
We fine-tuned GAD over the following hyperparameters setting:
• Weight decay ∈ { 3 × 10 − 3 , 3 × 10 − 4 , 3 × 10 − 5 , 3 × 10 − 6 , 3 × 10 − 7 } .
• Dropout ∈ { 0 , 0 . 1 , 0 . 2 , 0 . 3 } .
• lr ∈ { 1 × 10 − 3 } .
• Batch size ∈ { 128 , 256 } .
• Number of layers ∈ { 6 , 7 , 8 , 9 , 10 } .
• Hidden dimensions ∈ { 75 , 90 , 130 , 150 } .
• Number of eigenvectors k ∈ { 15 , 20 , 25 } .
• Aggregators ∈ { ( mean, av 1 , dx 1 ) , ( mean, max, min, dx 1 ) , ( mean, max, min, av 1 , dx 1 ) , ( mean, sum, max, dx 1 ) } .
For PNA (Corso et al., 2020) and DGN (Beaini et al., 2021) models, we ran each one with the hyperparameters obtained from the search above and increased the size of hidden dimensions to have the same number of parameters.
Aggregators used in PNA (Corso et al., 2020) are ( mean, max, min, std ) , and in DGN (Beaini et al., 2021) are ( mean, sum, max, dx 1 ) .