Graph Neural Networks as Gradient Flows
Abstract
Dynamical systems minimizing an energy are ubiquitous in geometry and physics.
We propose a gradient ﬂow framework for GNNs where the equations followthe direction of steepest descent of a learnable energy.
This approach explainsthe GNN evolution from a multi-particle perspective as learning attractive andrepulsive forces in feature space via the positive and negative eigenvalues of asymmetric ‘channel-mixing’ matrix.
We perform spectral analysis of the solutionsand conclude that gradient ﬂow graph convolutional models can induce a dynamicsdominated by the graph high frequencies, which is desirable for heterophilicdatasets.
We also describe structural constraints on common GNN architecturesallowing to interpret them as gradient ﬂows.
We perform thorough ablation studiescorroborating our theoretical analysis and show competitive performance of simpleand lightweight models on real-world homophilic and heterophilic datasets
Introduction and motivations
Graph neural networks (GNNs) [38, 20, 21, 36, 7, 15, 27] and in particular their Message Passing 14 formulation (MPNN) [19] have become the standard ML tool for dealing with different types ofrelations and interactions, ranging from social networks to particle physics and drug design.
One 16 of the often cited drawbacks of traditional GNN models is their poor ‘explainability’, making ithard to know why and how they make certain predictions [46, 47], and in which situations they 18 may work and when they would fail.
Limitations of GNNs that have attracted attention are oversmoothing [29, 30, 8], over-squashing and bottlenecks [1, 40], and performance on heterophilic data 20 [31, 51, 13, 4, 45] – where adjacent nodes usually have different labels.
attractive and repulsive forces lead to a non-smoothing process able to separate labels.
Contributions.
We propose a Gradient Flow Framework 22 ( GRAFF ) where the GNN equations follow the direction of steepest descent of a learnable energy .
Thanks to this framework we can 24 (i) interpret GNNs as a multi-particle dynamics where the learnedparameters determine pairwise attractive and repulsive potentials 26 in the feature space.
This sheds light on how GNNs can adapt toheterophily and explains their performance and the smoothness of 28 the prediction.
(ii) GRAFF leads to residual convolutional modelswhere the channel-mixing W is performed by a shared symmet- 30 ric bilinear form inducing attraction and repulsion via its positiveand negative eigenvalues, respectively.
We theoretically investi- 32 gate the interaction of the graph spectrum with the spectrum of thechannel-mixing, proving that if there is more mass on the negative 34 eigenvalues of W , then the dynamics is dominated by the graphhigh frequencies, which could be desirable on heterophilic graphs.
36 We also extend results of [29, 30, 8] by showing that when we dropthe residual connection intrinsic to the gradient ﬂow framework, graph convolutional models always induce a low-frequency dominated dynamics independent of the 39 sign and magnitude of the spectrum of the channel-mixing.
We also discuss how simple choicesmake common architectures ﬁt GRAFF and conduct thorough ablation studies to corroborate the 41 theoretical analysis on the role of the spectrum of W .
(iii)
We crystallize an instance of our framework into a linear, residual, convolutional model that retains explainability and achieves competitive 43 performance on homophilic and heterophilic real world graphs whilst being faster than GCN.
Related work.
Our analysis is related to studying GNNs as ﬁlters on the graph spectrum [15, 24, 45 2, 25] and over-smoothing [29, 30, 8, 50] and partly adopts techniques similar to [30].
The keydifference is that we also consider the spectrum of the ‘channel-mixing’ matrix.
The concept of 47 gradient ﬂows has been a standard tool in physics and geometry [16], from which they were adoptedfor image processing [26], and recently used in ML [35] for the analysis of Transformers [41] – see 49 also [18] for discussion of loss landscapes.
Our continuous-time evolution equations follows the spiritof Neural ODES [22, 12, 3] and the study of GNNs as continuous dynamical systems [44, 10, 17, 9].
Outline.
In Section 2, we review the continuous and discrete Dirichlet energy and the associated 52 gradient ﬂow framework.
We formalize the notion of over-smoothing and low(high)-frequencydominated dynamics to investigate GNNs and study the dominant components in their evolution.
We 54 extend the graph Dirichlet energy to allow for a non-trivial norm for the feature edge-gradient.
Thisleads to gradient ﬂow equations that diffuse the features and over-smooth in the limit.
Accordingly, 56 in Section 3 we introduce a more general energy with a symmetric channel-mixing matrix W givingrise to attractive and repulsive pairwise terms via its positive and negative eigenvalues and show 58 that the negative spectrum can induce high-frequency-dominant dynamics.
In Section 4 we ﬁrstcompare with continuous GNN models and then discretize the equations and provide a ‘recipe’ for 60 making standard GNN architectures ﬁt a gradient ﬂow framework.
We adapt the spectral analysis todiscrete-time showing that gradient ﬂow convolutional models can generate a dynamics dominated by 62 the high frequencies via the negative eigenvalues of W while this is impossible if we drop the residualconnection.
In Section 5 we corroborate our theoretical analysis on the role of the spectrum of W 64 via ablation studies on graphs with varying homophily.
Experiments on real world datasets show acompetitive performance of our model despite its simplicity and reduced number of parameters.
Gradient-ﬂow formalism
Notations adopted throughout the paper.
Let G =
( V , E ) be an undirected graph with n nodes.
We denote by F 2 R n ⇥ d the matrix of d -dimensional node features, by f i 2 R d
its i -th row 69 (transposed), by f r 2 R n its r -th column, and by vec( F ) 2 R nd the vectorization of F obtained 70 by stacking its columns.
Given a symmetric matrix B , we let  B + ,  B  denote its most positive and 71 negative eigenvalues, respectively, and ⇢ B be its spectral radius .
If B ⌫ 0 , then gap( B ) denotes thepositive smallest eigenvalue of B . ˙
f ( t ) denotes the temporal derivative, ⌦ is the Kronecker product 73 and ‘a.e.’ means almost every w.r.t.
Lebesgue measure and usually refers to data in the complementof some lower dimensional subspace in R n ⇥ d .
Proofs and additional results appear in the Appendix.
Starting point: a geometric parallelism.
To motivate a gradient-ﬂow approach for GNNs, we start 76 from the continuous case (see Appendix A.1 for details).
Consider a smooth map f : R n !
( R d , h ) 77 with h a constant metric represented by H ⌫ 0 .
The Dirichlet energy of f is deﬁned by
[Equation]
(1) and measures the ‘smoothness’ of f .
A natural approach to ﬁnd minimizers of E - called harmonic 79 maps - was introduced in [16] and consists in studying the gradient ﬂow of
E , wherein a given mapf (0) =
f 0 is evolved according to ˙ f ( t ) =
r
f E ( f ( t )) .
These type of evolution equations have 81 historically been the core of variational and PDE-based image processing ; in particular, gradientﬂows of the Dirichlet energy were shown [26] to recover the Perona-Malik nonlinear diffusion [32].
Motivation: GNNs for node-classiﬁcation.
We wish to extend the gradient ﬂow formalism to node 84 classiﬁcation on graphs.
Assume we have a graph G , node-features F 0 and labels { y i } on V train ⇢ V , 85 and that we want to predict the labels on V test ⇢ V .
A GNN typically evolves the features via some parametric rule, GNN ✓ ( G , F 0 ) , and uses a decoding map for the prediction y
= DE (GNN ✓ ( G , F 0 )) .
In graph convolutional models [15, 27], GNN ✓ consists of two operations: applying a shared lineartransformation to the features ( ‘channel mixing’ ) and propagating them along the edges of the graph 89 ( ‘diffusion’ ).
Our goal consists in studying when GNN ✓ is the gradient ﬂow of some parametric classof energies E ✓ : R n ⇥ d !
R , which generalize the Dirichlet energy.
This means that the parameters 91 can be interpreted as ‘ﬁnding the right notion of smoothness’ for our task.
We evolve the features by˙ F ( t ) =
r
F E ✓ ( F ( t )) with prediction y = DE ( F ( T ))
for some optimal time T .
Why a gradient ﬂow?
Since ˙ E ✓ ( F ( t )) =
||
r F E ✓ ( F ( t ))
|| 2 , the energy dissipates along the gradient 94 ﬂow.
Accordingly, this framework allows to explain the GNN dynamics as ﬂowing the node featuresin the direction of steepest descent of E ✓ .
Indeed, we ﬁnd that parametrizing an energy leads to 96 equations governed by attractive and repulsive forces that can be controlled via the spectrum ofsymmetric ‘channel-mixing’ matrices.
This shows that by learning to distribute more mass over the 98 negative (positive) eigenvalues of the channel-mixing, gradient ﬂow models can generate dynamicsdominated by the higher (respectively, lower) graph frequencies and hence tackle different homophily 100 scenarios.
The gradient ﬂow framework also leads to sharing of the weights across layers (since weparametrize the energy rather than the evolution equations , as usually done in GNNs), allowing us to 102 reduce the number of parameters without compromising performance (see Table 1).
Analysis on graphs: preliminaries.
Given a connected graph G with self-loops, its adjacency 104 matrix A is deﬁned as a ij = 1 if ( i, j ) 2 E and zero otherwise.
We let D =
diag
( d i ) be the degreematrix and write ¯ A :
= D  1 / 2 AD  1 / 2 .
Let F 2 R n ⇥ d be the matrix representation of a signal.
Itsgraph gradient is ( r F ) ij :=
f j / p d j  f i / p d i .
We deﬁne the Laplacian as  :=
12 div r (thedivergence div is the adjoint of r ), represented by
= I  ¯
A ⌫ 0 .
We refer to the eigenvalues of 108  as frequencies : the lowest frequency is always 0 while the highest frequency is ⇢   2 [14].
As 109 for the continuum case, the gradient allows to deﬁne a (graph) Dirichlet energy as [49]
[Equation]
where the extra 12 is for convenience.
As for manifolds, E Dir measures smoothness.
If we stack thecolumns of F into vec( F ) 2 R nd , the gradient ﬂow of E Dir yields the heat equation on each channel:
[Equation]
for 1  r  d .
Similarly to [8], we rely on E Dir to assess whether a given dynamics t 7!
F ( t ) is a 113 smoothing process.
A different choice of Laplacian L = D  A with non-normalized adjacency 114 induces the analogous Dirichlet energy
E
[Equation]
F )
[Equation]
12 trace( F > LF ) .
Throughout this paper, we rely 115 on the following deﬁnitions (see Appendix A.3 for further equivalent formulations and justiﬁcations):
Deﬁnition 2.1. ˙
F ( t ) =
GNN ✓ ( F ( t ) , t ) initialized at F (0) is smoothing if E Dir ( F ( t ))  C + ' ( t ) , 117 with C a constant only depending on
E Dir ( F (0)) and ˙ ' ( t )  0 .
Over-smoothing occurs if either
E Dir ( F ( t )) !
0 or E Dir L ( F ( t )) !
for t ! 1 .
Our notion of ‘over-smoothing’ is a relaxed version of the deﬁnition in [34] – although in the linear 120 case one always ﬁnds an exponential decay of
E Dir .
We note that E Dir ( F ( t )) !
iff  f r ( t ) !
for 121 each column f r .
As in [30], this corresponds to a loss of separation power along the solution wherenodes with equal degree become indistinguishable since we converge to ker(  ) (if we replaced with L then we would not even be able to separate nodes with different degrees in the limit).
To motivate the next deﬁnition, consider ˙ F ( t )
=
¯ AF ( t ) .
Despite || F ( t )
||
being unbounded for a.e. 125 F (0) , the low-frequency components are growing the fastest and indeed F ( t ) /
|| F ( t )
|| !
F 1 s.t.
f r 1 = 0 for 1  r  d .
We formalize this scenario – including the opposite case of high-frequency 127 components being dominant – by studying
E Dir ( F ( t ) / || F ( t ) || ) , i.e. the Rayleigh quotient of I d ⌦  .
Deﬁnition 2.2. ˙
F ( t ) =
GNN ✓ ( F ( t ) , t ) initialized at F (0) is Low/High - Frequency-Dominant(L/HFD) if E Dir ( F ( t ) / || F ( t ) || ) !
(respectively, E Dir ( F ( t ) / || F ( t ) || ) !
⇢  / 2 ) for t ! 1 .
We report a consequence of Deﬁnition 2.2 and refer to Appendix A.3 for additional details
Lemma 2.3. is
( HFD iff
If a graph is homophilic , adjacent nodes are likely to share the same label and we expect a smoothing 135 or LFD dynamics enhancing the low-frequency components to be successful at node classiﬁcationtasks [43, 28].
In the opposite case of heterophily , the high-frequency components might contain more 137 relevant information for separating classes [4, 5] – the prototypical example being the eigenvector of associated with largest frequency ⇢  separating a regular bipartite graph.
In other words, the class 139 of heterophilic graphs contain instances where signals should be sharpened by increasing
E Dir rather 140 than smoothed out.
Accordingly, an ideal framework for learning on graphs must accommodate bothof these opposite scenarios by being able to induce either an LFD or a HFD dynamics.
Parametric Dirichlet energy: channel-mixing as metric in feature space.
In eq.
(1) a constant 143 nontrivial metric h in R d leads to the mixing of the feature channels.
We adapt this idea by consideringa symmetric positive semi-deﬁnite H = W > W with W 2 R d ⇥ d and using it to generalize E Dir as
[Equation]
We note the analogy with eq.
(1), where the sum over the nodes replaces the integration over the 146 domain and the j -th derivative at some point i is replaced by the gradient along the edge ( i, j ) 2 E .
We generally treat W as learnable weights and study the gradient ﬂow of E Dir W :
[Equation]
We see that eq.
(5) generalizes eq.
Below ‘smoothing’ is intended as in Deﬁnition 2.1.
Proposition 2.4.
Let P ker W be the projection onto ker( W > W ) .
Equation (5) is smoothing since
E Dir ( F ( t ))
 e  2 t gap( W > W )gap(
[Equation]
[Equation]
( F
[Equation]
, t
[Equation]
In fact F ( t ) !
F 1 s.t. 9  1 2 R d : for each i 2 V we have ( f 1 ) i
= p d i  1 + P ker W f i (0) .
Proposition 2.4 implies that no weight matrix W in eq.
(5) can separate the limit embeddings F ( 1 ) 152 of nodes with same degree and input features .
If W has a trivial kernel, then nodes with same degreesconverge to the same representation and over-smoothing occurs as per Deﬁnition 2.1.
Differently 154 from [29, 30, 8], over-smoothing occurs independently of the spectral radius of the ‘channel-mixing’if its eigenvalues are positive – even for equations which lead to residual GNNs when discretized 156 [12].
According to Proposition 2.4, we do not expect eq.
(5) to succeed on heterophilic graphs wheresmoothing processes are generally harmful – this is conﬁrmed in Figure 2 (see prod -curve).
To 158 remedy this problem, we generalize eq.
(5) to a gradient ﬂow that can be HFD as per Deﬁnition 2.2.
A general parametric energy for pairwise interactions
We ﬁrst rewrite the energy E Dir W in eq.
(4) as
[Equation]
We then deﬁne a new, more general energy by replacing the occurrences of W > W with new 162 symmetric matrices ⌦ , W 2 R d ⇥ d since we also want to generate repulsive forces:
[Equation]
with associated gradient ﬂow of the form (see Appendix B)
[Equation]
Note that eq.
(8) is gradient ﬂow of some energy F 7!
E tot ( F ) iff both ⌦ and W are symmetric .
A multi-particle system point of view: attraction vs repulsion.
Consider the d -dimensional 166 node-features as particles in R d with energy
E tot .
While the term E ext ⌦ is independent of the graphtopology and represents an external ﬁeld in the feature space, the second term E pair W constitutes a 168 potential energy, with W a bilinear form determining the pairwise interactions of adjacent node representations.
Given a symmetric W , we write W = ⇥ > + ⇥ +  ⇥ > ⇥  , by decomposing thespectrum of W in positive and negative values.
We can rewrite E tot =
E ext ⌦  W + E Dir
[Equation]
+  [Equation] Dir
[Equation]
, i.
[Equation]
[Equation]
The gradient ﬂow of E tot minimizes E Dir ⇥ + and maximizes E Dir ⇥  .
The matrix W encodes repulsivepairwise interactions via its negative-deﬁnite component ⇥  which lead to terms || ⇥
( r F ) ij || 173 increasing along the solution.
The latter affords a ‘sharpening’ effect desirable on heterophilic graphswhere we need to disentangle adjacent node representations and hence ‘magnify’ the edge-gradient.
Spectral analysis of the channel-mixing.
We will now show that eq.
(8) can lead to a HFDdynamics.
To this end, we assume that ⌦ =
so that eq.
(8) becomes ˙ F ( t ) =
¯ AF ( t ) W .
Accordingto eq.
(9) the negative eigenvalues of W lead to repulsion.
We show that the latter can induce HFD 178 dynamics as per Deﬁnition 2.2.
We let P ⇢  W be the orthogonal projection into the eigenspace of
W ⌦ ¯ A associated with the eigenvalue ⇢  := |  W  | ( ⇢   1) .
We deﬁne ✏ HFD explicitly in eq.
Proposition 3.1.
If ⇢  >  W + , then ˙ F ( t ) = ¯ AF ( t ) W is HFD for a.e. F (0) : there exists ✏ HFD s.t.
[Equation]
and F ( t ) /
|| F ( t ) || converges to F 1 2 R n ⇥ d
such that  f r 1 = ⇢  f r 1 , for 1  r  d .
Proposition 3.1 shows that if enough mass of the spectrum of the ‘channel-mixing’ is distributed over 183 the negative eigenvalues, then the evolution is dominated by the graph high frequencies .
This analysisis made possible in our gradient ﬂow framework where W must be symmetric .
The HFD dynamics 185 induced by negative eigenvalues of W is conﬁrmed in Figure 2 ( neg-prod -curve in the bottom chart).
A more general energy.
Equations with a source term may have better expressive power [44, 11, 39].
In our framework this means adding an extra energy term of the form E source˜ W
[Equation]
to eq.
(7) with some learnable  and ˜ W .
This leads to the following gradient ﬂow:
[Equation]
We also observe that one could replace the ﬁxed matrix ¯ A with a more general symmetric graph 190 vector ﬁeld
A satisfying A ij = 0 if ( i, j ) / 2 E , although in this work we focus on the case
A
= ¯ A .
We also note that when ⌦ = W , then eq.
(8) becomes ˙ F ( t ) =
F ( t ) W .
We perform a spectral 192 analysis of this case in Appendix B.2.
Non-linear activations.
In Appendix B.3 we discuss non-linear gradient ﬂow equations. Here 194 we study what happens if the gradient ﬂow in eq. (10) is activated pointwise by  : R !
R .
We 195 show that although we are no longer a gradient ﬂow, the learnable multi-particle energy E tot is stilldecreasing along the solution, meaning that the interpretation of the channel-mixing W inducing 197 attraction and repulsion via its positive and negative eigenvalues respectively is preserved .
Proposition 3.2.
Consider a non-linear map  : R ! R such that the function x 7! x  ( x )  0 . If 199 t 7!
F ( t ) solves the equation
[Equation]
where  acts elementwise, thend E tot
[Equation]
A proof of this result and more details and discussion are reported in
Appendix E. We emphasize 202 here that differently from previous results about behaviour of ReLU wrt
E Dir [30, 8], we deal with a 203 much more general energy that can also induce repulsion and a more general family of activationfunctions (that include ReLU , tanh , arctan and many others).
Comparison with GNNs
In this Section, we study standard GNN models from the perspective of our gradient ﬂow framework.
4.1 Continuous case
Continuous GNN models replace layers with continuous time.
In contrast with Proposition 3.1, 209 we show that three main linearized continuous GNN models are either smoothing or LFD asper Deﬁnition 2.2.
The linearized PDE-GCN D model [17] corresponds to choosing  = 0 and 211 ⌦ =
W = K ( t ) >
K ( t ) in eq.
(10), for some time-dependent family t
[Equation] [Equation]
The CGNN model [44] can be derived from eq.
(10) by setting ⌦
[Equation] [Equation]
Finally, in linearized GRAND [10] a row-stochastic matrix A ( F (0)) is learned from the encoding 214 via an attention mechanism and we have
[Equation]
We note that if A is not symmetric, then GRAND is not a gradient ﬂow.
Proposition 4.1.
PDE  GCN D , CGNN and GRAND satisfy the following:
(i) PDE  GCN D is a smoothing model: ˙ E Dir ( F PDE  GCN D ( t ))  0 .(ii)
For a.e. F (0) it holds: CGNN is never HFD and if we remove the source term, thenE Dir ( F CGNN ( t ) /
|| F CGNN ( t ) || )  e  gap(  ) t .(iii)
If G is connected, F GRAND ( t ) !
µ as t ! 1 , with µ r = mean( f r (0)) , 1  r  d . By (ii) the
source-free CGNN-evolution is LFD independent of ˜ ⌦
Moreover, by (iii over-smoothingoccurs for GRAND as per Deﬁnition 2.1.
On the other hand, Proposition 3.1 shows that the negative 223 eigenvalues of W can make the source-free gradient ﬂow in eq.
(8) HFD .
Experiments in Section 5conﬁrm that the gradient ﬂow model outperforms CGNN and GRAND on heterophilic graphs.
4.2 Discrete case
We now describe a discrete version of our gradient ﬂow model and compare it to ‘discrete’ GNNs 227 where discrete time steps correspond to different layers.
In the spirit of [12], we use explicit Eulerscheme with step size ⌧  1 to solve eq.
(10) and set ˜ W
= I .
In the gradient ﬂow framework we 229 parametrize the energy rather than the actual equations, which leads to symmetric channel-mixingmatrices ⌦ , W 2 R d ⇥ d that are shared across the layers .
Since the matrices are square, an encoding 231 block
EN : R n ⇥ p !
R n ⇥ d is used to process input features F 0 2 R n ⇥ p and generally reduce the 232 hidden dimension from p to d .
Moreover, the iterations inherently lead to a residual architecturebecause of the explicit Euler discretization:
[Equation]
with prediction y = DE ( F ( T )) produced by a decoder DE : R n ⇥ d !
R n ⇥ k , where k is the 235 number of label classes and T integration time of the form T = m ⌧ , so that m 2 N represents the 236 number of layers .
Although eq.
(11) is linear, we can include non-linear activations in EN , DEmaking the entire model generally non-linear.
We emphasize two important points: • Since the framework is residual, even if the message-passing is linear, this is not equivalentto collapsing the dynamics into a single layer with diffusion matrix ¯ A m , with m the numberof layers, see eq.
(27) in the appendix where we derive the expansion of the solution.• We could also activate the equations pointwise and maintain the physics interpretation thanksto Proposition 3.2 to gain greater expressive power. In the following though, we mainlystick to the linear discrete gradient ﬂow unless otherwise stated.
Are discrete GNNs gradient ﬂows?
Given a (learned) symmetric graph vector ﬁeld A 2 R n ⇥ n 245 satisfying
A ij = 0 if ( i, j ) / 2 E , consider a family of linear GNNs with shared weights of the form
[Equation]
Symmetry is the key requirement to interpret GNNs in eq.
(12) in a gradient ﬂow framework.
Lemma 4.2.
Equation (12) is the unit step size discrete gradient ﬂow of E
ext
I  ⌦ + E pair A , W  E source˜ W ,with E pair A , W deﬁned by replacing ¯ A with A in eq.
(7) , iff ⌦ and W are symmetric.
Lemma 4.2 provides a recipe for making standard architectures into a gradient ﬂow, with symmetry 250 being the key requirement.
When eq.
(12) is a gradient ﬂow, the underlying GNN dynamics isequivalent to minimizing a multi-particle energy by learning attractive and repulsive directions in 252 feature space as discussed in Section 3.
In Appendix C.2, we show how Lemma 4.2 covers linearversions of GCN [27, 43], GAT [42], GraphSAGE [23] and GCNII [11] to name a few.
Over-smoothing analysis in discrete setting.
By Proposition 3.1 we know that the continuous 255 version of eq.
(11) can be HFD thanks to the negative eigenvalues of W .
The next result represents adiscrete counterpart of Proposition 3.1 and shows that residual, symmetrized graph convolutional 257 models can be HFD .
Below P ⇢  W is the projection into the eigenspace associated with the eigenvalue⇢  :=
|  W  | ( ⇢   1) and we report the explicit value of  HFD in eq.
(28) in Appendix C.3.
We let:
[Equation]
Theorem 4.3.
Given F ( t + ⌧ ) =
F ( t ) + ⌧ ¯ AF ( t ) W , with W symmetric, if eq.
(13) holds then
E
[Equation]
hence the dynamics is HFD for a.e. F (0) and in fact F ( m ⌧ ) / || F ( m ⌧ ) || !
F 1 s.t.
f r 1 = ⇢
{+ f r 1 . 
+}
Conversely, if G is not bipartite, then for a.e. F (0) the system F ( t + ⌧ ) =
⌧ ¯ AF ( t ) W , with Wsymmetric, is LFD independent of the spectrum of W .
Theorem 4.3 shows that linear discrete gradient ﬂows can be HFD due to the negative eigenvalues of 264 W .
This differs from statements that standard GCNs act as low-pass ﬁlters and thus over-smooth inthe limit.
Indeed, in these cases the spectrum of W is generally ignored [43, 11] or required to be 266 sufﬁciently small in terms of singular value decomposition [29, 30, 8] when no residual connectionis present .
On the other hand, Theorem 4.3 emphasizes that the spectrum of W plays a key role to 268 enhance the high frequencies when enough mass is distributed over the negative eigenvalues providedthat a residual connection exists – this is conﬁrmed by the neg-prod -curve in Figure 2.
The residual connection from a spectral perspective.
Given a sufﬁciently small step-size so 271 that the right hand side of inequality 13 is satisﬁed, F ( t + ⌧ ) =
F ( t ) + ⌧ ¯ AF ( t ) W is HFD for a.e.
F (0) if |  W  | ( ⇢   1) >  W + , i.e. ‘there is more mass’ in the negative spectrum of W than in the 273 positive one.
This means that differently from [29, 30, 8], there is no requirement on the minimalmagnitude of the spectral radius of W coming from the graph topology as long as  W + is small 275 enough.
Conversely, without a residual term, the dynamics is LFD for a.e. F (0) independently of thesign and magnitude of the eigenvalues of W .
This is also conﬁrmed by the GCN-curve in Figure 2.
Over-smoothing vs LFD .
We highlight how in general a linear GCN equation as F ( t + ⌧ ) = 278 ⌧ ¯ AF ( t ) W may avoid over-smoothing in the sense of Deﬁnition 2.1, meaning that
E Dir ( F ( t )) !
1 279 as soon as there exist   i 2 (0 , 1) and the spectral radius of W is large enough.
However, this 280 will not lead to over-separation since the dominating term is the lowest frequency one: in otherwords, once we re-set the scale right as per the normalization in Theorem 4.3, we encounter loss of 282 separability even with large (and possibly negative) spectrum of W .
Experiments
In this section we evaluate the gradient ﬂow framework ( GRAFF ).
We corroborate the spectral 285 analysis using synthetic data with controllable homophily.
We conﬁrm that having negative (positive)eigenvalues of the channel-mixing W are essential in heterophilic (homophilic) scenarios where the 287 gradient ﬂow should align with HFD ( LFD ) respectively.
We show that the gradient ﬂow in eq.
– a linear, residual, symmetric graph convolutional model – achieves competitive performance on real 289 world datasets.
Methodology.
We crystallize GRAFF in the model presented in eq.
(11) with EN , DE im- 291 plemented as single linear layers or MLPs, and we set ⌦ to be diagonal.
For the real-worldexperiments we consider diagonally-dominant (DD) , diagonal (D) and time-dependent choicesfor the structure of W that offer explicit control over its spectrum.
In the (DD) -case, we considera W 0 2 R d ⇥ d symmetric with zero diagonal and w 2 R d deﬁned by w ↵ =
q ↵
P  | W 0 ↵ | + r ↵ ,and set W = diag( w ) + W 0 .
Due to the Gershgorin Theorem the eigenvalues of W belong to 296 [ w ↵
P  | W 0 ↵ | , w ↵ +
P  | W 0 ↵ | ] , so the model ‘can’ easily re-distribute mass in the spectrum of
W via q ↵ , r ↵ .
This generalizes the decomposition of W in [11] providing a justiﬁcation in terms of 298 its spectrum and turns out to be more efﬁcient w.r.t.
the hidden dimension d as shown in Figure 4 inthe Appendix.
For (D) we take W to be diagonal, with entries sampled U [  1 , 1] and ﬁxed – i.e., wedo not train over W – and only learn EN , DE .
We also include a time-dependent model where W t 301 varies across layers.
To investigate the role of the spectrum of W on synthetic graphs, we constructthree additional variants: W = W 0 + W 0> , W = ± W 0> W 0 named sum , prod and neg-prod 303 respectively where prod ( neg-prod ) variants have only non-negative (non-positive) eigenvalues.
Complexity and number of parameters.
If we treat the number of layers as a constant, the discrete 305 gradient ﬂow scales as
O ( | V | pd + | E | d 2 ) , where p and d are input feature and hidden dimension 306 respectively, with p  d usually.
Note that GCN has complexity O ( | E | pd ) and in fact our model is 307 faster than GCN as conﬁrmed in Figure 5 in Appendix D. Since EN , DE are single linear layers(MLPs), we can bound the number of parameters by pd + d 2 + 3 d + dk , with k the number of labelclasses, in the (DD) -variant while in the (D) -variant we have pd + 3 d + dk .
Further ablation studiesappear in Figure 4 in the Appendix showing that (DD) outperforms sum and GCN – especially in the 311 lower hidden dimension regime – on real-world benchmarks with varying homophily.
Synthetic experiments and ablation studies
313 To investigate our claims in a controlled environment we use the synthetic Cora dataset of [51, Ap- 315 pendix G].
Graphs are generated for target levelsof homophily via preferential attachment – see 317 Appendix D.3 for details.
Figure 2 conﬁrms thespectral analysis and offers explainability in terms 319 of performance and smoothness of the predictions.
Each curve – except GCN – represents one ver- 321 sion of W as in ‘methodology’ and we implementeq.
(11) with  = 0 , ⌦ = 0 .
Figure 2 (top) reportsthe test accuracy vs true label homophily.
Neg-prod 324 is better than prod on low-homophily and viceversaon high-homophily.
This conﬁrms Proposition 3.1 326 where we have shown that the gradient ﬂow canlead to a HFD dynamics – that are generally de- 328 sirable with low-homophily – through the negativeeigenvalues of W .
Conversely, the prod conﬁgura- 330 tion (where we have an attraction-only dynamics)struggles in low-homophily scenarios even though 332 a residual connection is present .
Both prod and neg-prod are ‘extreme’ choices and serve the purposeof highlighting that by turning off one side of the spectrum this could be the more damaging depend- 334 ing on the underlying homophily.
In general though ‘neutral’ variants like sum and (DD) are indeedmore ﬂexible and better performing.
In fact, (DD) outperforms GCN especially in low-homophily 336 scenarios, conﬁrming Theorem 4.3 where we have shown that without a residual connection convolutional models are LFD – and hence more sensitive to underlying homophily – irrespectively of the 338 spectrum of W .
This is further conﬁrmed by additional ablation studies in Figure 3.
In Figure 2 (bottom) we compute the homophily of the prediction (cross) for a given method and we 340 compare with the homophily (circle) of the prediction read from the encoding (i.e. graph-agnostic ).
The homophily here is a proxy to assess whether the evolution is smoothing , the goal being explaining 342 the smoothness of the prediction via the spectrum of W as per our theoretical analysis.
For neg-prodthe homophily after the evolution is lower than that of the encoding, supporting the analysis that 344 negative eigenvalues of W enhance high-frequencies.
The opposite behaviour occurs in the case ofprod and explains that in the low-homophily regime prod is under-performant due to the prediction
[Table]
being smoother than the true homophily.
(DD) and sum variants adapt better to the true homophily.
347 We note how the encoding compensates when the dynamics can only either attract or repulse (i.e. the
[Table]
spectrum of W has a sign) by decreasing or increasing the initial homophily respectively.
[Table]
Real world experiments.
We test GRAFF against a range of datasets with varying homophily 350 [37, 33, 31] (see Appendix D.4 for additional details).
We use results provided in [45, Table 1],which includes standard baselines as GCN [27], GraphSAGE [23], GAT [42], PairNorm [48] and 352 recent models tailored towards the heterophilic setting (GGCN [45], Geom-GCN [31], H2GCN[51] and GPRGNN [13]).
For Sheaf [5], a recent top-performer on heterophilic datasets, we took 354 the best performing variant (out of six provided) for each dataset.
We also include continuousbaselines CGNN [44] and GRAND [10] to provide empirical evidence for Proposition 4.1.
Splits 356 taken from [31] are used in all the comparisons.
The GRAFF model discussed in ‘methodology’is a very simple architecture with shared parameters across layers and run-time smaller than GCN 358 and more recent models like GGCN designed for heterophilic graphs (see Figure 5 in the Appendix).
Nevertheless, it achieves competitive results on all datasets, performing on par or better than more 360 complex recent models.
Moreover, comparison with the ‘time-dependent’ (DD) variant conﬁrmsthat by sharing weights across layers we do not lose performance.
We note that on heterophilic 362 graphs short integration time is usually needed due to the topology being harmful and the negativeeigenvalues of W leading to exponential behaviour (see Appendix D).
Conclusions
In this work, we developed a framework for explainable GNNs where the evolution can be interpreted 366 as minimizing a multi-particle learnable energy.
This translates into studying the interaction betweenthe spectrum of the graph and the spectrum of the ‘channel-mixing’ leading to a better understanding 368 of when and why the induced dynamics is low (high) frequency dominated.
From a theoreticalperspective, we reﬁned existing asymptotic analysis of GNNs to account for the role of the spectrum 370 of the channel-mixing as well.
From a practical perspective, our framework allows for ‘educated’choices resulting in a simple, more explainable convolutional model that achieves competitive 372 performance on homophilic and heterophilic benchmarks while being faster than GCN.
Our resultsrefute the folklore of graph convolutional models being too simple for complex benchmarks.
Limitations and future works.
We limited our attention to a constant bilinear form W , which 375 might be excessively rigid.
It is possible to derive non-constant alternatives that are aware of thefeatures or the position in the graph.
The main challenge amounts to matching the requirement for 377 local ‘heterogeneity’ with efﬁciency: we reserve this question for future work.
Our analysis is also aﬁrst step into studying the interaction of the graph and ‘channel-mixing’ spectra; we did not explore 379 other dynamics that are neither LFD nor HFD as per our deﬁnitions.
The energy formulation pointsto new models more ‘physics’ inspired this will be explored in future work.
Societal impact.
Our work sheds light on the actual dynamics of GNNs and could hence improve 382 their explainability, which is crucial for assessing their impact on large-scale applications.
We alsoshow that instances of our framework achieve competitive performance on heterophilic data despite 384 being faster than GCN , providing evidence for efﬁcient methods with reduced footprint.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on 524 how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing 526 the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the 532 Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] , in Section 6.(c) Did you discuss any potential negative societal impacts of your work?
[Yes] in theSocietal impact paragraph in Section 6.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes] in Appendix A,Appendix B and Appendix C.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Code andREADME in SM, dataloaders in code(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Splits and hyperparameters provided in code zip(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Standard deviations are stated in results table(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] in appendix D4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes] datasets and standardlibraries cited in appendix D(b) Did you mention the license of the assets?
[Yes] industry standard libraries andbenchmark datasets were used in accordance with licences(c)
Did you include any new assets either in the supplemental material or as a URL?
[Yes]code provided in SM zip(d)
Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[Yes] no personal data is contained within benchmarking datasets5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]