Graph Neural Networks as Gradient Flows
AfﬁliationAddress
Abstract
Dynamical systems minimizing an energy are ubiquitous in geometry and physics.
We propose a gradient ﬂow framework for GNNs where the equations follow thedirection of steepest descent of a learnable energy.
This approach allows to explainthe GNN evolution from a multi-particle perspective as learning attractive andrepulsive forces in feature space via the positive and negative eigenvalues of asymmetric ‘channel-mixing’ matrix.
We perform spectral analysis of the solutionsand conclude that gradient ﬂow graph convolutional models can induce a dynamics dominated by the graph high frequencies which is desirable for heterophilicdatasets.
We also describe structural constraints on common GNN architecturesallowing to interpret them as gradient ﬂows.
We perform thorough ablation studiescorroborating our theoretical analysis and show competitive performance of simpleand lightweight models on real-world homophilic and heterophilic datasets
Introduction and motivations
Graph neural networks (GNNs) [35, 17, 18, 33, 6, 13, 24] and in particular their Message Passing 14 formulation (MPNN) [16] have become the standard ML tool for dealing with different types ofrelations and interactions, ranging from social networks to particle physics and drug design.
One 16 of the often cited drawbacks of traditional GNN models is their poor ‘explainability’, making ithard to know why and how they make certain predictions [43, 44], and in which situations they 18 may work and when they would fail.
Limitations of GNNs that have attracted attention are oversmoothing [26, 27, 7], over-squashing and bottlenecks [1, 37], and performance on heterophilic data 20 [28, 47, 11, 4, 42] – where adjacent nodes usally have different labels.
attractive and repulsive forces lead to a non-smoothing process able to separate labels.
Contributions.
We propose a Gradient Flow Framework 22 ( GRAFF ) where the GNN equations follow the direction of steepest descent of a learnable energy .
Thanks to this framework we can 24 (i) interpret GNNs as a multi-particle dynamics where the learnedparameters determine pairwise attractive and repulsive potentials 26 in the feature space.
This sheds light on how GNNs can adapt toheterophily and explains their performance and the smoothness of 28 the prediction.
(ii) GRAFF leads to residual convolutional modelswhere the channel-mixing W is performed by a shared symmet- 30 ric bilinear form inducing attraction and repulsion via its positiveand negative eigenvalues, respectively.
We theoretically investi- 32 gate the interaction of the graph spectrum with the spectrum of thechannel-mixing proving that if there is more mass on the negative 34 eigenvalues of W , then the dynamics is dominated by the graphhigh frequencies, which could be desirable on heterophilic graphs.
36 We also extend results of [26, 27, 7] by showing that when we dropthe residual connection intrinsic to the gradient ﬂow framework, graph convolutional models always induce a low-frequency dominated dynamics independent of the 39 sign and magnitude of the spectrum of the channel-mixing.
We also discuss how simple choicesmake common architectures ﬁt GRAFF and conduct thorough ablation studies to corroborate the 41 theoretical analysis on the role of the spectrum of W .
(iii)
We crystallize an instance of our framework into a linear, residual, convolutional model that retains explainability and achieves competitive 43 performance on homophilic and heterophilic real world graphs whilst being faster than GCN.
Related work.
Our analysis is related to investigating GNNs as ﬁlters on the graph spectrum 45 [13, 21, 2, 22] and studying the over-smoothing effect [26, 27, 7] and partly adopts techniques similarto [27].
The key difference is that we also consider the spectrum of the ‘channel-mixing’ matrix.
The 47 concept of gradient ﬂows has been a standard tool in physics and differential geometry [14], fromwhich they were adopted for image processing [23], and more recently also used in ML [32] for the 49 analysis of Transformers [38].
Our continuous-time evolution equations follows the spirit of Neural ODES [19, 10, 3] and the study of GNNs as continuous dynamical systems [41, 8, 15].
Outline.
In Section 2, we review the continuous and discrete Dirichlet energy and the associated 52 gradient ﬂow framework.
We formalize the notion of over-smoothing and low(high)-frequencydominated dynamics to investigate GNNs and study the dominant components in their evolution.
We 54 extend the graph Dirichlet energy to allow for a non-trivial norm for the feature edge-gradient.
Thisleads to gradient ﬂow equations that diffuse the features and over-smooth in the limit.
Accordingly, 56 in Section 3 we introduce a more general energy with a symmetric channel-mixing matrix W givingrise to attractive and repulsive pairwise terms via its positive and negative eigenvalues and show 58 that the negative spectrum can induce a high-frequency-dominant dynamics.
In Section 4 we ﬁrstcompare with continuous GNN models and we then discretize the equations and provide a ‘recipe’ for 60 making standard GNN architectures ﬁt a gradient ﬂow framework.
We adapt the spectral analysis todiscrete-time showing that gradient ﬂow convolutional models can generate a dynamics dominated by 62 the high frequencies via the negative eigenvalues of W while this is impossible if we drop the residualconnection.
In Section 5 we corroborate our theoretical analysis on the role of the spectrum of W 64 via ablation studies on graphs with varying homophily.
Experiments on real world datasets show acompetitive performance of our model despite its simplicity and reduced number of parameters.
Gradient-ﬂow formalism
Notations adopted throughout the paper.
Let G =
( V , E ) be an undirected graph with n nodes.
We denote by F 2 R n ⇥ d the matrix of d -dimensional node features, by f i 2 R d
its i -th row 69 (transposed), by f r 2 R n its r -th column, and by vec( F ) 2 R nd the vectorization of F obtained 70 by stacking its columns.
Given a symmetric matrix B , we let  B + ,  B  denote its most positive and 71 negative eigenvalues, respectively, and ⇢ B be its spectral radius .
If B ⌫ 0 , then gap( B ) denotes thepositive smallest eigenvalue of B . ˙
f ( t ) denotes the temporal derivative, ⌦ is the Kronecker product 73 and ‘a.e.’ means almost every w.r.t.
Lebesgue measure and usually refers to data in the complementof some lower dimensional subspace in R n ⇥ d .
Proofs and additional results appear in the Appendix.
Starting point: a geometric parallelism.
To motivate a gradient-ﬂow approach for GNNs, we start 76 from the continuous case (see Appendix A.1 for details).
Consider a smooth map f : R n !
( R d , h ) 77 with h a constant metric represented by H ⌫ 0 .
The Dirichlet energy of f is deﬁned by
[Equation]
and measures the ‘smoothness’ of f .
A natural approach to ﬁnd minimizers of E - called harmonic 79 maps - was introduced in [14] and consists in studying the gradient ﬂow of
E , wherein a givenmap f (0) =
f 0 is evolved according to ˙ f ( t ) =
r
f E ( f ( t )) .
This type of evolution equations have 81 historically been the core of variational and PDE-based image processing ; in particular, gradientﬂows of the Dirichlet energy were shown [23] to recover the Perona-Malik nonlinear diffusion [29].
Motivation: GNNs for node-classiﬁcation.
We wish to extend the gradient ﬂow formalism to node 84 classiﬁcation on graphs.
Assume we have a graph G , node-features F 0 and labels { y i } on V tr ⇢ V , 85 and that we want to predict the labels on V test ⇢ V .
A GNN typically evolves the features via some parametric rule, GNN ✓ ( G , F 0 ) , and uses a decoding map for the prediction y
= DE (GNN ✓ ( G , F 0 )) .
In graph convolutional models [13, 24], GNN ✓ consists of two operations: applying a shared lineartransformation to the features ( ‘channel mixing’ ) and propagating them along the edges of the graph 89 ( ‘diffusion’ ).
Our goal consists in studying when GNN ✓ is the gradient ﬂow of some parametric classof energies E ✓ : R n ⇥ d !
R , which generalize the Dirichlet energy.
This means that the parameters 91 can be interpreted as ‘ﬁnding the right notion of smoothness’ for our task.
We evolve the features by˙ F ( t ) =
r
F E ✓ ( F ( t )) with prediction y = DE ( F ( T ))
for some optimal time T .
Why a gradient ﬂow?
Since ˙ E ✓ ( F ( t )) =
||
r F E ✓ ( F ( t ))
|| 2 , the energy dissipates along the gradient 94 ﬂow.
Accordingly, this framework allows to explain the GNN dynamics as ﬂowing the node featuresin the direction of steepest descent of E ✓ .
Indeed, we ﬁnd that parametrizing an energy leads to 96 equations governed by attractive and repulsive forces that can be controlled via the spectrum ofsymmetric ‘channel-mixing’ matrices.
This shows that by learning to distribute more mass over the 98 negative (positive) eigenvalues of the channel-mixing, gradient ﬂow models can generate dynamicsdominated by the higher (respectively, lower) graph frequencies and hence tackle different homophily 100 scenarios.
The gradient ﬂow framework also leads to sharing of the weights across layers (since weparametrize the energy rather than the evolution equations , as usually done in GNNs), allowing us to 102 reduce the number of parameters without compromising performance (see Table 1).
Analysis on graphs: preliminaries.
Given a connected graph G with self-loops, its adjacency 104 matrix A is deﬁned as a ij = 1 if ( i, j ) 2 E and zero otherwise.
We let D =
diag
( d i ) be the degreematrix and write ¯ A :
= D  1 / 2 AD  1 / 2 .
Let F 2 R n ⇥ d be the matrix representation of a signal.
Itsgraph gradient is ( r F ) ij :=
f j / p d j  f i / p d i .
We deﬁne the Laplacian as  :=
12 div r (thedivergence div is the adjoint of r ), represented by
= I  ¯
A ⌫ 0 .
We refer to the eigenvalues of 108  as frequencies : the lowest frequency is always 0 while the highest frequency is ⇢   2 [12].
As 109 for the continuum case, the gradient allows to deﬁne a (graph) Dirichlet energy as [46]
[Equation]
where the extra 12 is for convenience.
As for manifolds, E Dir measures smoothness.
If we stack thecolumns of F into vec( F ) 2 R nd , the gradient ﬂow of E Dir yields the heat equation on each channel:
[Equation]
for 1  r  d .
Similarly to [7], we rely on E Dir to assess whether a given dynamics t 7!
F ( t ) is a 113 smoothing process.
A different choice of Laplacian L = D  A with non-normalized adjacency 114 induces the analogous Dirichlet energy
E
[Equation]
F )
[Equation]
12 trace( F > LF ) .
Throughout this paper, we rely 115 on the following deﬁnitions (see Appendix A.3 for further equivalent formulations and justiﬁcations):
Deﬁnition 2.1. ˙
F ( t ) =
GNN ✓ ( F ( t ) , t ) initialized at F (0) is smoothing if E Dir ( F ( t ))  C + ' ( t ) , 117 with C a constant only depending on
E Dir ( F (0)) and ˙ ' ( t )  0 .
Over-smoothing occurs if either
E Dir ( F ( t )) !
0 or E Dir L ( F ( t )) !
for t ! 1 .
Our notion of ‘over-smoothing’ is a relaxed version of the deﬁnition in [31] – although in the linear 120 case one always ﬁnds an exponential decay of
E Dir .
We note that E Dir ( F ( t )) !
iff  f r ( t ) !
for 121 each column f r .
As in [27], this corresponds to a loss of separation power along the solution wherenodes with equal degree become indistinguishable since we converge to ker(  ) (if we replaced with L then we would not even be able to separate nodes with different degrees in the limit).
To motivate the next deﬁnition, consider ˙ F ( t )
=
¯ AF ( t ) .
Despite || F ( t )
||
being unbounded for a.e. 125 F (0) , the low-frequency components are growing the fastest and indeed F ( t ) /
|| F ( t )
|| !
F 1 s.t.
f r 1 = 0 for 1  r  d .
We formalize this scenario – including the opposite case of high-frequency 127 components being dominant – by studying
E Dir ( F ( t ) / || F ( t ) || ) , i.e. the Rayleigh quotient of I d ⌦  .
Deﬁnition 2.2. ˙
F ( t ) =
GNN ✓ ( F ( t ) , t ) initialized at F (0) is Low/High - Frequency-Dominant(L/HFD) if E Dir ( F ( t ) / || F ( t ) || ) !
(respectively, E Dir ( F ( t ) / || F ( t ) || ) !
⇢  / 2 ) for t ! 1 .
We report a consequence of Deﬁnition 2.2 and refer to Appendix A.3 for additional details
Lemma 2.3. is
( HFD iff
If a graph is homophilic , adjacent nodes are likely to share the same label and we expect a smoothing 135 or LFD dynamics enhancing the low-frequency components to be successful at node classiﬁcationtasks [40, 25].
In the opposite case of heterophily , the high-frequency components might contain more 137 relevant information for separating classes [4, 5] – the prototypical example being the eigenvector of associated with largest frequency ⇢  separating a regular bipartite graph.
In other words, the class 139 of heterophilic graphs contain instances where signals should be sharpened by increasing
E Dir rather 140 than smoothed out.
Accordingly, an ideal framework for learning on graphs must accommodate bothof these opposite scenarios by being able to induce either an LFD or a HFD dynamics.
Parametric Dirichlet energy: channel-mixing as metric in feature space.
In eq.
(1) a constant 143 nontrivial metric h in R d leads to the mixing of the feature channels.
We adapt this idea by consideringa symmetric positive semi-deﬁnite H = W > W with W 2 R d ⇥ d and using it to generalize E Dir as
[Equation]
We note the analogy with eq.
(1), where the sum over the nodes replaces the integration over the 146 domain and the j -th derivative at some point i is replaced by the gradient along the edge ( i, j ) 2 E .
We generally treat W as learnable weights and study the gradient ﬂow of E Dir W :
[Equation]
We see that eq.
(5) generalizes eq.
Below ‘smoothing’ is intended as in Deﬁnition 2.1.
Proposition 2.4.
Let P ker W be the projection onto ker( W > W ) .
Equation (5) is smoothing since
[Equation]
In fact F ( t ) !
F 1 s.t. 9  1 2 R d : for each i 2 V we have ( f 1 ) i
= p d i  1 + P ker W f i (0) .
Proposition 2.4 implies that no weight matrix W in eq.
(5) can separate the limit embeddings F ( 1 ) 152 of nodes with same degree and input features .
If W has a trivial kernel, then nodes with same degreesconverge to the same representation and over-smoothing occurs as per Deﬁnition 2.1.
Differently 154 from [26, 27, 7], over-smoothing occurs independently of the spectral radius of the ‘channel-mixing’if its eigenvalues are positive – even for equations which lead to residual GNNs when discretized 156 [10].
According to Proposition 2.4, we do not expect eq.
(5) to succeed on heterophilic graphs wheresmoothing processes are generally harmful – this is conﬁrmed in Figure 2 (see prod -curve).
To 158 remedy this problem, we generalize eq.
(5) to a gradient ﬂow that can be HFD as per Deﬁnition 2.2.
A general parametric energy for pairwise interactions
We extend the Dirichlet energy associated with H = W > W ⌫ 0 to an energy accounting for mutual 161 – possibly repulsive – interactions in feature space R d .
We ﬁrst rewrite the energy E Dir W in eq.
(4) as
[Equation]
[Equation] Replacing the occurrences of W > W with symmetric matrices ⌦ , W 2 R d ⇥ d leads to
[Equation]
with associated gradient ﬂow of the form (see Appendix B)
[Equation]
Note that eq.
(8) is gradient ﬂow of some energy F 7!
E tot ( F ) iff both ⌦ and W are symmetric .
A multi-particle system point of view: attraction vs repulsion.
Consider the d -dimensional 166 node-features as particles in R d with energy
E tot .
While the term E ext ⌦ is independent of the graphtopology and represents an external ﬁeld in the feature space, the second term E pair W constitutes a 168 potential energy, with W a bilinear form determining the pairwise interactions of adjacent node representations.
Given a symmetric W , we write W = ⇥ > + ⇥ +  ⇥ > ⇥  , by decomposing thespectrum of W in positive and negative values.
We can rewrite E tot =
E ext ⌦  W
[Equation]
i
[Equation]
[Equation]
The gradient ﬂow of E tot minimizes E Dir ⇥ + and maximizes E Dir ⇥  .
The matrix W encodes repulsivepairwise interactions via its negative-deﬁnite component ⇥  which lead to terms || ⇥
( r F ) ij || 173 increasing along the solution.
The latter affords a ‘sharpening’ effect desirable on heterophilic graphswhere we need to disentangle adjacent node representations and hence ‘magnify’ the edge-gradient.
Spectral analysis of the channel-mixing.
We will now show that eq.
(8) can lead to a HFDdynamics.
To this end, we assume that ⌦ =
so that eq.
(8) becomes ˙ F ( t ) =
¯ AF ( t ) W .
Accordingto eq.
(9) the negative eigenvalues of W lead to repulsion.
We show that the latter can induce HFD 178 dynamics as per Deﬁnition 2.2.
We let P ⇢  W be the orthogonal projection into the eigenspace of
W ⌦ ¯ A associated with the eigenvalue ⇢  := |  W  | ( ⇢   1) .
Moreover, we deﬁne [Equation]
Proposition 3.1.
If ⇢  >  W + , then ˙ F ( t ) = ¯ AF ( t ) W is HFD for a.e. F (0) : we have [Equation] [Figure]
[Equation]
and F ( t ) /
|| F ( t ) || converges to F 1 2 R n ⇥ d
such that  f r 1 = ⇢  f r 1 , for 1  r  d .
Proposition 3.1 shows that if enough mass of the spectrum of the ‘channel-mixing’ is distributed over 183 the negative eigenvalues, then the evolution is dominated by the graph high frequencies .
This analysisis made possible in our gradient ﬂow framework where W must be symmetric .
The HFD dynamics 185 induced by negative eigenvalues of W is conﬁrmed in Figure 2 ( neg-prod -curve in the bottom chart).
A more general energy.
Equations with a source term may have better expressive power [41, 9, 36].
In our framework this means adding an extra energy term of the form E source˜ W
[Equation]
to eq.
(7) with some learnable  and ˜ W .
This leads to the following gradient ﬂow:
[Equation]
We also observe that one could replace the ﬁxed matrix ¯ A with a more general symmetric graph vector 190 ﬁeld
A satisfying A ij = 0 if ( i, j ) / 2 E . This in particular includes the case where A =
A ( F (0)) is 191 learned based on the initial encoding via an attention mechanism [38, 39].
In this case, the pairwiseenergy generalizes to
E
[Equation]
Since in the experiments we have not observed improvements from learning A and this option does 194 make the model slower, we stick to the special choice
A = ¯ A .
We also note that when ⌦ = W , theneq.
(8) becomes ˙ F ( t ) =
F ( t ) W .
We perform a spectral analysis of this case in Appendix B.2.
Non-linear activations.
The inner products in the formulations of E ext ⌦ , E pair W can be combined 197 with non-linear activations  of the form (the gradient ﬂow derivation is reported in Appendix B.3)
[Equation]
While non-linear activations offer greater expressive power, the analysis presented thus far and the 199 following comparisons with existing GNN models are investigated in the linear case for a few reasons.
First, in the spirit of [40, 27, 9], by dropping non-linear maps we can perform spectral analysis in 201 closed form.
Second, in all our experiments we have seen no gain in performance when includingnon-linear activations.
Third, we can always ‘push the non-linear maps’ in either the encoding block 203 or the decoding one without affecting the linear gradient ﬂow as discussed in Section 4.
Comparison with GNNs
In this Section, we study standard GNN models from the perspective of our gradient ﬂow framework.
4.1 Continuous case
Continuous GNN models replace layers with continuous time.
In contrast with Proposition 3.1, 208 we show that three main linearized continuous GNN models are either smoothing or LFD asper Deﬁnition 2.2.
The linearized PDE-GCN D model [15] corresponds to choosing  = 0 and 210 ⌦ =
W = K ( t ) >
K ( t ) in eq.
(10), for some time-dependent family t
[Equation] [Equation]
The CGNN model [41] can be derived from eq.
(10) by setting ⌦
[Equation] [Equation]
Finally, in linearized GRAND [8] a row-stochastic matrix A ( F (0)) is learned from the encoding via 213 an attention mechanism and we have
[Equation]
We note that if A is not symmetric, then GRAND is not a gradient ﬂow.
Proposition 4.1.
PDE  GCN D , CGNN and GRAND satisfy the following:
(i) PDE  GCN D is a smoothing model: ˙ E Dir ( F PDE  GCN D ( t ))  0 .(ii)
For a.e. F (0) it holds: CGNN is never HFD and if we remove the source term, thenE Dir ( F CGNN ( t ) /
|| F CGNN ( t ) || )  e  gap(  ) t . (iii)
If G is connected, F GRAND ( t ) !
µ as t ! 1 , with µ r = mean( f r (0)) , 1  r  d .
By (ii) the source-free CGNN-evolution is LFD independent of ˜ ⌦ .
Moreover, by (iii), over-smoothingoccurs for GRAND as per Deﬁnition 2.1.
On the other hand, Proposition 3.1 shows that the negative 222 eigenvalues of W can make the source-free gradient ﬂow in eq.
(8) HFD .
Experiments in Section 5conﬁrm that the gradient ﬂow model outperforms CGNN and GRAND on heterophilic graphs.
4.2 Discrete case
We now describe a discrete version of our gradient ﬂow model and compare it to ‘discrete’ GNNs 226 where discrete time steps correspond to different layers.
In the spirit of [10], we use explicit Eulerscheme with step size ⌧  1 to solve eq.
(10) and set ˜ W
= I .
In the gradient ﬂow framework we 228 parametrize the energy rather than the actual equations, which leads to symmetric channel-mixingmatrices ⌦ , W 2 R d ⇥ d that are shared across the layers .
Since the matrices are square, an encoding 230 block
EN : R n ⇥ p !
R n ⇥ d is used to process input features F 0 2 R n ⇥ p and generally reduce the 231 hidden dimension from p to d .
Moreover, the iterations inherently lead to a residual architecturebecause of the explicit Euler discretization:
[Equation]
with prediction y = DE ( F ( T )) produced by a decoder DE : R n ⇥ d !
R n ⇥ k , where k is the 234 number of label classes and T integration time of the form T = m ⌧ , so that m 2 N represents the 235 number of layers .
Although eq.
(12) is linear, we can include non-linear activations in EN , DE .
Until this point, we have considered equations that minimize a multi-particle energy.
We now studywhen GNNs minimize an energy E tot = E ext ⌦ + E pair A , W + E source˜ W .
Are discrete GNNs gradient ﬂows?
Given a (learned) symmetric graph vector ﬁeld A 2 R n ⇥ n 239 satisfying
[Equation] i, j ) / 2 E , consider a family of linear GNNs with shared weights of the form
[Equation]
Symmetry is the key requirement to interpret GNNs in eq.
(13) in a gradient ﬂow framework.
Lemma 4.2.
Equation (13) is the unit step size discrete gradient ﬂow of E
ext
I  ⌦ + E pair A , W  E source˜ W ,with E pair A , W deﬁned in eq.
(11) , iff ⌦ and W are symmetric.
Lemma 4.2 provides a recipe for making standard architectures into a gradient ﬂow, with symmetry 244 being the key requirement.
When eq.
(13) is a gradient ﬂow, the underlying GNN dynamics becomesexplainable in terms of minimizing a multi-particle energy by learning attractive and repulsive 246 directions in feature space as discussed in Section 3.
In Appendix C.2, we show how Lemma 4.2covers linear versions of GCN [24, 40], GAT [39], GraphSAGE [20] and GCNII [9] to name a few.
Over-smoothing analysis in discrete setting.
By Proposition 3.1 we know that the continuous 249 version of eq.
(12) can be HFD thanks to the negative eigenvalues of W .
The next result represents adiscrete counterpart of Proposition 3.1 and shows that residual, symmetrized graph convolutional 251 models can be HFD .
Below P ⇢  W is the projection into the eigenspace associated with the eigenvalue⇢  :=
|  W  | ( ⇢   1) and we report the explicit value of  HFD in eq.
(17) in Appendix C.3.
We let:
[Equation]
Theorem 4.3.
Given F ( t + ⌧ ) =
F ( t ) + ⌧ ¯ AF ( t ) W , with W symmetric, if eq.
(14) holds then
[Equation]
hence the dynamics is HFD for a.e. F (0) and in fact F ( m ⌧ ) / [Equation]
F [Equation]
m ⌧ [Equation]
[- f r 1 . 
-][Equation] s.
[Equation] [Equation] 255 Conversely, if G is not bipartite, then for a.e. F (0) the system F ( t + ⌧ ) =
⌧ ¯ AF ( t ) W , with Wsymmetric, is LFD independent of the spectrum of W .
Theorem 4.3 shows that linear discrete gradient ﬂows can be HFD due to the negative eigenvalues of 258 W .
This differs from statements that standard GCNs act as low-pass ﬁlters and thus over-smoothin the limit.
Indeed, in these cases the spectrum of W is generally ignored [40, 9] or required to be 260 sufﬁciently small in terms of singular value decomposition [26, 27, 7] when no residual connectionis present .
On the other hand, Theorem 4.3 emphasizes that the spectrum of W plays a key role to 262 enhance the high frequencies when enough mass is distributed over the negative eigenvalues providedthat a residual connection exists – this is conﬁrmed by the neg-prod -curve in Figure 2.
The residual connection from a spectral perspective.
Given a sufﬁciently small step-size so 265 that the right hand side of inequality 14 is satisﬁed, F ( t + ⌧ ) =
F ( t ) + ⌧ ¯ AF ( t ) W is HFD for a.e.
F (0) if |  W  | ( ⇢   1) >  W + , i.e. ‘there is more mass’ in the negative spectrum of W than in the 267 positive one.
This means that differently from [26, 27, 7], there is no requirement on the minimalmagnitude of the spectral radius of W coming from the graph topology as long as  W + is small 269 enough.
Conversely, without a residual term, the dynamics is LFD for a.e. F (0) independently of thesign and magnitude of the eigenvalues of W .
This is also conﬁrmed by the GCN-curve in Figure 2.
Gradient ﬂow as spectral GNNs.
We ﬁnally discuss eq. (12) from the perspective of spectral 272 GNNs as in [2]. Let us assume that  = 0 , ⌦ = 0 . We can write eq.
(12) as (see Appendix C.3) [Equation] with  = U
⇤ U > the eigendecomposition of the graph Laplacian.
Namely, if we let {  W r } be the 274 spectrum of W with associated orthonormal basis of eigenvectors given by {  W r } , and we introduce 275 z r ( t ) : V ! R deﬁned by z ri ( t ) = h f i ( t ) ,  W r
i , then we can rephrase eq.
(15) as the systemz
[Equation] Accordingly, for each projection into the r -th eigenvector of W , we have a spectral function in the 277 graph frequency domain given by   7! 1 + ⌧ W r (1    ) . If  W r > 0 we have a low-pass ﬁlter 278 while if  W r < 0 we have a high-pass ﬁlter. Moreover, we see that along the eigenvectors of W , if 279  W r < 0 then the dynamics is equivalent to ﬂipping the sign of the edge weights, which offers a direct 280 comparison with methods proposed in [4, 42] where some ‘attentive’ mechanism is proposed to learnnegative edge weights based on feature information.
The very same procedure of changing the sign 282 of the edge weights – which is equivalent to reversing the time orientation for the evolution of z r –can indeed be accomplished via the repulsive forces generated by the negative spectrum of W .
Experiments
In this section we evaluate the gradient ﬂow framework ( GRAFF ).
We corroborate the spectral 286 analysis using synthetic data with controllable homophily.
We conﬁrm that having negative (positive)eigenvalues of the channel-mixing W are essential in heterophilic (homophilic) scenarios where the 288 gradient ﬂow should align with HFD ( LFD ) respectively.
We show that the gradient ﬂow in eq.
(12)– a linear, residual, symmetric graph convolutional model – achieves competitive performance on real 290 world datasets.
Methodology.
We crystallize GRAFF in the model presented in eq.
(12) with EN , DE im- 292 plemented as single linear layers or MLPs, and we set ⌦ to be diagonal.
For the real-worldexperiments we consider diagonally-dominant (DD) , diagonal (D) and time-dependent choicesfor the structure of W that offer explicit control over its spectrum.
In the (DD) -case, we considera W 0 2 R d ⇥ d symmetric with zero diagonal and w 2 R d deﬁned by w ↵ =
q ↵
P  | W 0 ↵ | + r ↵ ,and set W = diag( w ) + W 0 .
Due to the Gershgorin Theorem the eigenvalues of W belong to 297 [ w ↵
P  | W 0 ↵ | , w ↵ +
P  | W 0 ↵ | ] , so the model ‘can’ easily re-distribute mass in the spectrum of
W via q ↵ , r ↵ .
This generalizes the decomposition of W in [9] providing a justiﬁcation in terms of its 299 spectrum and turns out to be more efﬁcient w.r.t.
the hidden dimension d as shown in Figure 4 in the Appendix.
For (D) we take W to be diagonal, with entries sampled U [  1 , 1] and ﬁxed – i.e., we donot train over W – and only learn EN , DE .
We also include a time-dependent model where W t 302 varies across layers.
To investigate the role of the spectrum of W on synthetic graphs, we constructthree additional variants: W = W 0 + W 0> , W = ± W 0> W 0 named sum , prod and neg-prod 304 respectively where prod ( neg-prod ) variants have only non-negative (non-positive) eigenvalues.
Complexity and number of parameters.
If we treat the number of layers as a constant, the discrete 306 gradient ﬂow scales as
O ( | V | pd + | E | d 2 ) , where p and d are input feature and hidden dimension 307 respectively, with p  d usually.
Note that GCN has complexity O ( | E | pd ) and in fact our model is 308 faster than GCN as conﬁrmed in Figure 5 in Appendix D. Since EN , DE are single linear layers(MLPs), we can bound the number of parameters by pd + d 2 + 3 d + dk , with k the number of labelclasses, in the (DD) -variant while in the (D) -variant we have pd + 3 d + dk .
Further ablation studiesappear in Figure 4 in the Appendix showing that (DD) outperforms sum and GCN – especially in the 312 lower hidden dimension regime – on real-world benchmarks with varying homophily.
Synthetic experiments and ablation studies.
314 To investigate our claims in a controlled environment we use the synthetic Cora dataset of [47, Ap- 316 pendix G].
Graphs are generated for target levelsof homophily via preferential attachment – see 318 Appendix D.2 for details.
Figure 2 conﬁrms thespectral analysis and offers explainability in terms 320 of performance and smoothness of the predictions.
Each curve – except GCN – represents one ver- 322 sion of W as in ‘methodology’ and we implementeq.
(12) with  = 0 , ⌦ = 0 .
Figure 2 (top) reportsthe test accuracy vs true label homophily.
Neg-prod 325 is better than prod on low-homophily and viceversaon high-homophily.
This conﬁrms Proposition 3.1 327 where we have shown that the gradient ﬂow canlead to a HFD dynamics – that are generally de- 329 sirable with low-homophily – through the negativeeigenvalues of W .
Conversely, the prod conﬁgura- 331 tion (where we have an attraction-only dynamics)struggles in low-homophily scenarios even though 333 a residual connection is present .
Both prod and neg-prod are ‘extreme’ choices and serve the purposeof highlighting that by turning off one side of the spectrum this could be the more damaging depend- 335 ing on the underlying homophily.
In general though ‘neutral’ variants like sum and (DD) are indeedmore ﬂexible and better performing.
In fact, (DD) outperforms GCN especially in low-homophily 337 scenarios, conﬁrming Theorem 4.3 where we have shown that without a residual connection convolutional models are LFD – and hence more sensitive to underlying homophily – irrespectively of the 339 spectrum of W .
This is further conﬁrmed by additional ablation studies in Figure 3.
In Figure 2 (bottom) we compute the homophily of the prediction (cross) for a given method and we 341 compare with the homophily (circle) of the prediction read from the encoding (i.e. graph-agnostic ).
The homophily here is a proxy to assess whether the evolution is smoothing , the goal being explaining 343 the smoothness of the prediction via the spectrum of W as per our theoretical analysis.
For neg-prodthe homophily after the evolution is lower than that of the encoding, supporting the analysis that 345 negative eigenvalues of W enhance high-frequencies.
The opposite behaviour occurs in the case ofprod and explains that in the low-homophily regime prod is under-performant due to the prediction
[Table]
being smoother than the true homophily.
(DD) and sum variants adapt better to the true homophily.
[Table]
We note how the encoding compensates when the dynamics can only either attract or repulse (i.e. the
[Table]
spectrum of W has a sign) by decreasing or increasing the initial homophily
[Table]
[Table]
Real world experiments.
We test GRAFF against a range of datasets with varying homophily 351 [34, 30, 28] (see Appendix D.3 for additional details).
We use results provided in [42, Table 1],which includes standard baselines as GCN [24], GraphSAGE [20], GAT [39], PairNorm [45] and 353 recent models tailored towards the heterophilic setting (GGCN [42], Geom-GCN [28], H2GCN[47] and GPRGNN [11]).
For Sheaf [5], a recent top-performer on heterophilic datasets, we took 355 the best performing variant (out of six provided) for each dataset.
We also include continuousbaselines CGNN [41] and GRAND [8] to provide empirical evidence for Proposition 4.1.
Splits 357 taken from [28] are used in all the comparisons.
The GRAFF model discussed in ‘methodology’is a very simple architecture with shared parameters across layers and run-time smaller than GCN 359 and more recent models like GGCN designed for heterophilic graphs (see Figure 5 in the Appendix).
Nevertheless, it achieves competitive results on all datasets, performing on par or better than more 361 complex recent models.
Moreover, comparison with the ‘time-dependent’ (DD) variant conﬁrmsthat by sharing weights across layers we do not lose performance.
We note that on heterophilic 363 graphs short integration time is usually needed due to the topology being harmful and the negativeeigenvalues of W leading to exponential behaviour (see Appendix D).
Conclusions
In this work, we developed a framework for explainable GNNs where the evolution can be interpreted 367 as minimizing a multi-particle learnable energy.
This translates into studying the interaction betweenthe spectrum of the graph and the spectrum of the ‘channel-mixing’ leading to a better understanding 369 of when and why the induced dynamics is low (high) frequency dominated.
From a theoreticalperspective, we reﬁned existing asymptotic analysis of GNNs to account for the role of the spectrum 371 of the channel-mixing as well.
From a practical perspective, our framework allows for ‘educated’choices resulting in a simple, more explainable convolutional model that achieves competitive 373 performance on homophilic and heterophilic benchmarks while being faster than GCN.
Our resultsrefute the folklore of graph convolutional models being too simple for complex benchmarks.
Limitations and future works.
We limited our attention to a constant bilinear form W , which 376 might be excessively rigid.
It is possible to derive non-constant alternatives that are aware of thefeatures or the position in the graph.
The main challenge amounts to matching the requirement for 378 local ‘heterogeneity’ with efﬁciency: we reserve this question for future work.
Our analysis is also aﬁrst step into studying the interaction of the graph and ‘channel-mixing’ spectra; we did not explore 380 other dynamics that are neither LFD nor HFD as per our deﬁnitions.
The energy formulation pointsto new models more ‘physics’ inspired; this will be explored in future work.
Societal impact.
Our work sheds light on the actual dynamics of GNNs and could hence improve 383 their explainability, which is crucial for assessing their impact on large-scale applications.
We alsoshow that instances of our framework achieve competitive performance on heterophilic data despite 385 being faster than GCN, providing evidence for efﬁcient methods with reduced footprint.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on 504 how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing 506 the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the 512 Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] , in Section 6.(c) Did you discuss any potential negative societal impacts of your work?
[Yes] in theSocietal impact paragraph in Section 6.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes] in Appendix A,Appendix B and Appendix C.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Code andREADME in SM, dataloaders in code(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Splits and hyperparameters provided in code zip [Table] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Standard deviations are stated in results table(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] in appendix D4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes] datasets and standardlibraries cited in appendix D [Table] (b) Did you mention the license of the assets?
[Yes] industry standard libraries andbenchmark datasets were used in accordance with licences [Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]code provided in SM zip(d)
Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[Yes] no personal data is contained within benchmarking datasets [Table]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table] [Table]