ACMP: Allen-Cahn Message Passing with Attractive and Repulsive Forces for Graph Neural Networks
Abstract
Neural message passing is a basic feature extraction unit for graph-structured data considering neighboring node features in network propagation from one layer to the next.
We model such process by an interacting particle system with attractive and repulsive forces and the Allen-Cahn force arising in the modeling of phase transition.
The dynamics of the system is a reaction-diffusion process which can separate particles without blowing up.
This induces an Allen-Cahn message passing (ACMP) for graph neural networks where the numerical iteration for the particle system solution constitutes the message passing propagation.
ACMP which has a simple implementation with a neural ODE solver can propel the network depth up to one hundred of layers with theoretically proven strictly positive lower bound of the Dirichlet energy.
It thus provides a deep model of GNNs circumventing the common GNN problem of oversmoothing.
GNNs with ACMP achieve state of the art performance for real-world node classification tasks on both homophilic and heterophilic datasets.
Introduction
Graph neural networks (GNNs) have received a great attention in the past five years due to its powerful expressiveness for learning graph structured data, with broad applications from recommendation systems to drug and protein designs [4, 6, 11, 12, 24, 53].
Neural message passing [26] serves as a fundamental feature extraction unit for graph-structured data that aggregates the features of neighbors in network propagation.
We develop a GNN message passing, called the Allen-Cahn message passing (ACMP) , using interacting particle dynamics, where nodes are particles and edges representing the interactions of particles.
The system is driven by both attractive and repulsive forces, plus the Allen-Cahn double-well potential from phase transition modeling.
This model is motivated by the behavior of the particle system of collective behaviors common in nature and human society, for example, insects forming swarms to work; birds forming flocks to immigrate; humans forming parties to express public opinions.
Various mathematical models have been proposed to model these behaviors [1, 38, 14, 47, 20].
The phase transition of the particle evolution of ACMP clusters particles into desired flocks, which guarantees the Dirichlet energy for ACMP-based GNNs has a lower bound strictly above zero–which will be proven mathematically– thus avoids oversmoothing.
The new system can be viewed as the graph version of the bi-cluster Cucker-Smale swarming model introduced in [23].
This naturally induces a GNN model to simulate the message passing dynamics.
There are two major components in this model.
First, while the attractive force forces all particles into one cluster, the repulsive forces allow particles to separate into two different clusters, which is essential to avoid oversmoothing.
However, repulsive forces could make the Dirichlet energy diverge. We augment the model with the Allen-Cahn [2] term (or Rayleigh friction [48]), which is crucial in preventing the Dirichlet energy in the evolution from becoming unbounded, allowing us to prove mathematically that the lower bound of the Dirichlet energy is strictly bigger than zero, hence avoiding oversmoothing.
Specifically, we will prove that under suitable conditions on the parameters, the dynamics of the ACMP particle system will time-asymptotically form 2 d different clusters and the Dirichlet energy has a strictly positive lower bound.
One of our main contribution is introducing a repulsive force into message passing framework from the many-particle system perspective.
Most existing message passing neural networks are driven by attractive forces associated with the Dirichlet energy, and the corresponding graph neural network suffers from oversmoothing, and fails in heterophilic dataset prediction.
The repulsion makes particles separate into two different clusters, hence provides a simple and neat solution for prediction tasks on heterophilic datasets. However, the presence of repulsive force would lead to feature blow-up when the networks go deep. Our second main idea is to include the Allen-Cahn potential which can circumvent the Dirichlet energy and node features to go to infinity, as can be proven mathematically. Overall, the benefit of the Allen-Cahn message passing with repulsion is manifold.
1) It circumvents oversmoothing issue, namely the Dirichlet energy is bounded from below.
The network is stable in the sense that features and Dirichlet energy are bounded from above.
Feature smoothness (energy decreasing) and the balance between nodes features and edge features can be adjusted easily by network parameters that control the attraction, repulsion and phase transition.
The model can then reach an acceptable trade-off on self-features and neighbor effect, as shown in Figure [-
-]1.
Our model can thus handle node classification tasks for both homophilic and heterophilic datasets by using only one-hop neighbour information.
The proposed model can be easily implemented by neural ODE solvers for the system with attractive and repulsive forces via Allen-Cahn potential.
In theory, we prove that Dirichlet energy of GNNs with ACMP has a lower bound above zero (limiting oversmoothing), as well as an upper bound (circumventing blow-up) under specific conditions.
This agrees with the experimental results (Section 6).
We also prove that ACMP is a process for the features to generate clusters thanks to the double-well potential, which provides an interpretable theory for node classification.
Background
Message Passing in Graph Neural Networks Graph neural networks are a kind of deep neural networks which take graph data as input.
Neural Message Passing (MP) [26, 7] is a most widely used propagator for node feature update in GNNs, which takes the following form: for the undirected graph G =
( V , E ) is with sets of nodes V and edges E , with x ( k − 1) i ∈
R d denoting features of node i in layer ( k − 1) and a j,i ∈ R D edge features from node j to node i,
[Equation]
where □ denotes a differentiable, (node) permutation invariant function, e.g., sum, mean or max, and γ and ϕ denote differentiable functions such as MLPs (MultiLayer Perceptrons), and N i is the set of one-hop neighbors of node i.
The message passing updates the feature of each node by aggregating the self-feature with neighbors’ features.
Many GNN feature extraction modules such as GCN [30], GAT [52] and GIN [55] can be written as message passing.
For example, the MP of GCNs reads, with learnable parameter matrix Θ , x ′ i =
Θ ⊤ (cid:80) j ∈N i ∪{ i } a j,i √ ˆ
d j ˆ d i x j , where ˆ d i = 1 + (cid:80) j ∈N ( i ) a j,i and ˆ D = diag( ˆ d 1 , . . , ˆ d N ) is the degree matrix for A + I .
Graph attention network (GAT) uses attention coefficients α i,j as similarity information between nodes in the MP update x ′ i
= α i,i Θx i + (cid:80) j ∈N
i α
i,j Θx j , with
[Equation]
The MP framework was also developed as PDE solvers in [10] by embedding differential equations as a parameter into message passing like [9].
This paper regards particle system evolution (ODE) as message passing propagation, and the appropriate design of the particle system offers desired properties for the resulting GNN.
Graph neural diffusion Neural diffusion equations on graphs (GRAND) are proposed by [15], which provides a unified mathematical framework for some message passings:
[Equation]
where G = diag( a ( x i ( t ) , x j ( t ) , t )) where a is a function reflecting similarity between nodes i and j, and x i is the scale-valued feature for node i, and x = ⊕ x i .
Motivations 3.1 Attractive and repulsive forces
The equation (2) itself can be interpreted in a formulation different from diffusion.
In this paper, we study the neural equations of interacting particle system , which has a similar structure to (2).
We rewrite (2) into a component-wise version and obtain a particle system
[Equation]
In the formulation of particle systems, one can easily discover the evolution trend of the features.
If a ( x i , x j ) > 0 , the direction of x i ’s velocity is towards x j , which means that x i is attracted by x j .
In the contrast, if a ( x i , x j ) < 0 , x i has a trend to move away from x j .
Hence, a ( x i , x j ) serves as the attractiveness or repulsiveness of the force between x i and x j .
In the diffusion model above, all a ( x i , x j ) ’s are positive, therefore all the node features in one connected component attract each other.
If the weight matrix ( a ( x i , x j )) N × N is right-stochastic, one can prove that the convex hull of the features will not dilate in time (see [39, 15]).
Such feature aggregation means that message propagates along the edges of the graph and some potential consensus forms in the process.
However, the message propagation does not limit to consensus (corresponding to diffusion).
Information interaction can derive polarization of final judgement when negative message matters in some problems rather than positive message.
For instance, in a node classification task on a bipartite, the neighbour message is negative since connected nodes belong to different classes.
In the formulation of particle systems, the mechanism of positive and negative messages can be modelled by adding bias β i,j into (
[Equation]
The coefficient term a ( x i , x j ) − β i,j corresponds to the interactive force.
By adjusting β i,j , both in the system attractive and repulsive forces co-exist.
If a ( x i , x j ) − β i,j > 0 , x i is attracted by x j .
While if a ( x i , x j ) − β i,j < 0 , x i is repelled by x j .
If the coefficient equates zero, there is no interaction between x i and x j .
Then, the dynamics is enabled to adapt both positive and negative message passing.
In this way, the neural message passing can handle either homophilic or heterophilic datasets (see Section 6 for detailed discussion).
3.2 Pseudo-Ginzburg-Landau energy
However, adding the term of repulsive force may cause the particles being pushed away to infinite, thus the Dirichlet energy becomes unbounded.
To avoid this problem, we add a forcing term δx i (1 − x 2 i ) , which we call an Allen-Cahn term .
Here, the coefficient α > 0 is multiplied just for technical convenience.
[Equation]
Gradient Flow
The variational principle governing many PDE models states that the equilibrium state is actually the minimizer of one specific energy.
The equilibrium state carries meaningful information and can therefore be used as embedded features in the context of machine learning.
We first introduce the Dirichlet energy and show that (3) can be characterized by looking into the corresponding Euler-Lagrange equation of the Dirichlet energy.
Let adjacent matrix
A represent the undirected connectivity between nodes x i and x j , with a i,j = 1 for ( i, j ) ∈ E and a i,j = 0 for ( i, j ) ̸∈ E .
The Dirichlet energy E in terms of G =
( V , E ) and node features x ∈ R N × d takes the form
[Equation]
By calculus of variation, we can formulate the corresponding particle equation
[Equation]
On the RHS of (7), the summation takes over the one-hop neighbors N i of node i, which aggregates the impact from the neighboring nodes.
Equation (7) is (5) when one takes adjacent matrix A as the weight matrix ( a ( x i , x j ))
N × N .
Particle equation with Allen-Cahn potential To avoid blowing-up of the solution, one can design an external potential to control the solutions so they are bounded.
Here, we define the pseudoGinzburg-Landau energy on graph G denoted by Φ : L 2 ( V ) → R , as a combination of the interacting energy and double-well potential W : R → R + , with
[Equation]
where parameters α, δ > 0 are used to balance the two types of energy.
From now on, we denote a ( x i , x j ) by a i,j for simplicity.
The pseudo-Ginzburg-Landau energy is not a true energy because the matrix ( a i,j − β i,j ) N × N can be non-positive definite.
If β i,j ’s all equate zero, it then becomes the Ginzburg-Landau energy defined in [8, 34].
Using this combined energy, we can obtain the
Allen-Cahn equation with repulsion on graph as ∂ x ∂t = −∇ x Φ , which is equivalent to (5).  Allen-Cahn Message Passing
We propose the Allen-Cahn Message Passing (ACMP) neural network based on equation (5), where the message is updated by the evolution of the equation via a neural ODE solver.
To our best knowledge, this is the first time to introduce a type of message passing to amplify the difference between connected nodes by repulsive force.
Network Architecture Suppose d -dimensional node-wise features represented by a matrix x in where row i represents feature of node i. Our scheme first embeds the node feature x (0) =
MLP( x in ) by a simple multi-layer perceptron (MLP), which is treated as an input for ACMP propagation A : R d → R d , by x (0) (cid:55)→ x ( T ) , where x ( T ) = x [Equation]
x
[Equation]
, where ∂ X ( t ) ∂t is estimated by ACMP defined on G based on (5).
The node features x ( T ) at the ending time are fed into an MLP based classifier.
Then, we define the Allen-Cahn message passing by
[Equation]
Here α , δ ∈ R d are learnable vectors of the same length as the node feature x i .
While we can use a more general case when each edge ( i, j ) uses different trainable β i,j , we have simplied to single hyper-parameter β ∈ R + ∪ { 0 } , which makes the network and optimization easier.
The β in our model is a crucial parameter, which can be adjusted such that the attractive and repulsive forces both present to enrich the message passing effect.
If one chooses δ = 0 , β = 0 , our model is reduced to the graph neural diffusion network (GRAND) in [15].
In experiments, we would make significant use of nontrivial δ and β.
The operations of all terms are channel-wise, involving d channels, except a ( x i ( t ) , x j ( t )) , and ⊙ represents channel-wise multiplication for d feature channels.
Figure 1 illustrates the one-step ACMP mechanism (8): Nodes with close colors attracts each other otherwise repel.
Nodes in the same block tend to attract each other and both colors are strengthened during message passing propagation.
The double-well potential prevents the features and Dirichlet energy from blowup.
In this process, node feature x ( t ) is updated to x ( t + ∆ t ) for a time increment ∆ t .
Ultimately, a bi-cluster flock is formed for node classification.
In the propagation of ACMP in (8), we need to specify how the neighbors are interacted, that is how the a ( x i ( t ) , x j ( t )) is evolved with time.
There are many kinds of methods to update the edge weights.
Two typical types of ACMP are GCN based [30] and graph attention (GAT) based [52].
ACMP-GCN : this model uses deterministic a ( x i ( t ) ,
x j ( t )) , which is given by the adacency matrix A = ( a i,j ) of the original input graph G and does not change with time. That is, the coefficients in GCNs a GCN i,j := a i,j / (cid:113) ˆ d i ˆ d j .
The message passing of (8) is reduced to
[Equation]
ACMP-GAT : we can replace a GCN i,j in (9) by the attention coefficients (1) of GAT, which with extra trainable parameters measures the similarity between two nodes by taking account of both node and structure features.
The system then drives edges to update in each iteration of message passing.
Neural ODE Solver Our method uses an ODE solver to numerically solving the equation ((8) and (9)) for ACMP.
To obtain the node features x ( T ) , we need a stable numerical integrator for solving the ODE efficiently and backpropagation of gradients.
{+Since our model is stable in terms of evolution time, most explicit and implicit numerical methods such as explicit Euler, Runge-Kutta 4th-order, midpoint, Dormand-Prince
+}
[17, 32, 41, 15] work well as long as the step size τ is small enough.
In experiments, we implement ACMP using Dormand-Prince5 method which provides a fast and stable numerical solver.
The number of runs of our message passing, or the network depth of ACMP-GNN is equal to the numerical iteration number n t set in the solver.
Computational Complexity
The computational complexity of the ACMP is O ( NEdn t ) , where n t , N, E and d are number of time steps in time interval [0 , T ] , number of nodes, number of edges and number of feature dimension, respectively.
Since our model only considers nearest (one-hop) neighbors, E is significantly smaller than that of graph rewiring [25, 3] and multi-hop [57] methods.
Channel Mixer Channel mixing can be spontaneously introduced from the perspective of diffusion coefficients though our model is previously written in the channel-wise form.
Whether channel mixing happens depends on the specific GNN driver we choose for ACMP.
When the coefficients a ( x i ( t ) , x j ( t )) in (8) that do not update with time are a scalar or vector, like in ACMP-GCN, the operations of the message passing propagator are channel-wise and channel mixing is not incorporated.
On the other hand, the ACMP-GAT with graph attention driver incorporates a learnable channel mixing when the coefficients are tensors.
The channel mixer can be introduced by generalizing the
Dirichlet energy to high dimension, for example, E ( x ) :
= 1 N (cid:80) i ∈V (cid:80) j ∈N i ( x i − x j ) T a i,j ( x i − x j ) , when a i,j ∈ R d × d are connectivity tensors.
Dirichlet Energy
The dynamics (5) can circumvent the oversmoothing issue of GNNs [42, 44, 31].
Oversmoothing phenomenon means that all node features converge to the same constant – consensus forms – as the network deepens, and equivalently, the Dirichlet energy will decay to zero exponentially.
This idea was first introduced in [13].
[31] gives an explicit form for oversmoothing.
In our model, as we will show below, the node features in each channel tend to evolve into two clusters departing from each other under certain conditions. This implies a strictly positive lower bound of the Dirichlet energy.
In addition, the system will not blow up thanks to the Allen-Cahn term.
We put all the proofs and some related supplementary results in the appendix.
Proposition
If δ > 0 , the node features x i in (5) is bounded in terms of ∥ · ∥ and energy for all t > 0 , i.e., E ( x ( t )) ≤ C, and ∥ x ∥ ≤ C, where the constant C only depends on N and λ max .
In the following propositions, we imitate the emergent behavior analysis in [23] (see Appendix for details).
For a graph G with N nodes, its vertices are said to form bi-cluster flocking if there exist two disjoint sets of vertex subsets { x (1) i } N 1 i =1 and { x (2) i } N 2 j =1 satisfying
( i
[Equation]
( ii ) ∃ C ′ , T ∗∗ > 0 such that
[Equation]
where x (1) i , x (2) i denote any component of x (1) i , x (2) i .
We now show the long-time behaviour of model (5) following the analysis of [23] for strength coupling ( α, δ ) that satisfies the following condition: there exists { β i,j } such that I := { 1 , . .
, N } can be divided into two disjoint groups I 1 , I 2 with N 1 and N 2 particles respectively:
[Equation]
where S, D are independent of time t.
The S and D in (11) are the repulsive and attractive forces.
We prove that if the repulsive force between the particles is stronger than the attractive force, that is, S > D, the system is guaranteed to have bi-cluster flocking, as shown in Proposition 2 below.
For time t ≥ 0 , suppose x (1) c ( t ) and x (2)
c
( t ) are the feature centers of the two groups of the particles { x (1) i ( t ) }
N 1 i =1 and { x (2) j ( t ) } N 2 j =1 which are partitioned as above from the whole vertex set V , given by
[Equation]
Suppose x ( s ) c ( t ) has the d -dimensional feature, and let x ( s ) c,k ( t ) , k = 1 , . . .
, d, be the k th (dimension) component of the feature x ( s ) c ( t ) , s = 1 , 2 .
Proposition 2 The system (5) has a bi-cluster flocking if for each k = 1 , . . .
, d, the initial | x (1) c,k (0) − x (2) c,k (0) | ≫ 1 , and if there exists a positive constant η such that
[Equation]
) where the δ is the weight factor for the double-well potential in the equation (5).
Proposition 3 For system (5) with bi-cluster flocking, there exists a constant C > 0 and some time
T ∗ such that ∀ t ≥ T ∗ ,
[Equation]
Thus, if the non-zero a i,j are all positive, the Dirichlet energy for ACMP is lower bounded by a positive constant.
Experiments
Dirichlet Energy We first illustrate the evolution of the Dirichlet energy of ACMP by an undirected synthetic random graph.
The synthetic graph has 100 nodes with two classes and 2D feature which is sampled from the normal distribution with the same standard deviation σ = 2 and two means µ 1 = − 0 .
5 , µ 2 = 0 .
The nodes are connected randomly with probability p = 0 .
9 if they are in the same class, otherwise nodes in different classes are connected with probability p = 0 .
We compare the performance of GNN models with four message passing propagators: GCNs [30], GAT [52], GRAND [15] and ACMP-GCN.
In Figure 3, we show the Dirichlet energy of each layer’s output in logarithm scales.
Traditional GNNs such as GCNs and GAT suffer oversmoothing as the Dirichlet energy exponentially decays to zero in the first ten layers.
GRAND relieves this problem by multiplying a small constant which can delay all nodes’ features to collapse to the same value.
For ACMP, the energy stabilizes at the level that relies upon the roots of the Allen-Cahn potential in (8) after slightly decaying in the first two layers.
Node Classification We compare the performance of ACMP with several popular GNN model architectures on various node classification benchmarks, containing both homophilic and heterophilic datasets.
Graph data is considered as homophilic [45] if similar nodes in the graph tend to connect together.
Conversely, the graph data is said heterophilic if it has a small homophily level, when most neighbors do not have the same label with source nodes.
We aim to demonstrate that ACMP is a flexible GNN model which can learn well both kinds of datasets by balancing the attractive and repulsive forces.
The GCN for examples cannot perform well for heterophilic dataset as its message passing aggregates only the neighbor (1-hop) nodes.
The neural ODE is solved by Torchdiffeq package with Dormand–Prince adaptive step size scheme.
Homophilic datasets The results of our study are presented for the most widely used citation networks: Cora [35], Citeseer [50] and Pubmed [40].
Moreover, we evaluate our model on the
Amazon co-purchasing graphs Computer and Photo [40], and CoauthorCS [51].
We compare our model with traditional GNN models: Graph Convolutional Network (GCN) [30], Graph Attention
Network (GAT) [52], Mixture Model Networks [37] and GraphSage [29].
We also compare our results with recent ODE-based GNNs, Continuous Graph Neural Networks (CGNN) [54], Graph
Neural Ordinary Differential Equations (GDE) [46] and Graph Neural Diffusion (GRAND) [15].
To address the limitations of this evaluation methodology proposed by [51], we report results for all datasets using 100 random splits with 10 random initialization’s, and show the node classification result with mean and standard deviation in Table 1.
[Table]
Heterophilic datasets We evaluate ACMP-GCN on the heterophilic graphs; Cornell, Texas and
Wisconsin from the WebKB dataset 3 .
In this case, the assumption of common neighbors does not hold.
The poor performance of GCN and GAT models shown in Table 2 indicates that many GNN models struggle in this setting.
Introducing repulsion can improve the performance of GNNs on heteroplilic datasets significantly.
ACMP-GCN scores 30% higher than the original GCN for the
Texas dataset which has the smallest homophily level among the datasets in the table.
Attractive and Repulsive interpretation As shown in Table 2 and Table 1, ACMP-GCN and
ACMP-GAT achieve better performance than GCN and GAT on both homophilic and heterophilic datasets.
The majority of a i,j − β in the homophilic are positive, which means most nodes are attracted to each other.
Conversely, most a i,j − β for the heterophilic are negative, which means that most nodes are repelled by their neighbors.
Several GNNs exploiting multi-hop information can achieve high performance in node classification [57, 33].
However, high-order neighbor information will make the adjacency matrix dense and therefore can not be extended to large graphs, due to heavier computational cost.
In our model, we take only one-hop information into account and add repulsive force ( β ≥ 0 ) to message passing, which has achieved the same or higher level of accuracy as multi-hop models in heterophilic datasets. [Table]
Performance of ACMP to β
Hyperparameter β is the key to introduce the repulsive force in GNN, meaning that when a ij − β is negative, the two nodes repel one another.
To illustrate β ’s impact on different datasets, we use GCN as a diffusion term as a ij do not change during the ODE process and all the changes are related to β.
As shown by Figure 4, ACMP performs best in Cora (orange curve) when all nodes are attracted to one another i.e., all a ij − β is positive.
As the beta increases, the performance of the model degrades.
In contrast, for the Texas dataset, when all force is attractive, ACMP achieves only 70% accuracy (blue curve).
As β increases, most a ij − β is negative, and the model’s performance gets better.
When all the force is repulsive, ACMP achieves highest accuracy on Texas datasets, which is in accordance with our claim that repulsive force is important for heterophilic datasets.
Related work
Neural differential equations The topic of neural ODEs becomes an emerging field since [21] and [17], with many follow-up works in the GNN field: [5] used continuous residual modules for graph kernels; [46] extended the framework of GNN to continuous time.
[49] applied Hamiltonian mechanics to graph networks to predict future states.
GRAND [15] approached graph deep learning as a continuous diffusion process and propagated GNNs by the graph diffusion equation.
[22] combined diffusion and wave PDEs for GNNs, and GraphCON [31] generalized this method.
The latter employed a second-order system to conquer oversmoothing of deep graph neural networks.
Allen-Cahn based variational graph models In [8, 34, 36] and references therein, authors extended Allen-Cahn related potential to graphical framework and developed a class of variational algorithms to solve the clustering, semisupervised learning and graph cutting problems.
The new ingredient of graph neural network which enables us to combine learnable attraction and repulsion separates our method from the classical variational graph models.
Conclusion
We develop a new message passing method with simple implementation.
The method is based on the Allen-Cahn particle system with repulsive force.
The proposed ACMP inherits the characteristic dynamics of the particle system and thus shows adaption for node classification tasks with high homophily difficulty.
Also, it propels networks to dozens of layers without getting oversmoothing.
A strictly positive lower bound of the Dirichlet energy is shown by theoretical and experimental results which guarantees oversmoothing A Flocking and consensus
The microscopic (agent-based particle systems) modeling of flocking and consensus has been extensively studied. [39] reviews a general class of models for self-organized dynamics and shows the relationship between heterophily and consensus.
[14] presents a series of social and dynamics under the formulation of statistical physics.
The flocking problem is to some degree similar to the general consensus problem [43] which studies the emergent behaviours for multi-agent systems. The
Cucker-Smale (in short C-S) model [19] is a famous model in this field considering a second-order system adopting to classical dynamics. [28] and [27] discuss asymptotic flocking for the C-S model with the Rayleigh friction.
[23] furthermore studies frameworks leading to bi-cluster flocking for the C-S model with the Raleigh friction and attractive-repulsive coupling.
B Model variants
More clusters We can simply replace the double well potential W by a multi-well potential to generate more equilibria.
We provide two alternatives here.
One can use a higher-order polynomial to construct additional wells.
In general, a (2 k +1)
th order polynomial can produce k +1 stable equilibria in a proper form, which gives rise to more stable clusters.
One can also use sin(( 32 + l ) πx + π 2 ) , l = 0 , · · · , k, defined on the interval [ − 1 , 1] as the multi-well potential, which has l + 2 stable equilibria.
Stronger trapping force As the consensus state (i.e., x i = x j for all
i, j ) might not be a global equilibrium of (9), particles could escape from one well of the potential of W to another well.
We can circumvent this instability by enhancing the attraction of the wells, which can be achieved by reducing the diffusion power around wells:
[Equation]
13) where ‘GNN’ in a GNN can be GCN or attn, and
z ⊙ 2 is z ⊙ z. With this modification in (13), in any channel k, if any particle x ( k ) i gets caught in one potential well, then it is not likely to escape:
Proposition 4 For (13), there exists a proper δ ′ >
such that x ( k ) i [Equation] ] , then particle x ( k ) i cannot transition into another well.
Proof.
For the β = 0 case, assume x i = − 1 + ϵ for ϵ ≤ δ ′ < 1 at a certain time t 0 , that is, x i ∈ [ − 1 , − 1 + δ ′ ) .
We want to show dx i dt (cid:12)(cid:12) t =
t 0 < 0 , which means
[Equation]
By (cid:80) j ∈N i a i,j = 1 from (25), the above inequality is equivalent to
[Equation]
Since { x j } Nj =1 are bounded (See Proposition 1.), (14) is satisfied for a sufficiently small δ ′ .
The other case x i
For the β ̸ = 0 case, we also assume x i = − 1 + ϵ for ϵ ≤ δ ′ < 1 at a certain time t 0 .
Similarly with (14), we have
[Equation]
By the boundedness of { x j } Nj =1 , a properly small δ ′ can be found.
■
C Supplementary Results and Proofs of Propositions in Section
We assume that a i,j is symmetric, and a i,j > 0 if a i,j ̸ = 0 .
This condition means that graph is undirected.
Since we deal with each channel independently, we abuse the notation to let x i denote one feature component of node x i to simplifying the notation in proofs.
C.1 The GRAND model
First, we consider the oversmoothing phenomenon if there is only the diffusion process with diffusion coefficients independent of x i , which is a specific model of graph diffusion network (GRAND) [15],
[Equation]
Proposition 5 Let D denote the degree matrix, i.e., D :=
diag( d 1 , · · · , d N ) , where d i = (cid:80) j a i,j .
Then D − A is symmetric positive semi-definite with the eigenvalues
= λ 0 ≤ λ 1 ≤ · · · ≤ λ max < ∞ .
Let λ min > 0 be the smallest positive eigenvalue, then for all t ≥ 0 , there exists a constant C >such that E ( x ( t ))
≤ C exp( − λ 2min t ) .
Proof.
Let L := D − A , we have,
[Equation]
Using eigenvalue decomposition, the solution x ( t ) writes
[Equation]
Since the Dirichlet energy can also be written as
[Equation]
Taking (16) to (17) gives
[Equation]
Therefore, E ( x ( t )) ≤ C exp( − λ 2min t ) for some constant C > 0 .
■
Proposition 6 We also consider a more general case,
[Equation]
with a ( x i , x j ) = a ( x j , x i ) ≥ a min > 0 , for any x i , x j .
Let the mass center x c = 1 N (cid:80) i ∈V x i .
From the symmetry of a ( x i , x j ) and (19), we obtain d x c / d t = 0 for any t > 0 .
Without loss of generality, we may assume [Equation] and graph G is connected, i.e., ∀ ( i,
j ) ∈ V × V , G contains a path from i to j.
Then we have, ∥ x ( t ) ∥ 2 ≤ ∥ x (0) ∥ 2 e − 2 a min λ min t and E ( x ( t )) ≤ λ max ∥ x (0) ∥ 2 e − 2 a min λ min t .
Note that the above estimates hold true for any initial condition x c (0)
=
c, since x satisfies the ODE system (19) up to a constant.
If x c (0) =
c, x i will converge to c in time.
If G is not connected, then we just need to consider each connected sub-graph separately with the assumption x c ′ (0)
= 1 N ′ (cid:80) i ∈V ′ x
i = c ′ for each sub-graph G ′
=
( V ′ , E ′ ) .
x ′ i in each sub-graph will converge to constant c ′ independently.
Proof .
We multiply x i on both sides of the equation (19) and sum over x i to obtain
[Equation]
The RHS in (23) can be written in matrix form with L := D − A , [Equation]
Since G is a connected graph
[Equation]
1 is the only eigenvector consisting of the kernel space of L
[Equation]
therefore [Equation] x T L x ≥ λ min ∥ x ∥ 2 for any x satisfying (cid:80) i ∈V x i = 0 .
Then, (23) leads to
[Equation]
This yields the decay estimates for ∥ x ∥ and E ( x ( t )) :
[Equation]
C.2 The model with Allen-Cahn term
Next, we consider the case β = 0 but with Allen-Cahn term:
[Equation]
Proposition 7 Suppose x ∗ =
( x ∗ 1 , . . .
, x ∗ N ) is a global equilibrium (or steady state solution) of (25) on R and x , then x ∗ i ∈ [ − 1 , 1] .
Proof.
Suppose x ∗ achieves the equilibrium of (25), and x ∗ k ≥ x ∗ i
∀ i. If x ∗ k > 1 , then α (cid:80) j :( k,j ) ∈E a ( x ∗ k , x ∗ j )( x ∗ j − x ∗ k ) ≤ 0 and x ∗ k (1 − x ∗ 2 k ) < 0 , which contradicts with ∂∂t x ∗ k = 0 .
■
The emergence of clusters depends on the distribution of initial features.
If all the initial features are in only one potential well, then intuitively it is impossible to produce more than one cluster in the dynamics (25).
As a simple transference of Lemma 3.2 in [27], we can prove this.
Set
[Equation]
26) where x i is still some component of node feature x i .
Assume x m , x M are both Lipschitz continuous and therefore they are almost differentiable everywhere in time t.
Proposition 8 Let { x i } be the solutions of (25), then the following holds.
(i)
If x m (0) > 0 , then x m ( t ) ≥ 0 for all t > 0 .
(ii)
If x M (0) < 0 , then x M ( t ) ≤ 0 for all t > 0 .
Proof.
The proof was essentially given by [27].
For the sake of completeness, we give a proof here. (i)
If x m ( ) > 0 , we assert there exists a time sequence { t j } ∞ j =0 satisfying t 0 = 0 < t 1 < · · · < t j <
. , x m ( t ) is differentiable in each time interval ( t j − 1 , t j ) and x mi ≥ 0 when t ∈ [0 , t 1 ] .
By induction, firstly we set x m ( t ) ≥ 0 , t ∈ [0 , t l ] .
If x m becomes negative in the time interval ( t l , t l +1 )
there exists t ∗ ∈ ( t l , t l +1 ) such that x m ( t ∗ )
=by the continuity of x m ( t ) .
One can assume x m ( t ) ≡ x i ( t ) for some node x i in some time interval subset to ( t l , t l +1 ) .
At that moment,
[Equation]
Hence, the trajectory x m becomes non-decreasing at t = t ∗ .
By induction, we derive (i). (ii) can be proved by the same argument as those for (i).
■
Now we consider the second kinetic model (13).
We can prove that if any particle x i gets caught in one potential well, then it will not escape from that well.
C.3 The Attractive-repulsive Model
We first show that the solution features of graph in Allen-Cahn model below is bounded.
For simplicity of the proof, we rewrite (9) in component form where we let a ( x i , x j ) :
= a i,j − β i,j :
[Equation]
Model (28) allows negative a ( x i , x j ) which is different from the condition in (25).
Proof of Proposition 1.
We multiply x i on both sides of the following equation and sum over x i to obtain
[Equation]
By grouping a ( x i , x j ) ( x j − x i ) x
i , then
[Equation]
Note that a ( x i , x j ) are bounded for any ( x i , x j ) .
Let the | a ( x i , x j ) | < D 1 for a constant Ddepending on hyper-parameters β i,j .
By the Cauchy-Schwarz inequality,
[Equation]
Hence, − (cid:88)
[Equation]
Also, (cid:80) i ∈V x
i ≥ c 3 ∥ x ∥ 4 for a constant c 3 depending only on N. Taking the above estimates to (30) gives
[Equation]
If ∥ x ∥ blows up for t > 0 , the ∥ x ∥ → ∞ as time t increases, and ddt ∥ x ∥ 2 > 0 for all t before the blowing-up time T end .
However, one can find a t ∗ <
T end such that ∥ x ( t ∗ ) ∥ is large enough and
[Equation]
which produces a contradiction.
Thus, ∥ x ∥ ≤ c 5 for a constant c 5 only depending on N and D 1 and
[Equation]
where λ max is the largest eigenvalue of L := D − A .
Thus, we proved the assertion in Proposition 1.
■
Recall (5) under (11) and rewrite it as  
[Equation]
For the attractive-repulsive model (31), we can refer to the the proof of its Theorem 5.1.
in [23].
We define the following notations for further proof:
[Equation]
Remark 1 (12) indicates that the repulsive force between the particles should be weaker than the attractive force( S > D ) [Equation]
To prove Proposition
[Equation]
we need the following two lemmas, which we would postpone to prove.
Lemma 1 Let { x i } be a solution to (31)
[Equation]
Then (cid:99) M 2 satisfies
[Equation]
Suppose that the system parameters satisfy
[Equation]
then there exists a positive constant M ∞ 2 such that
[Equation]
[-Remark 
-]
[Equation]
Proof of Lemma 1.
[Equation]
Similarly,
[Equation]
Sum the M 2 ( V ) and M 2 ( W ) . Note that a ij = a ji . Then
[Equation]
By the Cauchy-Schwarz inequality
[Equation]
These relations and (36) yield a Riccati-type differential inequality:
[Equation]
Let y be a solution of the following ODE:
[Equation]
[Equation] Then [Equation] the solution y ( t ) to (38) satisfies
[Equation]
Lemma 2 Let { x i } be a solution to (31) with δ > 0 .
Then (cid:99) M 2 satisfies
[Equation]
where ζ = max { N 1 , N 2 } and η is the positive constant in Proposition 2.
Proof of Lemma 2.
By computation,
[Equation]
Note that ˙ (cid:100) x
(1) i = ˙
x (1) i − ˙ x (1) c .
Take the inner product 2 (cid:100) x (1) i with the above equation and sum it over all i = 1 , . .
, N 1 , combining with (cid:80) (cid:100) x (1) i = 0 .
Then, d
[Equation]
− 2 α
N 2 (cid:88) k
=
N 1 (cid:88) i = a k,i ( x (2)
k − x (1) i ) (cid:100) x (1) i + 2 δ
N 1 (cid:88) i = (cid:100) x (1) i x (1) i (1 − ( x (1) i ) 2 ) 
 =
N  − α
N 1 (cid:88) i,k = a k,i ( (cid:100) x (1) k − (cid:100) x (1) i ) 2 − 2 α
N 1 (cid:88) i =
N 2 (cid:88) k
= a k,i ( x (2) c − x (1) c
+ (cid:100) x
(2) k − (cid:100) x (1) i ) (cid:100) x (1) i   +
[Equation]
N
[Equation]
N 1 (cid:88) i = (cid:100) x (1) i x (1) i (1 − ( x (1) i ) 2 ) .
Similarly,
[Equation]
Combine the two equations, ddt (cid:99) M 2 = 6 (cid:80) i
{+=
+}
I i , where
[Equation]
I 3 := − 2 α
N 2 (cid:88) k
=
N 1 (cid:88) i = a k,i (
[Table]
x
[Table]
2) k − (cid:100) x
[Table] [Table]
i ) (cid:100) x
[Table]
N 1 −
[Table]
N
[Table]
k
[Table]
N
[Table]
j
[Table]
a j,
[Table] [Table]
x
[Table] [Table]
−
[Table]
x
[Table] [Table]
j )
[Table]
x
[Table]
j
N≤ max (cid:26)
[Table]
N 1 ,
N(cid:27)
[Table]
a
[Table]
≤ max (cid:26)
[Table]
N [Table]
N
[Table]
−
[Table] = [Table]
max (cid:26)
[Table]
N
[Table]
N(cid:27) N
[Table]
N
[Table]
M
[Table]
= 2 αDζ (cid:99) M 2 ,
I 4 := − 2 α
N 2 (cid:88) k
=
N
[Table]
i = a k,i ( x (2) c − x (1) c ) (cid:100) x (1) i N 1 − 2 α N
[Table]
k =
[Table]
N
[Table]
j =
[Table]
a j, [Table] x
[Table] [Table]
− x
[Table] [Table] c ) [Table]
x
[Table] [Table] N [Table]
≤ 2 αDζ | x (1) c − [Table] 2) c | [Table]
[Equation]
Using x (1)
i
[Equation]
we obtain
[Equation]
The last inequality is based on
[Equation]
i )
[Equation]
i
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
i = [Equation] [Equation] [Equation]
Similarly on I 6 , one has I 6 ≤ 2 δM 2 ( (cid:99) W ) .
Thus, I 5 + I 6 ≤ 2 δ (cid:99) M 2 .
Note that
[Equation]
Proof of Proposition
(a) (Uniform
Cauchy’s
[Equation]
[Equation]
Use Gronwall’s lemma to obtain
[Equation]
(c) (Separation of the particle centers)
By (31)
[Equation]
we have
[Equation]
By the symmetry of ( a i,j ) ,
[Equation]
= 0 for p = 1 , 2 .
Thus,
[Equation]
[Equation]
By a similar estimate with Lemma
[Equation]
we have
[Equation]
N 1 (cid:88) k
= a k,j ( (cid:100) x (1) k
+ x (1) c − (cid:100) x (2) i − x (2) c ) + (cid:32)
[Equation]
δ
N
N 1 (cid:88) i = (cid:100) x (1) i − 2 δ
N
[Equation]
N 2 (cid:88) i = (cid:100) x (2) i (cid:33) ( x (1) c
[Equation]
(2) c ) + (cid:32)
[Equation]
N
[Equation]
N
[Equation]
(cid:88) i [Equation] N
[Equation]
(cid:88)
i
[Equation]
[Equation]
[Equation]
(cid:33) (
[Equation]
[Equation]
[Equation]
N
[Equation]
N
[Equation]
(cid:88) i = [Equation] [Equation] [Equation]
[Equation]
[Equation]
(cid:88)
i
[Equation]
( x (2)
i
[Equation]
( x (1) c
[Equation]
x (2) c )
[Equation]
N [Equation] N
[Equation] (cid:88) [Equation]
= N 1 (cid:88) k [Equation]
[Equation] i ( x (2) c [Equation] (1) c
[Equation]
(cid:88) i
[Equation]
N
[Equation]
( (cid:100)
[Equation]
[Equation]
[Equation]
[Equation]
N 2 (cid:88)
j =
N 1 (cid:88) k
= a k,i ( (cid:100) x
[Equation]
[Equation]
(52) where
I
[Equation]
[Equation]
By the Cauchy-Schwarz inequality,
[Equation]
For I c 2 , note that
[Equation]
Then, one gets
I c
[Equation]
Hence,
[Equation]
Combining with Lemma 1 and (48) [Equation] one obtains the estimate
[Equation]
By Gronwall’s lemma
[Equation]
if the initial data satisfy:
[Equation]
then,
[Equation]
(d)(Spatial separation of the two sub-ensembles) For any i = 1 , . . . N 1 , j [Equation] , N 2 ,
[Equation]
Then, there exists some time T ∗ such that ∀ t ≥ T ∗ , [Equation] Combing with Proposition 1, we finish the proof.
■
{+Remark 
+}
The proof of Proposition 3 is included in part (d) of proof of Proposition 2.
Now denote η 2 := (cid:80) i ∈I 1 ,j ∈Ia i,j > 0 in some channel, then the Dirichlet energy in this channel has a lower bound:
[Equation]
D Experiments
The code for the experiments is available at: https: [Equation] anonymous.4open.science [Equation]
r [Equation] ACMP-092A We will replace this anonymous link with a non-anonymous GitHub link after the acceptance.
We implement all experiments in Python 3.8.13 with PyTorch Geometric on one NVIDIA ® Tesla [-A
-]A
GPU with 6,912 CUDA cores and 80GB HBM2 mounted on an HPC cluster.
In addition, we take the official implementation of the Graph Neural Diffusion (GRAND) as diffusion term in (8) from the repository: https://github.com/twitter-research/graph-neural-pde
D.1 Details for Experiments
Datasets We consider two types of datasets: Homophilic and Heterophilic.
They are differentiated by the homophily level of a graph [45]:
[Equation]
v ∈ V Number of v ’s neighbors who have the same label as v Number of v ’s neighbors .
In the experiments, we have used six homophilic datasets, including Cora [35], Citeseer [50] and
Pubmed [40], Computer and Photo [40], and CoauthorCS [51], and three heterophilic datasets:
Cornell, Texas and Wisconsin from the WebKB dataset 4 .
For completeness, we list the numbers of classes, features, nodes and edges of each dataset, and their homophily level in Table 3.
The low homophily level means that the dataset is more heterophilic when most of neighbours are not in the same class, and the high homophily level indicates that the dataset close to homophilic when similar nodes tent to be connected.
The datasets we used in Table 3 covers various homophily levels.
Experiment setup For homophilic datasets, we use 10 random weight initializations andrandom splits, which contains 1,000 tests.
Each combination randomly select 20 numbers for each class.
For heterophilic data, we use the original fixed 10 split datasets.
We fine-tune our model within hyper-parameter search space, as detailed in Table 4.
We use the Dormand–Prince adaptive step size scheme (DOPRI5) as the neural ODE solver for all datasets.
Hyperparameter search used Ray Tune with a hundred trials using an asynchronous hyperband scheduler with a grace period of 50 epochs.
All the details to reproduce our results have been included in the submission and will be publicly available after publication.
[Table] [Table]
D.2 Ablation study for ACMP
Message Passing Performance vs Depths We compare ACMP with various GNN models such as
GRAND, GCN, GAT
[Table]
and GraphSage with different depths on the planetoid datasets.
Table 5 lists the nodes classification accuracy on Cora, Citeseer and Pubmed.
We observe that ACMP can maintain its model performance as the network deepens and achieve top test accuracy among all listed models using the same depth.
ACMP can thus overcome the oversmoothing
[Table] [Table]
Allen-Cahn term We now show in Figure 5 how Allen-Cahn term can stabilize training and prevent node features from blowing up.
The first row is the evolution of the diffusion equation without Allen-Cahn term while the second row has Allen-Cahn term added.
We can observe that introducing the repulsive term is essential for bounding GNN outputs, particularly when learning heterophilic datasets.
However, naively adding β to message passing will result in all node’s features becoming infinite.
In the first row of Figure 5 when Allen-Cahn term is not incorporated, the node’s features have increased to 3 × 10 3 when T = 10 , from 0 .
when T = 1 .
By the time T equals 30 , the node’s largest feature becomes 1 × 10 20 , which the neural ODE solver and message passing can hardly handle numerically corrected.
When we introduce Allen-Cahn term, the system contains two strong attractors of ± 1 , and the nodes are attracted to the two ends of 1 and − 1 by their own features.