ACMP: A LLEN -C AHN M ESSAGE P ASSING WITH A TTRACTIVE AND R EPULSIVE F ORCES FOR G RAPH N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural message passing is a basic feature extraction unit for graph-structured data considering neighboring node features in network propagation from one layer to the next.
We model such process by an interacting particle system with attractive and repulsive forces and the Allen-Cahn force arising in the modeling of phase transition.
The dynamics of the system is a reaction-diffusion process which can separate particles without blowing up.
This induces an Allen-Cahn message passing (ACMP) for graph neural networks where the numerical iteration for the particle system solution constitutes the message passing propagation.
ACMP which has a simple implementation with a neural ODE solver can propel the network depth up to one hundred of layers with theoretically proven strictly positive lower bound of the Dirichlet energy.
It thus provides a deep model of GNNs circumventing the common GNN problem of oversmoothing.
GNNs with ACMP achieve state of the art performance for real-world node classification tasks on both homophilic and heterophilic datasets.
I NTRODUCTION
Graph neural networks (GNNs) have received a great attention in the past five years due to its powerful expressiveness for learning graph structured data, with broad applications from recommendation systems to drug and protein designs (Atz et al., 2021;
Baek et al., 2021; Bronstein et al., 2021; 2017;
Gainza et al., 2020; Wu et al., 2020).
Neural message passing (Gilmer et al., 2017)
serves as a fundamental feature extraction unit for graph-structured data that aggregates the features of neighbors in network propagation.
We develop a GNN message passing, called the Allen-Cahn message passing (ACMP) , using interacting particle dynamics, where nodes are particles and edges representing the interactions of particles.
The system is driven by both attractive and repulsive forces, plus the
Allen-Cahn double-well potential from phase transition modeling.
This model is motivated by the behavior of the particle system of collective behaviors common in nature and human society, for example, insects forming swarms to work; birds forming flocks to immigrate; humans forming parties to express public opinions.
Various mathematical models have been proposed to model these behaviors (Albi et al., 2019; Motsch & Tadmor, 2011; Castellano et al., 2009; Proskurnikov & Tempo, 2017; Degond & Motsch, 2008).
There are two major components in this model.
First, while the attractive force forces all particles into one cluster, the repulsive forces allow particles to separate into two different clusters, which is essential to avoid oversmoothing.
However, repulsive forces could make the Dirichlet energy diverge.
We augment the model with the Allen-Cahn (Allen &
Cahn, 1979) term (or Rayleigh friction (Rayleigh, 1894)), which is crucial in preventing the Dirichlet energy in the evolution from becoming unbounded, allowing us to prove mathematically that the lower bound of the Dirichlet energy is strictly bigger than zero, hence avoiding oversmoothing.
Specifically, we will prove that under suitable conditions on the parameters, the dynamics of the
ACMP particle system will time-asymptotically form 2 d different clusters and the Dirichlet energy has a strictly positive lower bound.
The structure of ACMP can handle two problems in GNNs: oversmoothing and heterophily.
Oversmoothing (Nt & Maehara, 2019; Oono & Suzuki, 2019; Konstantin Rusch et al., 2022) means that all node features become undistinguishable, and equivalently, in the formulation of particle systems, features form only one consensus.
Heterophily problems means GNNs perform worse in potential gradient flow blowup heterophilic graphs (Lim et al., 2021; Yan et al., 2021).
It is due to the neighboring nodes of different classes are mistaken for the same class in GNNs like GCN and GAT.
However, he presence of repulsion in ACMP makes particles separate into two different clusters, hence provides a simple and neat solution for prediction tasks on both two problems.
Overall, the benefit of the Allen-Cahn message passing with repulsion is manifold.
1) It circumvents oversmoothing issue, namely the Dirichlet energy is bounded from below.
The network is stable in the sense that features and Dirichlet energy are bounded from above.
Feature smoothness (energy decreasing) and the balance between nodes features and edge features can be adjusted by network parameters that control the attraction, repulsion and phase transition.
The model can then reach an acceptable trade-off on self-features and neighbor effect, as shown in Figure 1.
Our model can thus handle node classification tasks for both homophilic and heterophilic datasets by using only one-hop neighbour information.
The proposed model can be implemented by neural ODE solvers for the system with attractive and repulsive forces.
In theory, we prove that Dirichlet energy of GNNs with ACMP has a lower bound above zero (limiting oversmoothing), as well as an upper bound (circumventing blow-up) under specific conditions.
This agrees with the experimental results (Section 6).
We also prove that ACMP is a process for the features to generate clusters thanks to the double-well potential, which provides an interpretable theory for node classification.
B ACKGROUND
Message Passing in GNNs Graph neural networks are a kind of deep neural networks which take graph data as input.
Neural Message Passing (MP) (Gilmer et al., 2017; Battaglia et al., 2018) is a most widely used propagator for node feature update in GNNs, which takes the following form: for the undirected graph G =
( V , E ) is with sets of nodes V and edges E , with x ( k − 1) i ∈
R d denoting features of node i in layer ( k − 1) and a j,i ∈ R D edge features from node j to node i,
[Equation]
where □ denotes a differentiable, (node) permutation invariant function, e.g., sum, mean or max, and γ and ϕ denote differentiable functions such as MLPs (MultiLayer Perceptrons), and N i is the set of one-hop neighbors of node i.
The message passing updates the feature of each node by aggregating the self-feature with neighbors’ features.
Many GNN feature extraction modules such as GCN (Kipf & Welling, 2017), GAT (Veliˇckovi´c et al., 2018) and GIN (Xu et al., 2018) can be written as message passing.
For example, the MP of GCNs reads, with learnable parameter matrix Θ , x ′ i
=
Θ ⊤ (cid:80) j ∈N i ∪{ i } a j,i √ ˆ
d
j ˆ d i x j , where ˆ d i = 1 + (cid:80) j ∈N ( i ) a j,i and ˆ D = diag( ˆ d 1 , . .
, ˆ d N ) is the degree matrix for A
+ I .
Graph attention network (GAT) uses attention coefficients α i,j as similarity information between nodes in the MP update x ′ i
= α
i,i Θx i + (cid:80) j ∈N
i
α
i,j Θx j , with α i,j =
[Equation]
The MP framework was also developed as PDE solvers in Brandstetter et al.
(2022b)
by embedding differential equations as a parameter into message passing like Brandstetter et al.
(2022a).
This paper regards particle system evolution (ODE) as message passing propagation, and the appropriate design of the particle system offers desired properties for the resulting GNN.
Graph neural diffusion Neural diffusion equations on graphs (GRAND) are proposed by Chamberlain et al.
(2021), which provides a unified mathematical framework for some message passings:
[Equation]
where G = diag( a ( x i ( t ) , x j ( t ) , t )) where a is a function reflecting similarity between nodes i and j, and x i is the scale-valued feature for node i, and x = ⊕ x i .
M OTIVATIONS 3. A TTRACTIVE AND REPULSIVE FORCES
The equation (2) itself can be interpreted in a formulation different from diffusion.
In this paper, we study the neural equations of interacting particle system , which has a similar structure to (2).
We rewrite (2) into a component-wise version and obtain a particle system
[Equation]
In the formulation of particle systems, one can easily discover the evolution trend of the features.
If a ( x i , x j ) > 0 , the direction of x i ’s velocity is towards x j , which means that x i is attracted by x j .
In the contrast, if a ( x i , x j ) < 0 , x i has a trend to move away from x j .
Hence, a ( x i , x j ) serves as the attractiveness or repulsiveness of the force between x i and x j .
In the diffusion model above, all a ( x i , x j ) ’s are positive, therefore all the node features in one connected component attract each other.
If the weight matrix ( a ( x i , x j )) N × N is right-stochastic, one can prove that the convex hull of the features will not dilate in time (see Motsch & Tadmor (2014);
Chamberlain et al.
Such feature aggregation means that message propagates along the edges of the graph and some potential consensus forms in the process.
However, the message propagation does not limit to consensus (corresponding to diffusion).
Information interaction can derive polarization of final judgement when negative message matters in some problems rather than positive message.
For instance, in a node classification task on a bipartite, the neighbour message is negative since connected nodes belong to different classes.
In the formulation of particle systems, the mechanism of positive and negative messages can be modelled by adding bias β i,j into (3)
[Equation]
The coefficient term a ( x i , x j ) − β i,j corresponds to the interactive force.
By adjusting β i,j , both in the system attractive and repulsive forces co-exist.
If a ( x i , x j ) − β i,j > 0 , x i is attracted by x j .
While if a ( x i , x j ) − β i,j < 0 , x i is repelled by x j .
If the coefficient equates zero, there is no interaction between x i and x j .
Then, the dynamics is enabled to adapt both positive and negative message passing.
In this way, the neural message passing can handle either homophilic or heterophilic datasets (see Section 6 for detailed discussion).
3. P SEUDO -G INZBURG -L ANDAU ENERGY
However, adding the term of repulsive force may cause the particles being pushed away to infinite, thus the Dirichlet energy becomes unbounded.
To avoid this problem, we add a forcing term δx i (1 − x 2 i ) , which we call an Allen-Cahn term .
Here, the coefficient α > 0 is multiplied just for technical convenience.
[Equation]
Gradient Flow
The variational principle governing many PDE models states that the equilibrium state is actually the minimizer of one specific energy.
The equilibrium state carries meaningful information and can therefore be used as embedded features in the context of machine learning.
We first introduce the Dirichlet energy and show that (3) can be characterized by looking into the corresponding Euler-Lagrange equation of the Dirichlet energy.
Let adjacent matrix
A represent the undirected connectivity between nodes x i and x j , with a i,j = 1 for ( i, j ) ∈ E and a i,j = 0 for ( i, j ) ̸∈ E .
The Dirichlet energy E in terms of G =
( V , E ) and node features x ∈ R N × d takes the form
[Equation]
By calculus of variation, we can formulate the corresponding particle equation
[Equation]
On the RHS of (7), the summation takes over the one-hop neighbors N i of node i, which aggregates the impact from the neighboring nodes.
Equation (7) is (5) when one takes adjacent matrix A as the weight matrix ( a ( x i , x j ))
N × N .
Particle equation with the double-well potential To avoid blowing-up of the solution, one can design an external potential to control the solutions so they are bounded.
Here, we define the pseudoGinzburg-Landau energy on graph G denoted by Φ : L 2 ( V ) → R , as a combination of the interacting energy and double-well potential W : R → R + , with
[Equation]
where parameters α, δ > 0 are used to balance the two types of energy.
From now on, we denote a ( x i , x j ) by a i,j for simplicity.
The pseudo-Ginzburg-Landau energy is not a true energy because the matrix ( a i,j − β i,j ) N × N can be non-positive definite.
If β i,j ’s all equate zero, it then becomes the
Ginzburg-Landau energy defined in Bertozzi & Flenner (2012)
; Luo & Bertozzi (2017).
Using this combined energy, we can obtain the Allen-Cahn equation with repulsion on graph as ∂ x ∂t =
−∇ x Φ , which is equivalent to (5).
A LLEN -C AHN M ESSAGE P ASSING
We propose the Allen-Cahn Message Passing (ACMP) neural network based on equation (5), where the message is updated by the evolution of the equation via a neural ODE solver.
To our best knowledge, this is the first time to introduce a type of message passing to amplify the difference between connected nodes by repulsive force.
Network Architecture Suppose d -dimensional node-wise features represented by a matrix x in where row i represents feature of node i. Our scheme first embeds the node feature x (0) =
MLP( x in ) by a simple multi-layer perceptron (MLP), which is treated as an input for ACMP propagation A : R d → R d , by x (0) (cid:55)→ x ( T ) , where x ( T ) =
x (0)+ (cid:82)
[Equation]
x
[Equation] [Equation]
, where ∂ X ( t ) ∂t is estimated by ACMP defined on G based on (5).
The node features x ( T ) at the ending time are fed into an MLP based classifier.
Then, we define the Allen-Cahn message passing by
[Equation]
Here α , δ ∈ R d are learnable vectors of the same length as the node feature x i .
While we can use a more general case when each edge ( i, j ) uses different trainable β i,j , we have simplied to single hyper-parameter β ∈ R + ∪ { 0 } , which makes the network and optimization easier.
The β in our model is a crucial parameter, which can be adjusted such that the attractive and repulsive forces both present to enrich the message passing effect.
If one chooses δ = 0 , β = 0 , our model is reduced to the graph neural diffusion network (GRAND) in Chamberlain et al.
In experiments, we would make significant use of nontrivial δ and β.
The operations of all terms are channel-wise, involving d channels, except a ( x i ( t ) , x j ( t )) , and ⊙ represents channel-wise multiplication for d feature channels.
Figure 1 illustrates the one-step
ACMP mechanism (9): Nodes with close colors attracts each other otherwise repel.
Nodes in the same block tend to attract each other and both colors are strengthened during message passing propagation.
The double-well potential prevents the features and Dirichlet energy from blowup.
In this process, node feature x ( t ) is updated to x ( t + ∆ t ) for a time increment ∆ t .
Ultimately, a bi-cluster flock is formed for node classification.
In the propagation of ACMP in (9), we need to specify how the neighbors are interacted, that is how the a ( x i ( t ) , x j ( t )) is evolved with time.
There are many kinds of methods to update the edge weights.
Two typical types of ACMP are GCN based (Kipf & Welling, 2017) and graph attention (GAT) based (Veli ˇ ckovi ´
c et al., 2018).
ACMP-GCN : this model uses deterministic a ( x i ( t ) ,
x j ( t )) , which is given by the adacency matrix
A = ( a i,j ) of the original input graph G and does not change with time.
That is, the coefficients in
GCNs a GCN i,j := a i
[Equation]
ˆ d i ˆ d j .
The message passing of (9) is reduced to
[Equation]
ACMP-GAT : we can replace a GCN i,j in (10) by the attention coefficients (1) of GAT, which with extra trainable parameters measures the similarity between two nodes by taking account of both node and structure features.
The system then drives edges to update in each iteration of message passing.
Neural ODE Solver Our method uses an ODE solver to numerically solving the equation ((9) and (10)) for ACMP.
To obtain the node features x ( T ) , we need a stable numerical integrator for solving the ODE efficiently and backpropagation of gradients.
Since our model is stable in terms of evolution time, most explicit and implicit numerical methods such as explicit Euler, Runge-Kutta 4th-order, midpoint,
Dormand-Prince
(Chen et al., 2018;
Lu et al., 2018; Norcliffe et al., 2020;
Chamberlain et al., 2021) work well as long as the step size τ is small enough.
In experiments, we implement ACMP using Dormand-Prince5 method which provides a fast and stable numerical solver.
The number of runs of our message passing, or the network depth of ACMP-GNN is equal to the numerical iteration number n t set in the solver.
Computational Complexity
The computational complexity of the ACMP is O ( NEdn t ) , where n t , N, E and d are number of time steps in time interval [0 , T ] , number of nodes, number of edges and number of feature dimension, respectively.
Since our model only considers nearest (one-hop) neighbors, E is significantly smaller than that of graph rewiring (Gasteiger et al., 2019;
Alon &
Yahav, 2021) and multi-hop (Zhu et al., 2020) methods.
Channel Mixer Channel mixing can be spontaneously introduced from the perspective of diffusion coefficients though our model is previously written in the channel-wise form.
Whether channel mixing happens depends on the specific GNN driver we choose for ACMP.
When the coefficients a ( x i ( t ) , x j ( t )) in (9) that do not update with time are a scalar or vector, like in ACMP-GCN, the operations of the message passing propagator are channel-wise and channel mixing is not incorporated.
On the other hand, the ACMP-GAT with graph attention driver incorporates a learnable channel mixing when the coefficients are tensors.
The channel mixer can be introduced by generalizing the Dirichlet energy to high dimension, for example, E ( x ) :
= 1 N (cid:80) i ∈V (cid:80) j ∈N i ( x i − x j ) T a i,j ( x i − x j ) , when a i,j ∈ R d × d are connectivity tensors.
D IRICHLET E NERGY
The dynamics (5) can circumvent the oversmoothing issue of GNNs (Nt & Maehara, 2019; Oono & Suzuki, 2019; Konstantin Rusch et al., 2022).
Oversmoothing phenomenon means that all node features converge to the same constant – consensus forms – as the network deepens, and equivalently, the Dirichlet energy will decay to zero exponentially.
This idea was first introduced in Cai & Wang (2020).
Konstantin Rusch et al.
(2022) gives an explicit form for oversmoothing.
In our model, as we will show below, the node features in each channel tend to evolve into two clusters departing from each other under certain conditions.
This implies a strictly positive lower bound of the Dirichlet energy.
In addition, the system will not blow up thanks to the Allen-Cahn term.
We put all the proofs and some related supplementary results in the appendix.
Proposition 1.
If δ > 0 , the node features x i in (5) is bounded in terms of ∥ · ∥ and energy for all t > 0 , i.e., E ( x ( t )) ≤ C, and ∥ x ∥ ≤ C, where the constant C only depends on N and λ max .
In the following propositions, we imitate the emergent behavior analysis in Fang et al.
(2019) (see
Appendix for details).
For a graph G with N nodes, its vertices are said to form bi-cluster flocking if there exist two disjoint sets of vertex subsets { x (1) i } N 1 i =1 and { x (2) i } N 2 j =1 satisfying
( i )
[Equation]
( ii ) ∃ C ′ , T ∗∗ > 0 such that
[Equation]
where x (1) i , x (2) i denote any component of x (1) i , x (2) i .
We now show the long-time behaviour of model (5) following the analysis of Fang et al.
for strength coupling ( α, δ ) that satisfies the following condition: there exists { β i,j } such that I := { 1 , . . .
, N } can be divided into two disjoint groups I 1 , I 2 with N 1 and N 2 particles respectively:
[Equation]
S ≤ a i,j with a i,j := a i
[Equation]
j − β i
[Equation]
S ≤ a i
[Equation]
with a i
[Equation]
j
[Equation]
a i
[Equation]
D
[Equation]
a i
[Equation] [Equation]
where S, D are independent of time t.
The S and D in (12) are the repulsive and attractive forces.
We prove that if the repulsive force between the particles is stronger than the attractive force, that is, S > D, the system is guaranteed to have bi-cluster flocking, as shown in Proposition 2 below.
For time t ≥ 0 , suppose x (1) c ( t ) and x (2)
c
( t ) are the feature centers of the two groups of the particles { x (1) i ( t ) }
N 1 i =1 and { x (2) j ( t ) } N 2 j =1 which are partitioned as above from the whole vertex set V , given by
[Equation]
GCN, GAT, GRAND ACMP-GCN Suppose x ( s ) c ( t ) has the d -dimensional feature, and let x ( s ) c,k ( t ) , k = 1 , . . .
, d, be the k th (dimension) component of the feature x
( s ) c ( t ) , s = 1 , 2 .
Proposition 2.
The system (5) has a bi-cluster flocking if for each k = 1 , . . .
, d, the initial | x (1) c,k (0) − x (2) c,k (0) | ≫ 1 , and if there exists a positive constant η such that
[Equation]
where the δ is the weight factor for the double-well potential in the equation (5).
Proposition 3.
For system (5) with bi-cluster flocking, there exists a constant C >
and some time
T ∗ such that ∀ t ≥ T ∗ ,
[Equation]
Thus, if the non-zero a i,j are all positive, the Dirichlet energy for ACMP is lower bounded by a positive constant.
E XPERIMENTS
Dirichlet Energy We first illustrate the evolution of the Dirichlet energy of ACMP by an undirected synthetic random graph.
The synthetic graph has 100 nodes with two classes and 2D feature which is sampled from the normal distribution with the same standard deviation σ = 2 and two means µ 1 = − 0 .
5 , µ 2 = 0 .
The nodes are connected randomly with probability p = 0 .
9 if they are in the same class, otherwise nodes in different classes are connected with probability p = 0 .
We compare the performance of GNN models with four message passing propagators: GCNs (Kipf & Welling, 2017), GAT (Veliˇckovi´c et al., 2018), GRAND (Chamberlain et al., 2021) and ACMPGCN.
In Figure 2, we visualize how the node features evolve from their initial state to their final steady state when 50 layers of GNN are applied.
Additionally, in Figure 3, we show the Dirichlet energy of each layer’s output in logarithm scales.
Traditional GNNs such as GCNs and GAT suffer oversmoothing as the Dirichlet energy exponentially decays to zero in the first ten layers.
GRAND relieves this problem by multiplying a small constant which can delay all nodes’ features to collapse to the same value.
For ACMP, the energy stabilizes at the level that relies upon the roots of the double-well potential in (9) after slightly decaying in the first two layers.
Node Classification We compare the performance of ACMP with several popular GNN model architectures on various node classification benchmarks, containing both homophilic and heterophilic datasets.
Graph data is considered as homophilic (Pei et al., 2020) if similar nodes in the graph tend to connect together.
Conversely, the graph data is said heterophilic if it has a small homophily level, when most neighbors do not have the same label with source nodes.
We aim to demonstrate that ACMP is a flexible GNN model which can learn well both kinds of datasets by balancing the attractive and repulsive forces.
The GCN for examples cannot perform well for heterophilic dataset as its message passing aggregates only the neighbor (1-hop) nodes.
The neural ODE is solved by
Torchdiffeq package with Dormand–Prince adaptive step size scheme.
Only few hyperparameters are needed to be tuned in our model.
For all the experiments, we fine tune the learning rate, weight decay, dropout, hidden dimensional of the β which controls the repulsive force between nodes.
We outline the details of hyperparameter search space in Appendix D.
Homophilic datasets The results of our study are presented for the most widely used citation networks: Cora (McCallum et al., 2000), Citeseer (Sen et al., 2008) and Pubmed (Namata et al., 2012).
Moreover, we evaluate our model on the Amazon co-purchasing graphs Computer and Photo (Namata et al., 2012), and CoauthorCS (Shchur et al., 2018).
We compare our model with traditional
GNN models: Graph Convolutional Network (GCN) (Kipf & Welling, 2017), Graph Attention Network (GAT) (Veliˇckovi´c et al., 2018), Mixture Model Networks (Monti et al., 2017) and GraphSage (Hamilton et al., 2017).
We also compare our results with recent ODE-based GNNs, Continuous Graph Neural Networks (CGNN) (Xhonneux et al., 2020), Graph Neural Ordinary Differential
Equations (GDE) (Poli et al., 2020) and Graph Neural Diffusion (GRAND) (Chamberlain et al., 2021).
To address the limitations of this evaluation methodology proposed by Shchur et al.
(2018), we report results for all datasets using 100 random splits with 10 random initialization’s, and show the node classification result with mean and standard deviation in Table 1.
[Table]
Heterophilic datasets We evaluate ACMP-GCN on the heterophilic graphs; Cornell, Texas and
Wisconsin from the WebKB dataset 1 .
In this case, the assumption of common neighbors does not hold.
The poor performance of GCN and GAT models shown in Table 2 indicates that many GNN models struggle in this setting.
Introducing repulsion can improve the performance of GNNs on heteroplilic datasets significantly.
ACMP-GCN scores 30% higher than the original GCN for the
Texas dataset which has the smallest homophily level among the datasets in the table.
Attractive and Repulsive interpretation As shown in Table 2 and Table 1, ACMP-GCN and
ACMP-GAT achieve better performance than GCN and GAT on both homophilic and heterophilic datasets.
The majority of a i,j − β in the homophilic are positive, which means most nodes are attracted to each other.
Conversely, most a i,j − β for the heterophilic are negative, which means that most nodes are repelled by their neighbors.
Several GNNs exploiting multi-hop information can achieve high performance in node classification (Zhu et al., 2020; Luan et al., 2021).
However, highorder neighbor information will make the adjacency matrix dense and therefore can not be extended to large graphs, due to heavier computational cost.
In our model, we take only one-hop information into account and add repulsive force ( β ≥ 0 ) to message passing, which has achieved the same or higher level of accuracy as multi-hop models in heterophilic datasets.
Performance of ACMP to β
Hyperparameter β is the key to introduce the repulsive force in
GNN, meaning that when a ij − β is negative, the two nodes repel one another.
To illustrate β ’s impact on different datasets, we use GCN as a diffusion term as a ij do not change during the ODE process and all the changes are related to β.
As shown by Figure 4, ACMP performs best in Cora [Table] (orange curve) when all nodes are attracted to one another i.e., all a [Table]
β is positive.
As the beta increases, the performance of the model degrades.
In contrast, for the Texas dataset, when all force is attractive, ACMP achieves only 70% accuracy (blue curve).
As β increases, most a ij − β is negative, and the model’s performance gets better.
When all the force is repulsive, ACMP achieves highest accuracy on Texas datasets, which is in accordance with our claim that repulsive force is important for heterophilic datasets.
R ELATED WORK
Neural differential equations The topic of neural ODEs becomes an emerging field since E (2017) and Chen et al.
(2018), with many follow-up works in the GNN field: Avelar et al.
(2019) used continuous residual modules for graph kernels; Poli et al.
(2020) extended the framework of
GNN to continuous time.
Sanchez-Gonzalez et al.
(2019) applied Hamiltonian mechanics to graph networks to predict future states.
GRAND (Chamberlain et al., 2021) approached graph deep learning as a continuous diffusion process and propagated GNNs by the graph diffusion equation.
Eliasof et al.
(2021) combined diffusion and wave PDEs for GNNs, and GraphCON (Konstantin Rusch et al., 2022) generalized this method.
The latter employed a second-order system to conquer oversmoothing of deep graph neural networks.
By exploiting the fixed point of the dynamical system,
Gallicchio & Micheli (2020) proposed FDGNN as an approach to graph classification.
Allen-Cahn based variational graph models In Bertozzi & Flenner (2012);
Luo & Bertozzi (2017); Merkurjev et al.
and references therein, authors extended Allen-Cahn related potential to graphical framework and developed a class of variational algorithms to solve the clustering, semisupervised learning and graph cutting problems.
The new ingredient of graph neural network which enables us to combine learnable attraction and repulsion separates our method from the classical variational graph models.
C ONCLUSION
We develop a new message passing method with simple implementation.
The method is based on the Allen-Cahn particle system with repulsive force.
The proposed ACMP inherits the characteristic dynamics of the particle system and thus shows adaption for node classification tasks with high homophily difficulty.
Also, it propels networks to dozens of layers without getting oversmoothing.
A strictly positive lower bound of the Dirichlet energy is shown by theoretical and experimental results which guarantees non-oversmoothing of ACMP.
[-For (14), there exists a proper δ ′ > 
-]
[-Then D − A is symmetric positive semi-definite with the eigenvalues 
-]
[-= 
-]
Experiments show excellent performance of the model for various real datasets.
[-Also, (cid:80) i ∈V x 
-]
[-i =
-]
[-= a k,i ( x (2) c − x (1) c ) 
-]