Paper: Unleashing the Potential of Fractional Calculus in Graph Neural Networks with FROND (ICLR-2024)

Reviewer comment (Experiments):
To further highlight the strengths of FROND and to provide clearer guidance on its optimal application scenarios, I would recommend conducting additional evaluations on datasets that necessitate long-range interactions [1].

Author response:
We apologize for the previous misunderstanding. To illustrate the effectiveness of our oversmoothing mitigation, especially in contexts where depth is beneficial, we conducted additional experiments on the Peptides-func dataset. This dataset is recognized for its dependence on depth to facilitate interactions between long-range nodes. 
In our experiments, we exclusively focused on the impact of depth, avoiding any techniques like rewiring, to isolate its effect.

From the preliminary results in Table R5, F-GRAND significantly enhances GRAND's performance on this dataset. More specifically, we observe that when we have 5 layers, GRAND and its fractional counterpart F-GRAND both have similar performances. As more layers are added, GRAND’s performance remains stable but begins to decline when exceeding 10 layers. In contrast, F-GRAND demonstrates improved performance with an increased number of layers. We will include more experiments on additional datasets in the camera-ready revision.

We also direct the reviewer to our oversmoothness experiment detailed in Table R1 (reproduced in the following Table R6). In the Airport datasets, we note that FLODE achieves optimal performance at around 8 layers. In contrast, F-FLODE, with additional layers, shows increased performance, peaking at 80 layers.

We hope these clarifications adequately demonstrate how mitigating oversmoothness can indeed bolster model performance.


| Depth | 5 | 10 | 15 |
|-------|---|----|----|
| GRAND | **57.35±0.08** | 57.32±0.33 | 55.74±0.63 |
| F-GRAND | 58.46±0.45 | 62.41±0.36 | **62.53±0.15** |

**Table R5:** Graph classification on Peptides-func dataset. Performance metric is Average Precision (AP) (%)






|       | Depth | 4 | 8 | 16 | 32 | 64 | 80 | 128 | 256 |
|--------|-------|---|---|----|----|----|----|-----|-----|
| FLODE  |       | 80.90±1.64 | **82.79±1.24** | 82.45±1.41 | 76.95±9.96 | 69.88±9.88 | 65.03±9.46 | 62.42±10.43 | 56.93±6.98 |
| F-FLODE|       | 83.83±1.46 | 85.95±0.87 | 86.47±1.50 | 87.90±1.46 | 88.12±1.28 | **89.08±1.50** | 88.94±1.11 | 88.74±1.57 |


**Table R6:** Node classification on Airport dataset

Reviewer comment (Theory):
Drawing a more explicit connection between fractal characteristics and FROND’s efficacy, particularly in handling tree-like data, would contribute to a more coherent narrative and justification for the framework. I kindly suggest expanding on this aspect.

Author response:
**Response:**  It is crucial to highlight that the fractal dimension computation is solely based on the graph topology, similar the concept of $\delta$-hyperbolicity. Thus, it can only depict a trend between the fractal dimension and the optimal $\beta$ value derived from our experiments. For F-GRAND-l, the optimal $\beta$ is identified as 0.9, whereas for F-GRAND-nl, it is 0.4. This variation is notable even among the same model variants.  Developing a measure that integrates both feature similarity and graph topology remains a significant challenge. However, we propose that the optimal $\beta$ could act as an indicator of the dataset's fractal nature.


Dynamical processes exhibiting self-similarity in fractal media are known to be more accurately described by Fractional Differential Equations (FDEs). For instance, in scenarios where heat or mass disperses across such structures, the concentration is best modeled using fractional diffusion equations. 
_In existing literature, the relationship between fractional order and the fractal properties of media has been extensively studied._ Theoretical analyses have been conducted under rigorous assumptions of the media [F1, F2, F3], encompassing structures like comb- and brush-like graphs [F4] and fractal meshes [F5]. Additionally, empirical investigations, such as those presented in [F6], have further explored this relationship.
_They have shown that decreasing the fractal dimension leads to increasing the anomalous diffusion exponent and equivalently the fractional order $\beta$._ The fractal dimension effectively quantifies the complexity or the degree of 'brokenness' of a fractal structure, essentially measuring how pattern details—specifically, the number of self-similar pieces—alter with the scale of observation. Conversely, the fractional order in differential equations introduces a 'memory' or 'history dependence' within the system. It is hypothesized that a larger fractal dimension, indicative of a more complex structure, might be associated with a smaller fractional order in the differential equations modeling diffusion processes. This is because a higher fractal dimension suggests increased complexity and more obstacles in the medium, potentially slowing down processes like diffusion. Consequently, a smaller fractional order could more accurately represent the slower, more intricate diffusion process in such a medium. _The trend observed between the fractal dimension and the optimal $\beta$ value from our experiments aligns well with the anomalous diffusion model described in the aforementioned literature._


[F1] Nigmatullin, R. R. "The realization of the generalized transfer equation in a medium with fractal geometry." Physica status solidi (b) 133.1 (1986): 425-430.

[F2] Butera, Salvatore, and Mario Di Paola. "A physically based connection between fractional calculus and fractal geometry." Annals of Physics 350 (2014): 146-158.

[F3] A. Iomin, Subdiffusion on a fractal comb, Phys. Rev. E - Stat. Nonlinear, Soft Matter Phys. 83 (2011) 52106. doi:10.1103/PhysRevE.83.052106.

[F4] Plyukhin, Alex V., and Dan Plyukhin. "Random walks on uniform and non-uniform combs and brushes." Journal of Statistical Mechanics: Theory and Experiment 2017.7 (2017): 073204.

[F5] Sandev, Trifce, Alexander Iomin, and Holger Kantz. "Anomalous diffusion on a fractal mesh." Physical Review E 95.5 (2017): 052107.

[F6] Zhokh, Alexey, Andrey Trypolskyi, and Peter Strizhak. "Relationship between the anomalous diffusion and the fractal dimension of the environment." Chemical Physics 503 (2018): 71-76.

Reviewer comment (Reproducibility):
What is the computational complexity of FROND? What is the T chosen for each experiment? and how is the short memory principle applied?

Author response:
**Response:** Thank you for your attention to this matter. We concur with the significance of examining model computation. In our initial submission, we have comprehensively addressed both the computational costs in Appendix Sections D.4, D.5, and D.6.
The time complexity for the graph ODE framework is $\mathcal{O}(C E)$, with $C$ as the complexity for evaluating $\mathcal{F}(\mathbf{W}, \mathbf{X}(t))$, and $E=T / h$ for discretization steps. In contrast, FROND has a higher time complexity of $\mathcal{O}(C E \log (E))$ with fast convolution or $\mathcal{O}(C E^2)$ using direct calculation.
To effectively address the computation cost, we have introduced the Short Memory solver in Section C.3 of our paper. This solver leverages a shifting memory window with a fixed width $K$, a strategy discussed in detail in Section D.5.1. For additional complexity analysis and tests, we kindly refer the reviewer to these specific sections in our Appendix.

Reviewer comment (Experiments):
It is not immediately clear how this current approach quantitatively/qualitatively compares to existing approaches that exploit long-range memory in the modeling process, for instance Maskey et al. 2023 (see question 1).

Author response:
In our initial submission, we briefly discussed the differences between our FROND model and FLODE (Maskey et al. 2023) in the second paragraph on page 2 and the "Graph Neural ODE Models" section of the Related Work on page 17. In our revised version, we have included FLODE as a baseline and expanded the discussion in Section E.6 of the appendix. Fundamentally, FLODE differs from our work in the following key aspects:

- FLODE employs the fractional (real-valued) power of $\mathbf{L}$, namely $\mathbf{L}^\alpha$. 
The feature evolution model used by FLODE is given by:
\begin{align*}
\frac{\mathrm{d}\mathbf{X}(t)}{\mathrm{d}t} = -\mathbf{L}^{\alpha}\mathbf{X}(t)\mathbf{\Phi}.  \quad  \quad  \quad   \quad \quad \text{[FLODE]} 
\end{align*}
This is a graph spatial domain rewiring technique, as $\mathbf{L}^\alpha$ introduces dense connections compared to $\mathbf{L}$. As a result, FLODE introduces space-based long-range interactions during the feature updating process.

- In contrast, our FROND model incorporates the time-fractional derivative $D_t^\beta$ to update graph node features in a memory-inclusive dynamical process. In this context, time acts as a continuous counterpart to the layer index, leading to significant dense skip connections between layers due to memory dependence. Thus, FROND induces time/layer-based long-range interactions in the feature update process. Note that FLODE does not utilize time-fractional derivatives.
Our method is not only __compatible with various graph ODE modes, including FLODE (see F-FLODE below),__ but also extends them to graph fractional differential equation (FDE) models. 

In Section E.6, we introduce a new FROND model based on FLODE, called the F-FLODE model, which utilizes time-fractional derivatives for updating graph node features in FLODE:
\begin{align*}
D_t^\beta \mathbf{X}(t)=-\mathbf{L}^\alpha \mathbf{X}(t) \mathbf{\Phi},   \quad  \quad  \quad   \quad \quad \text{[F-FLODE]} 
\end{align*}
Note the difference in the equations in [FLODE] and [F-FLODE], where the two are equivalent only when $\beta=1$.  
This example illustrates that the FROND framework encompasses the FLODE model as a special case when $\beta=1$. Our experimental results indicate that F-FLODE outperforms FLODE with the optimal $\beta\ne 1$ in general. For more detailed information, we refer the reviewer to the revised version of our paper.


To better distinguish between FLODE and F-FLODE, we interpret them using Euler discretization, illustrating how skip/dense connections operate in each model and the flow of information across nodes and layers.

In the FLODE model, as detailed in the paper, the feature update rule is represented by the explicit Euler scheme:
\begin{align*}
\mathbf{X}(t_{n})=\mathbf{X}(t_{n-1})-h \mathbf{L}^\alpha \mathbf{X}(t_{n-1}) \mathbf{\Phi},
\end{align*}
Here, the dense connections between nodes created by $\mathbf{L}^\alpha$ and the direct (i.e., memory-less) connection to the preceding layer $\mathbf{X}(t_{n-1})$ enable layer $n$ to assimilate information from distantly connected nodes in layer $n-1$.

In contrast, in the F-FLODE model, following the fractional Euler scheme in Eq. (33), the feature update rule is:
\begin{align*}
\mathbf{X}\left(t_n\right)=\mathbf{X}(0)-\frac{1}{\Gamma(\beta)} \sum_{j=0}^{n-1} \mu_{j, n}\mathbf{L}^{\alpha}\mathbf{X}(t_j)\mathbf{\Phi},
\end{align*}
with $\mu_{j, n}=\frac{h^\beta}{\beta}\left((n-j)^\beta-(n-1-j)^\beta\right)$ and $h=t_n-t_{n-1}$ as the temporal step size. 
This model, through its dense connection between nodes by $\mathbf{L}^\alpha$ and historical links to all previous layers (${\mathbf{X}(0),\ldots,\mathbf{X}(t_{n-1})}$), facilitates layer $n$ in leveraging information from distant nodes across layers $0$ to $n-1$. 
Notably, when $\alpha=1$, FLODE reduces to local aggregation, with information exchange restricted to immediate neighbors from the preceding layer $\mathbf{X}(t_{n-1})$. 
However, in such cases with $\beta<1$, historical connections to previous layers in F-FLODE are maintained. Even though at layer $n$ both F-FLODE and FLODE have information exchange within the nodes' $n$-hop neighbors (where the shortest distance $\leq n$ ), F-FLODE enables more explicit information exchange since the past states have been stored in the previous layers. To see this, note that $\mathbf{X}\left(t_k\right)$ are the node states after the $k$-hop local aggregation. By using F-FLODE, we have more direct information exchange within the nodes' all $n$-hop neighbors, more than within the nodes with exactly distance $n$.
For a visual representation of information flow in FROND, we direct the reviewer to Fig. 3 in our paper.

Reviewer comment (Novelty):
Is it possible to get a clearer distinction between the approach of this work, which is modeling node feature evolution through the layers as a FDE, versus Maskey et al. 2023, which proposes using fractional graph Laplacian in the usual ODE framework. It appears that Maskey et al. 2023 approach also tackles oversmoothing via long-range dependency of the dynamics, which is the main theoretical justification of the current work as well. More specifically, are there simple examples in which one framework strictly encapsulates another? Most importantly, what is the advantage of using this framework over Maskey et al. 2023 framework?

Author response:
In our initial submission, we briefly discussed the differences between our FROND model and FLODE (Maskey et al. 2023) in the second paragraph on page 2 and the "Graph Neural ODE Models" section of the Related Work on page 17. In our revised version, we have included FLODE as a baseline and expanded the discussion in Section E.6 of the appendix. Fundamentally, FLODE differs from our work in the following key aspects:

- FLODE employs the fractional (real-valued) power of $\mathbf{L}$, namely $\mathbf{L}^\alpha$. 
The feature evolution model used by FLODE is given by:
\begin{align*}
\frac{\mathrm{d}\mathbf{X}(t)}{\mathrm{d}t} = -\mathbf{L}^{\alpha}\mathbf{X}(t)\mathbf{\Phi}.  \quad  \quad  \quad   \quad \quad \text{[FLODE]} 
\end{align*}
This is a graph spatial domain rewiring technique, as $\mathbf{L}^\alpha$ introduces dense connections compared to $\mathbf{L}$. As a result, FLODE introduces space-based long-range interactions during the feature updating process.

- In contrast, our FROND model incorporates the time-fractional derivative $D_t^\beta$ to update graph node features in a memory-inclusive dynamical process. In this context, time acts as a continuous counterpart to the layer index, leading to significant dense skip connections between layers due to memory dependence. Thus, FROND induces time/layer-based long-range interactions in the feature update process. Note that FLODE does not utilize time-fractional derivatives.
Our method is not only __compatible with various graph ODE modes, including FLODE (see F-FLODE below),__ but also extends them to graph fractional differential equation (FDE) models. 

In Section E.6, we introduce a new FROND model based on FLODE, called the F-FLODE model, which utilizes time-fractional derivatives for updating graph node features in FLODE:
\begin{align*}
D_t^\beta \mathbf{X}(t)=-\mathbf{L}^\alpha \mathbf{X}(t) \mathbf{\Phi},   \quad  \quad  \quad   \quad \quad \text{[F-FLODE]} 
\end{align*}
Note the difference in the equations in [FLODE] and [F-FLODE], where the two are equivalent only when $\beta=1$.  
This example illustrates that the FROND framework encompasses the FLODE model as a special case when $\beta=1$. Our experimental results indicate that F-FLODE outperforms FLODE with the optimal $\beta\ne 1$ in general. For more detailed information, we refer the reviewer to the revised version of our paper.


To better distinguish between FLODE and F-FLODE, we interpret them using Euler discretization, illustrating how skip/dense connections operate in each model and the flow of information across nodes and layers.

In the FLODE model, as detailed in the paper, the feature update rule is represented by the explicit Euler scheme:
\begin{align*}
\mathbf{X}(t_{n})=\mathbf{X}(t_{n-1})-h \mathbf{L}^\alpha \mathbf{X}(t_{n-1}) \mathbf{\Phi},
\end{align*}
Here, the dense connections between nodes created by $\mathbf{L}^\alpha$ and the direct (i.e., memory-less) connection to the preceding layer $\mathbf{X}(t_{n-1})$ enable layer $n$ to assimilate information from distantly connected nodes in layer $n-1$.

In contrast, in the F-FLODE model, following the fractional Euler scheme in Eq. (33), the feature update rule is:
\begin{align*}
\mathbf{X}\left(t_n\right)=\mathbf{X}(0)-\frac{1}{\Gamma(\beta)} \sum_{j=0}^{n-1} \mu_{j, n}\mathbf{L}^{\alpha}\mathbf{X}(t_j)\mathbf{\Phi},
\end{align*}
with $\mu_{j, n}=\frac{h^\beta}{\beta}\left((n-j)^\beta-(n-1-j)^\beta\right)$ and $h=t_n-t_{n-1}$ as the temporal step size. 
This model, through its dense connection between nodes by $\mathbf{L}^\alpha$ and historical links to all previous layers (${\mathbf{X}(0),\ldots,\mathbf{X}(t_{n-1})}$), facilitates layer $n$ in leveraging information from distant nodes across layers $0$ to $n-1$. 
Notably, when $\alpha=1$, FLODE reduces to local aggregation, with information exchange restricted to immediate neighbors from the preceding layer $\mathbf{X}(t_{n-1})$. 
However, in such cases with $\beta<1$, historical connections to previous layers in F-FLODE are maintained. Even though at layer $n$ both F-FLODE and FLODE have information exchange within the nodes' $n$-hop neighbors (where the shortest distance $\leq n$ ), F-FLODE enables more explicit information exchange since the past states have been stored in the previous layers. To see this, note that $\mathbf{X}\left(t_k\right)$ are the node states after the $k$-hop local aggregation. By using F-FLODE, we have more direct information exchange within the nodes' all $n$-hop neighbors, more than within the nodes with exactly distance $n$.
For a visual representation of information flow in FROND, we direct the reviewer to Fig. 3 in our paper.

Reviewer comment (Theory):
Is it possible to give a proof sketch/intuition of Theorem 2, in particular, why should we expect the slow algebraic rate? It is also interesting that the rate is tight. Following the proof in the appendix seems to suggest that this comes from a deeper result by Mainardi but there is not a lot of intuition there.

Author response:
Thank you for your insightful query regarding the intuition behind the slow algebraic convergence rate in Theorem 2. To provide a clearer understanding, we approach this from three perspectives.

- Fractional Random Walk Perspective: In a standard random walk, a walker moves to a new position at each time step without delay. However, in a fractional random walk, which is more reflective of our model's behavior, the walker has a probability of revisiting past positions. This revisitation is not arbitrary; it is governed by a waiting time that follows a power-law distribution with a long tail. This characteristic fundamentally changes the walk's dynamics, introducing a memory component and leading to a slower, algebraic rate of convergence. This behavior is intrinsically different from normal random walks, where the absence of waiting times facilitates a quicker exponential convergence.

- Information Interaction Perspective: From the response to Weakness 1 , we clearly observe that all the previous layers $\left(\mathbf{X}(0), \ldots, \mathbf{X}\left(t_{n-1}\right)\right.$ ) have a direct impact on $\mathbf{X}\left(t_n\right)$. This means the initial features are less forgettable than only a skip connection to the preceding layer $\mathbf{X}\left(t_{n-1}\right)$. The asymptotic convergence rate with dense layer connections is therefore expected to be lower than the convergence rate with only skip layer connections.

- Analytic Perspective: From an analytic perspective, the essential slow algebraic rate primarily stems from the slow convergence of the Mittag-Leffler function towards zero. 
To elucidate this, let's consider the scalar scenario. Recall that the Mittag-Leffler function $E_\beta$ is defined as:
\begin{align*}
E_\beta(z):=\sum_{j=0}^{\infty} \frac{z^j}{\Gamma(j \beta+1)}
\end{align*}
for values of $z$ where the series converges. Specifically, when $\beta=1$,
\begin{align*}
E_1(z)=\sum_{j=0}^{\infty} \frac{z^j}{\Gamma(j+1)}=\sum_{j=0}^{\infty} \frac{z^j}{j !}=\exp (z)
\end{align*}
corresponds to the well-known exponential function. According to [A1, Theorem 4.3.], the eigenfunctions of the Caputo derivative are expressed through the Mittag-Leffler function. In more precise terms, if we define $y(t)$ as
\begin{align*}
y(t):=E_\beta\left(-\lambda t^n\right), \quad t \geq 0,
\end{align*}
it follows that
\begin{align*}
D_t^\beta y(t)=-\lambda y(t)
\end{align*}
Notably, when $\beta=1$, this reduces to $\frac{\mathrm{d} \exp (-\lambda t)}{\mathrm{d} t}=-\lambda \exp (-\lambda t)$.

Further, we examine the behavior of $E_\beta\left(-\lambda t^n\right)$. As per [A1, Theorem 7.3.], it is noted that:

(a) The function $y(t)$ is completely monotonic on $(0, \infty)$.

(b) As $x \rightarrow \infty$,
\begin{align*}
y(t)=\frac{t^{-\beta}}{\lambda \Gamma(1-\beta)}(1+o(1)) .
\end{align*}

Thus, the function $E_\beta\left(-\lambda t^\beta\right)$ converges to zero at a rate of $\Theta\left(t^{-\beta}\right)$. Our paper extends this to the general high-dimensional case by replacing the scalar $\lambda$ with the Laplacian matrix $\mathbf{L}$, wherein the eigenvalues of $\mathbf{L}$ play a critical role analogous to $\lambda$ in the scalar case.

For a diagonalizable Laplacian matrix $\mathbf{L}$, the proof essentially reverts to the scalar case as outlined above (refer to Eq. (54) in our paper). However, in scenarios where $\mathbf{L}$ is non-diagonalizable and has a general Jordan normal form, it becomes necessary to employ the Laplace transform to demonstrate that the algebraic rate remains valid ((refer to the context between Eq. (55) and (56) in our paper)

We have included the above discussion to in our revised version.

[A1] Kai Diethelm. The analysis of fractional differential equations: an application-oriented exposition using differential operators of Caputo type, volume 2004. Springer, 2010.

Reviewer comment (Theory):
There should also be conditions under which the Laplace transform exists (and the Laplace transform of the derivative).

Author response:
Thank you for pointing out these issues. We have made the necessary corrections in our revised version.

Reviewer comment (Novelty):
Missing neural ODE literature: 'Stable Architectures for Deep Neural Networks' and 'A Proposal on Machine Learning via Dynamical Systems'.

Author response:
**Response:** Thank you for your valuable literature suggestions. We have incorporated the additional related work you recommended into our revised manuscript. We invite the reviewer to examine the "Related Work Section", where these updates are highlighted in blue for ease of reference.

We appreciate the opportunity to clarify our approach to the literature survey. In our initial manuscript submission, particularly on pages 16 to 17, we conducted a comprehensive exploration of the literature in the domain of graph neural ODEs and skip connections in GNNs. This survey encompasses pioneering studies dating back to 2018 and the most recent advancements, up to 2023, including references [R0-12] for (graph) neural ODEs and [R13-18] for skip connections in GNNs. The range of these citations, from foundational works to the latest research, underlines our commitment to providing a thorough and current literature survey.

Additionally, in our updated manuscript, we have incorporated more references as recommended by you. We trust that these clarifications and additions demonstrate our rigorous approach to the literature review and our dedication to maintaining a comprehensive understanding of the field.

[R0] Chen, Ricky TQ, et al. "Neural ordinary differential equations." In Advances Neural Inf. Process. Syst.2018.

[R1] Avelar, Pedro HC, et al. "Discrete and continuous deep residual learning over graphs." arXiv preprint arXiv:1911.09554 (2019).

[R2] Poli, Michael, et al. "Graph neural ordinary differential equations." arXiv preprint arXiv:1911.07532 (2019).

[R3] Rusch, T. Konstantin, et al. "Graph-coupled oscillator networks", In Proc. Int. Conf. Mach. Learn., 2022.

[R4] Chamberlain, Ben, et al. "Grand: Graph neural diffusion", In Proc. Int. Conf. Mach. Learn., pp. 1407–1418, 2021a.

[R5] Chamberlain, Benjamin, et al. "Beltrami flow and neural diffusion on graphs", In Advances Neural Inf. Process. Syst., pp. 1594–1609, 2021b.

[R6] Song, Yang, et al. "On the robustness of graph neural diffusion to topology perturbations", In Advances Neural Inf. Process. Syst., 2022.

[R7] Bodnar, Cristian, et al. "Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in GNNs", In Advances Neural Inf. Process. Syst., 2022.

[R8] Wang, Yuelin, et al. "ACMP: Allen-cahn message passing with attractive and repulsive forces for graph neural networks",  In Proc. Int. Conf. Learn. Representations, 2022b.

[R9] Zhao, Kai, et al. "Graph neural convection-diffusion with heterophily", In Proc. Inter. Joint Conf. Artificial Intell., Macao, China, 2023.


[R10] Thorpe, Matthew, et al. "GRAND++: Graph neural diffusion with a source term", In Proc. Int. Conf. Learn. Representations, 2022.

[R11] Choi, Jeongwhan, et al. "GREAD: Graph Neural Reaction-Diffusion Networks", In Proc. Int. Conf. Mach. Learn., 2023.

[R12] Maskey, Sohir, et al. "A Fractional Graph Laplacian Approach to Oversmoothing." arXiv preprint arXiv:2305.13084, 2023.

[R13] Hongyang Gao and Shuiwang Ji. Graph u-nets. In Proc. Int. Conf. Mach. Learn., pp. 2083–2092, 2019.

[R14] Xu, Keyulu, et al. "Representation learning on graphs with jumping knowledge networks", In Proc. Int. Conf. Mach. Learn., pp. 5453–5462, 2018.

[R15] Chen, Ming, et al. "Simple and deep graph convolutional networks", In Proc. Int. Conf. Mach. Learn., pp. 1725–1735, 2020.

[R16] Li, Guohao, et al. "Deepgcns: Can gcns go as deep as cnns?", In Proc. Int. Conf. Mach. Learn., pp. 9267–9276, 2019.

[R17] Li, Guohao, et al. "Deepergcn: All you need to train deeper gcns." arXiv preprint arXiv:2006.07739, 2020a.

[R18] Gutteridge, Benjamin, et al. "DRew: dynamically rewired message passing with delay", In Proc. Int. Conf. Mach. Learn., pp. 1225212267, 2023.

Reviewer comment (Novelty):
Missing graph ODE literature: recent papers like 'Anti-Symmetric DGN: a stable architecture for Deep Graph Networks' and 'Ordinary differential equations on graph networks'.

Author response:
**Response:** Thank you for your valuable literature suggestions. We have incorporated the additional related work you recommended into our revised manuscript. We invite the reviewer to examine the "Related Work Section", where these updates are highlighted in blue for ease of reference.

We appreciate the opportunity to clarify our approach to the literature survey. In our initial manuscript submission, particularly on pages 16 to 17, we conducted a comprehensive exploration of the literature in the domain of graph neural ODEs and skip connections in GNNs. This survey encompasses pioneering studies dating back to 2018 and the most recent advancements, up to 2023, including references [R0-12] for (graph) neural ODEs and [R13-18] for skip connections in GNNs. The range of these citations, from foundational works to the latest research, underlines our commitment to providing a thorough and current literature survey.

Additionally, in our updated manuscript, we have incorporated more references as recommended by you. We trust that these clarifications and additions demonstrate our rigorous approach to the literature review and our dedication to maintaining a comprehensive understanding of the field.

[R0] Chen, Ricky TQ, et al. "Neural ordinary differential equations." In Advances Neural Inf. Process. Syst.2018.

[R1] Avelar, Pedro HC, et al. "Discrete and continuous deep residual learning over graphs." arXiv preprint arXiv:1911.09554 (2019).

[R2] Poli, Michael, et al. "Graph neural ordinary differential equations." arXiv preprint arXiv:1911.07532 (2019).

[R3] Rusch, T. Konstantin, et al. "Graph-coupled oscillator networks", In Proc. Int. Conf. Mach. Learn., 2022.

[R4] Chamberlain, Ben, et al. "Grand: Graph neural diffusion", In Proc. Int. Conf. Mach. Learn., pp. 1407–1418, 2021a.

[R5] Chamberlain, Benjamin, et al. "Beltrami flow and neural diffusion on graphs", In Advances Neural Inf. Process. Syst., pp. 1594–1609, 2021b.

[R6] Song, Yang, et al. "On the robustness of graph neural diffusion to topology perturbations", In Advances Neural Inf. Process. Syst., 2022.

[R7] Bodnar, Cristian, et al. "Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in GNNs", In Advances Neural Inf. Process. Syst., 2022.

[R8] Wang, Yuelin, et al. "ACMP: Allen-cahn message passing with attractive and repulsive forces for graph neural networks",  In Proc. Int. Conf. Learn. Representations, 2022b.

[R9] Zhao, Kai, et al. "Graph neural convection-diffusion with heterophily", In Proc. Inter. Joint Conf. Artificial Intell., Macao, China, 2023.


[R10] Thorpe, Matthew, et al. "GRAND++: Graph neural diffusion with a source term", In Proc. Int. Conf. Learn. Representations, 2022.

[R11] Choi, Jeongwhan, et al. "GREAD: Graph Neural Reaction-Diffusion Networks", In Proc. Int. Conf. Mach. Learn., 2023.

[R12] Maskey, Sohir, et al. "A Fractional Graph Laplacian Approach to Oversmoothing." arXiv preprint arXiv:2305.13084, 2023.

[R13] Hongyang Gao and Shuiwang Ji. Graph u-nets. In Proc. Int. Conf. Mach. Learn., pp. 2083–2092, 2019.

[R14] Xu, Keyulu, et al. "Representation learning on graphs with jumping knowledge networks", In Proc. Int. Conf. Mach. Learn., pp. 5453–5462, 2018.

[R15] Chen, Ming, et al. "Simple and deep graph convolutional networks", In Proc. Int. Conf. Mach. Learn., pp. 1725–1735, 2020.

[R16] Li, Guohao, et al. "Deepgcns: Can gcns go as deep as cnns?", In Proc. Int. Conf. Mach. Learn., pp. 9267–9276, 2019.

[R17] Li, Guohao, et al. "Deepergcn: All you need to train deeper gcns." arXiv preprint arXiv:2006.07739, 2020a.

[R18] Gutteridge, Benjamin, et al. "DRew: dynamically rewired message passing with delay", In Proc. Int. Conf. Mach. Learn., pp. 1225212267, 2023.

Reviewer comment (Theory):
In one of the main contributions, it is said that "We provide an interpretation from the perspective of a non-Markovian graph random walk when the model feature-updating dynamics is inspired by the fractional heat diffusion process. Contrasting with the traditional Markovian random walk implicit in traditional graph neural diffusion models whose convergence to the stationary equilibrium is exponentially swift, we establish that in FROND, convergence follows an algebraic rate." Why is it true? If $\beta=2$ then the process is not diffusive at all. Rather, it is oscillatory, as shown in GraphCON (Rusch et al.).

Author response:
**Response:** We believe there has been a misunderstanding. Our assertion is that when the dynamic is **inspired by the fractional heat diffusion process**, this corresponds to the fractional GRAND model (F-GRAND) presented in Eq. (7) and (9) in our paper, where we clearly stated that $0<\beta \le 1$. 
Furthermore, the subsection title where we discuss this interpretation is Sec. 3.2 "Random Walk Perspective of F-GRAND-L", which clearly refers to the diffusive model.
 This is different from the cases $\beta>1$ or $\beta=2$, which are oscillatory models similar to GraphCON. In the revision, we have emphasized this again in the contribution section, marked in blue color.

Reviewer comment (Novelty):
In section 2.3, the authors should also discuss FLODE ('A Fractional Graph Laplacian Approach to Oversmoothing') which is very similar to this work and also uses fractional calculus.

Author response:
**Response:** In our initial submission, we had addressed this point in the second paragraph on page 2 and in the "Graph Neural ODE Models" section of the Related Work on page 17. We kindly invite the reviewer to refer to these sections for detailed information. Additionally, in our revised version, we have included FLODE as a baseline and expanded the discussion in Section E.6 of the appendix. Fundamentally, FLODE differs from our work in the following key aspects: 

- FLODE employs the fractional (real-valued) power of $\mathbf{L}$, namely $\mathbf{L}^\alpha$. 
The feature evolution model used by FLODE, specifically in its first heat diffusion-type variant, is given by:
\begin{align*}
\frac{\mathrm{d}\mathbf{X}(t)}{\mathrm{d}t} = -\mathbf{L}^{\alpha}\mathbf{X}(t)\mathbf{\Phi}.  \quad  \quad  \quad   \quad \quad \text{[FLODE]} 
\end{align*}
This is a graph spatial domain rewiring technique, as $\mathbf{L}^\alpha$ introduces dense connections compared to $\mathbf{L}$. As a result, FLODE introduces space-based long-range interactions during the feature updating process.

- In contrast, our FROND model incorporates the time-fractional derivative $D_t^\beta$ to update graph node features in a memory-inclusive dynamical process. In this context, time acts as a continuous counterpart to the layer index, leading to significant dense skip connections between layers due to memory dependence. Thus, FROND induces time/layer-based long-range interactions in the feature update process. Note that FLODE does not utilize time-fractional derivatives.
Our method is not only __compatible with various graph ODE modes, including FLODE (see F-FLODE below),__ but also extends them to graph fractional differential equation (FDE) models. 

In Section E.6, we introduce a new FROND model based on FLODE, called the F-FLODE model, which utilizes time-fractional derivatives for updating graph node features in FLODE:
\begin{align*}
D_t^\beta \mathbf{X}(t)=-\mathbf{L}^\alpha \mathbf{X}(t) \mathbf{\Phi},   \quad  \quad  \quad   \quad \quad \text{[F-FLODE]} 
\end{align*}
Note the difference in the equations in [FLODE] and [F-FLODE], where the two are equivalent only when $\beta=1$.  
This example illustrates that the FROND framework encompasses the FLODE model as a special case when $\beta=1$. Our experimental results in Tabs 23 and 24 indicate that F-FLODE outperforms FLODE with the optimal $\beta\ne 1$ in general. For more detailed information, we refer the reviewer to the revised version of our paper.

Reviewer comment (Experiments):
In section 3.1 the authors discuss the initial conditions of the ODE. It is not clear to me how do you initialize $\beta$ time steps. From the text I can infer that it is the same condition as the input features. Is that what was actually done? If so, does it make sense from an ODE perspective? Have the authors tried other initialization procedures?

Author response:
**Response:** FROND employs initial conditions analogous to those used in graph ODE models to allow it to be smoothly integrated with existing graph ODE models. Our selection of the Caputo fractional derivative is guided by this compatibility with the initial conditions of ODEs (cf. Sections B.3 and B.4.).

- In typical first-order graph ODE models, preliminary node features $\mathbf{X}$, possibly processed through a learnable encoder, serve as the initial condition: $\mathbf{X}(0) = \mathbf{X}$.

- The second-order graph ODE model GraphCON, described by the equation 
$$\mathbf{X}^{\prime \prime}=\sigma\left(\mathbf{F}_\\theta(\mathbf{X}, t)\right)-\gamma \mathbf{X}-\alpha \mathbf{X}^{\prime},$$

utilizes the technique of introducing an auxiliary velocity variable, $\mathbf{Y}(t) = \mathbf{X}^{\prime}(t)$ to transform the second-order ODE into a first-order system (cf. Eq(3) in GraphCON). This is a common technique for handling higher-order ODEs:
\begin{align*}
\begin{aligned}
& \mathbf{Y}^{\prime}=\sigma\left(\mathbf{F}_\theta(\mathbf{X}, t)\right)-\gamma \mathbf{X}-\alpha \mathbf{Y} \\\\
& \mathbf{X}^{\prime}=\mathbf{Y}
\end{aligned}
\end{align*}
Here, the initial condition is set as $\mathbf{X}'(0) = \mathbf{X}(0) = \mathbf{X}$ in this paper. _Thus, GraphCON can be effectively treated as a first-order ODE, with the initial feature dimension doubled by duplicating the initial features._


- In our FROND framework, we adopt a similar approach for initial conditions when we have higher orders: $\mathbf{X}^{(\lceil\beta\rceil-1)}(0)=\ldots=\mathbf{X}(0)=\mathbf{X}$, paralleling the initial conditions in graph ODE models. Notably, for cases where $\beta>1$, akin to handling higher-order ODEs, we convert these into multiple lower-order FDEs [A1, Section 8], with a new $\beta$ value satisfying $0<\beta\le 1$. 


[A1] Kai Diethelm. The analysis of fractional differential equations: an application-oriented exposition using differential operators of Caputo type, volume 2004. Springer, 2010.

Reviewer comment (Theory):
The authors mention that here only $\beta$ is only considered between 0 and 1. I wonder why. How would your model behave theoretically and practically if it was larger than 1?

Author response:
**Response** In the general framework outlined in Equation (6), we do not assert that the parameter $\beta$ should be limited to the range between 0 and 1. This is also evident from our above detailed explanation regarding the initial conditions. The choice of the $\beta$ range depends on the application.

In our experimental and theoretical investigation of the fractional versions of the first-order graph ODE models GRAND and CDE, we keep $\beta$ within the range $(0,1]$ so that our model is a diffusion-type differential equation to align with those used in GRAND and CDE. The initial condition is established as $\mathbf{X}(0)=\mathbf{X}$. As the reviewer correctly notes, a $\beta$ value exceeding 1 would shift these models from diffusion to oscillation, which will make comparison with GRAND/CDE difficult. In Section E.5. in our initial submission, we had investigated the performance of oscillatory models via F-GraphCON, a fractional oscillator extension of GraphCON. This extension essentially constitutes an FDE with an order $\le 2$.

Reviewer comment (Experiments):
The experiments indeed show that the proposed method improves compared to baselines produced by the authors, but they are quite narrow and show a partial picture of the current state of the art and existing methods. I would expect that the authors compare their work (experimentally) with other methods like FLODE, CDE, GRAND++, as well as other recent methods like ACMII-GCN++ ('Is Heterophily A Real Nightmare For Graph Neural Networks To Do Node Classification?') or DRew ('DRew: Dynamically Rewired Message Passing with Delay').

Author response:
**Response:** It appears that the reviewer may have overlooked the experiments in our Appendix. In our original submission, we had presented F-GRAND, F-CDE, F-GRAND++, F-GREAD, and F-GraphCON, which are extensions of well-known graph ODE models recognized in the community. The referenced ACMII-GCN++ and DRew, however, are not graph ODE models. To further address the reviewer's concerns, we have included the fractional versions of F-FLODE during this rebuttal phase (see Section E.6 in our revised version).

Reviewer comment (Experiments):
The authors state that the proposed method can be applied to any ordinary differential equation GNN, so can the authors please also show the results when applied to other baseline methods as discussed in the paper?

Author response:
**Response:** It appears that the reviewer may have overlooked the experiments in our Appendix. In our original submission, we had presented F-GRAND, F-CDE, F-GRAND++, F-GREAD, and F-GraphCON, which are extensions of well-known graph ODE models recognized in the community. The referenced ACMII-GCN++ and DRew, however, are not graph ODE models. To further address the reviewer's concerns, we have included the fractional versions of F-FLODE during this rebuttal phase (see Section E.6 in our revised version).

Reviewer comment (Experiments):
$\beta$ is a hyperparameter. What would happen if you learn it? How will it influence the results and the stability of your solver? Is there a principled way to choose the hyperparameter?

Author response:
**Response:** The hyperparameter $\beta$ is found by binary search. A significant challenge in using a learnable $\beta$ arises from the gamma function in the fractional derivative definition:
\begin{align*}
\Gamma(\beta)=\int_0^{\infty} t^{\beta-1} e^{-t} d t .
\end{align*}
Backpropagating through the gamma function becomes difficult primarily due to the integral. The gradient computations can become problematic and often unstable due to the following possible reasons.

1. Behavior and Gradient of the Gamma Function: The gamma function has poles (i.e., points of infinity) at certain values of $\beta$. As $\beta$ approaches these poles, the gradients can explode. The derivative of the gamma function with respect to $\beta$ involves the digamma function (the logarithmic derivative of the gamma function). This derivative can introduce rapid changes, especially near the poles of the gamma function, causing the gradients to be unstable.

2. Numerical Techniques: The gamma function is defined using an integral. Differentiating an integral, especially one that might not have a closed form for its derivative, can introduce numerical inaccuracies. These inaccuracies can become pronounced for certain values of $\beta$. In Pytorch/Tensorflow, we are relying on numerical methods to compute the gradient (e.g., numerical integration). There is potential for inaccuracies that could destabilize training, especially if $\beta$ moves into regions where the function behaves unpredictably.


An alternative approach to learning $\beta$ is to utilize a distributed fractional-order differential equation [B1, B2]. Rather than designating a single $\beta$ that necessitates fine-tuning, we employ a learnable measure $\mu$ over a range $[a, b]$ for $\beta$. The distributed fractional-order operator applied to the function $X(t)$, represented as
\begin{align*}
\int_a^b D^\beta X(t) \mathrm{d} \mu(\beta),
\end{align*}
 can be perceived as the limiting case of a weighted summation over derivatives of multiple orders, denoted as $\sum_i w\left(\beta_i\right) D^{\beta_i} X(t)$. This operator addresses the limitation that a single fractional-order operator $D^\beta$ has limited capacity to accurately model complex physical phenomena despite its greater generality compared to its integer-order counterpart. The distributed fractional-order operator can be conceptualized as an amalgamation of behaviors depicted by distinct fractional-order operators across a particular range. The new approach is characterized by the following graph dynamical equation:
\begin{align*}
\int_a^b D^\beta \mathbf{X}(t) \mathrm{d} \mu(\beta)=\mathcal{F}(\mathbf{W}, \mathbf{X}(t)),  \quad  \quad  \quad   \quad \quad \text{[E.1]} 
\end{align*}
where $[a, b]$ denotes the range of the order $\beta, \mu$ is a learnable measure of $\beta$, and $\mathcal{F}$ is a dynamic operator as in Eq(6) in our paper. The initial step is to approximate Eq[E.1] as follows:
\begin{align*}
\sum_{j=0}^n w_{j \mathrm{C}} D^{\beta_j} \mathbf{X}(t)=\mathcal{F}(\mathbf{W}, \mathbf{X}(t)) \quad  \quad  \quad   \quad \quad \text{[E.2]} 
\end{align*}
where $D^{\beta_j} \in[a, b], j=1,2, \ldots, n$, are distinct interpolation points and $w_j$ are weights associated with the measure $\mu$. Reflecting the learnable nature of $\mu$ over $\beta$, $w_j$ can be directly set to be a learnable parameter in our implementation. The subsequent step is to solve the multi-term fractional differential equation given by [E.2]. As per [A1, Theorem 8.1.], the multi-term fractional differential equation can be converted into a system of equations of single order $D^\gamma$, where $\gamma:=1 / M$ with $M$ being the least common multiple of the denominators of $\beta_1, \beta_2, \ldots, \beta_n$ when they are rational numbers. The classical fractional solvers presented in our paper can resolve the resultant system of single-order equations. 

The new model above is out of the scope of our current work and will be investigated in detail in future work.


[B1] Ding, Wei, et al. "Applications of distributed-order fractional operators: A review." Entropy 23.1 (2021): 110.

[B2] Diethelm, Kai, and Neville J. Ford. "Numerical analysis for distributed-order differential equations." J. Computational Applied Math. (2009): 96-104.

Reviewer comment (Novelty):
I am not certain that the method is novel, as it was also shown in 'Fractional Graph Convolutional Networks (FGCN) for Semi-Supervised Learning'.

Author response:
**Response:** The referenced paper, 'Fractional Graph Convolutional Networks (FGCN) for Semi-Supervised Learning,' is unrelated to our work. While it employs the fractional power of the Laplacian matrix, denoted as the $L^\alpha$ operator, akin to the fractional operator in FLODE, it differs significantly from graph ODE models and FROND. Notably, this paper does not incorporate any differential equations or the time-fractional derivative in any form.

Reviewer comment (Novelty):
Missing literature about skip connections in GNNs: 'Understanding convolution on graphs via energies'.

Author response:
**Response:** Thank you for your valuable literature suggestions. We have incorporated the additional related work you recommended into our revised manuscript. We invite the reviewer to examine the "Related Work Section", where these updates are highlighted in blue for ease of reference.

We appreciate the opportunity to clarify our approach to the literature survey. In our initial manuscript submission, particularly on pages 16 to 17, we conducted a comprehensive exploration of the literature in the domain of graph neural ODEs and skip connections in GNNs. This survey encompasses pioneering studies dating back to 2018 and the most recent advancements, up to 2023, including references [R0-12] for (graph) neural ODEs and [R13-18] for skip connections in GNNs. The range of these citations, from foundational works to the latest research, underlines our commitment to providing a thorough and current literature survey.

Additionally, in our updated manuscript, we have incorporated more references as recommended by you. We trust that these clarifications and additions demonstrate our rigorous approach to the literature review and our dedication to maintaining a comprehensive understanding of the field.

[R0] Chen, Ricky TQ, et al. "Neural ordinary differential equations." In Advances Neural Inf. Process. Syst.2018.

[R1] Avelar, Pedro HC, et al. "Discrete and continuous deep residual learning over graphs." arXiv preprint arXiv:1911.09554 (2019).

[R2] Poli, Michael, et al. "Graph neural ordinary differential equations." arXiv preprint arXiv:1911.07532 (2019).

[R3] Rusch, T. Konstantin, et al. "Graph-coupled oscillator networks", In Proc. Int. Conf. Mach. Learn., 2022.

[R4] Chamberlain, Ben, et al. "Grand: Graph neural diffusion", In Proc. Int. Conf. Mach. Learn., pp. 1407–1418, 2021a.

[R5] Chamberlain, Benjamin, et al. "Beltrami flow and neural diffusion on graphs", In Advances Neural Inf. Process. Syst., pp. 1594–1609, 2021b.

[R6] Song, Yang, et al. "On the robustness of graph neural diffusion to topology perturbations", In Advances Neural Inf. Process. Syst., 2022.

[R7] Bodnar, Cristian, et al. "Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in GNNs", In Advances Neural Inf. Process. Syst., 2022.

[R8] Wang, Yuelin, et al. "ACMP: Allen-cahn message passing with attractive and repulsive forces for graph neural networks",  In Proc. Int. Conf. Learn. Representations, 2022b.

[R9] Zhao, Kai, et al. "Graph neural convection-diffusion with heterophily", In Proc. Inter. Joint Conf. Artificial Intell., Macao, China, 2023.


[R10] Thorpe, Matthew, et al. "GRAND++: Graph neural diffusion with a source term", In Proc. Int. Conf. Learn. Representations, 2022.

[R11] Choi, Jeongwhan, et al. "GREAD: Graph Neural Reaction-Diffusion Networks", In Proc. Int. Conf. Mach. Learn., 2023.

[R12] Maskey, Sohir, et al. "A Fractional Graph Laplacian Approach to Oversmoothing." arXiv preprint arXiv:2305.13084, 2023.

[R13] Hongyang Gao and Shuiwang Ji. Graph u-nets. In Proc. Int. Conf. Mach. Learn., pp. 2083–2092, 2019.

[R14] Xu, Keyulu, et al. "Representation learning on graphs with jumping knowledge networks", In Proc. Int. Conf. Mach. Learn., pp. 5453–5462, 2018.

[R15] Chen, Ming, et al. "Simple and deep graph convolutional networks", In Proc. Int. Conf. Mach. Learn., pp. 1725–1735, 2020.

[R16] Li, Guohao, et al. "Deepgcns: Can gcns go as deep as cnns?", In Proc. Int. Conf. Mach. Learn., pp. 9267–9276, 2019.

[R17] Li, Guohao, et al. "Deepergcn: All you need to train deeper gcns." arXiv preprint arXiv:2006.07739, 2020a.

[R18] Gutteridge, Benjamin, et al. "DRew: dynamically rewired message passing with delay", In Proc. Int. Conf. Mach. Learn., pp. 1225212267, 2023.

Reviewer comment (Novelty):
A general point - the focus of the paper is the mitigation of oversmoothing. But, also as the authors state, there are many methods that already do it. Then my question is what is the added value of using this mechanism? Also, another important issue is oversquashing in graphs. Can the authors discuss how and if the proposed method can help with that issue?

Author response:
**Response:** We address the issue of oversmoothing from a unique perspective by utilizing fractional differential equations, which allows us not only to tackle this problem but also to extend graph ODE models to their fractional counterpart, the graph FDE models. This approach sets our method apart from others, as it not only targets oversmoothing but also broadens the scope and applicability of graph ODE models, enabling them to function effectively in more complex scenarios. Unlike other methods aimed specifically at addressing oversmoothing, our approach can function as a plug-in for various graph ODE models with ease. For instance, our method can enhance the performance of the GraphCON model, designed to mitigate oversmoothing, across various datasets (refer to Table 22 in our paper). The random walk interpretation of F-GRAND-L also provides a theoretical underpinning to why oversmoothing mitigation is achieved in FROND, with a tight order bound. In particular, the result on algebraic convergence is of significant theoretical interest. 

While our paper is not explicitly designed to tackle the oversquashing problem, we note that FROND is a general framework by which any graph ODE model can be plugged into. This implies that a graph ODE model designed to tackle oversquashing can now be put into the FROND framework to simultaneously achieve oversmoothing mitigation. We will explore this possibility in future work.