Paper: Hypergraph Dynamic System (ICLR-2024)

Reviewer comment (Novelty):
The proposed model appears to be a straightforward combination of existing efforts on HGNN and neural graph ODEs, essentially replacing the message-passing scheme of GNNs with a diffusive hypergraph adjacency. It would be beneficial to clarify how this combination advances the field beyond existing approaches.

Author response:
Thank you for your valuable feedback. I appreciate the opportunity to clarify how our proposed model represents a significant benefit beyond the existing HGNN and graph ODEs.



**Leveraging Strengths of Hypergraph Neural Networks and Graph ODEs**. Our model leverages the strengths of both hypergraph neural networks and graph ODE. Our model captures high-order correlation beyond pairwise interactions that are often overlooked in graph models. Unlike discrete models, such as HGNN, which can only capture representation at a few hidden layers, our ODE-based hypergraph method continuously tracks the evolution of these representations until finally stable. 



**Effective Operation at Higher Layer Depths**. Another significant advantage of our model lies in its capacity to operate effectively at higher layer depths. This capability contrasts with most traditional (hyper)graph neural networks, which are typically constrained to lower layer depths due to issues like over-smoothing. In (hyper)graph neural networks, the restriction to fewer layers means that each vertex primarily interacts with its immediate, local neighbors. This localized view often neglects the potential influence of more distant vertices, leading to a limited understanding of the overall (hyper)graph. Our model overcomes this limitation with its capacity for higher layer depths enabled by the diffusive hypergraph.



**Stability with Increased Layers**. The key advantage of our model is its stability at increased layer depths. While additional layers in most existing (hyper)graph approaches can lead to instability, our model maintains stable performance as the number of layers grows, which means that our model is more robust. This stability is crucial for delivering consistent and reliable results, marking a notable improvement over current methods and a meaningful contribution to the hypergraph field.

Reviewer comment (Evaluation):
The application of hypergraph ODEs is not sufficiently motivated. Although the proposed model claims to capture system dynamics, it is evaluated on node classification tasks using static graphs, leaving the potential benefits on dynamical systems unclear.

Author response:
Thank you for your insightful feedback regarding the motivation and application of hypergraph ODEs in our work. We are confident that our hypergraph ODE has the potential to advance continuous industrial scenarios involving hypergraphs. While our current evaluation focuses on static graph node classification tasks, we believe this serves as a foundational step in showcasing the stability and accuracy of our model.



In industrial applications, particularly those involving hypergraphs, stability in model predictions is as crucial as accuracy. Our model's ability to provide consistent and reliable predictions across diverse data types and conditions is a key advantage. Our model is available for various tasks. The choice of vertex classification tasks for our experiment is driven by their effectiveness in evaluating the stability and performance of hypergraph neural networks across common benchmarks. 



In our article, we take a step towards achieving dynamic continuity in previous hypergraph neural networks by supporting the dynamic continuous representation of vertex and hyperedge representations through the ODE. We thank the reviewer for your suggestion that both hypergraph structure and representation be dynamic, which will serve as a future direction for our research.

Reviewer comment (Evaluation):
The experimental results show limited improvement over baseline methods, and from the comprehensive comparison it seems that variations in model structures have little influence on performance.

Author response:
Thank you for your observations regarding the experimental results. We understand your concerns about the performance improvement over baseline methods. However, we wish to emphasize that the **primary contribution** of our work lies in achieving stable representation learning in (hyper)graphs, a general challenge in (hyper)graph neural networks [1,2]. 



**We emphasize the dual aspects of performance** in our work: not just the direct performance metrics but also, and importantly, the stability of performance. In practical applications of hypergraph neural networks, achieving optimal performance often depends on specific conditions, a requirement that many existing methods struggle to meet consistently. This is why the robustness becomes valuable. Our focus on **robustness** and reduced dependence on such conditions ensures reliable and consistent performance across a variety of scenarios and layer depths. Further, our model not only solves the stability problem of increasing the number of layers, but also provides performance enhancement.



Our model exhibits consistently high performance across an increasing number of layers, a notable advancement compared to many existing hypergraph neural networks where performance typically degrades with layer depth. This focus on maintaining stability at greater layer depths is a key advancement compared to models that only emphasize peak performance under specific conditions.

Reviewer comment (Theory):
What is the rationale for applying ODEs and continuous methods to static graphs? How does the model benefit from the system dynamics brought by its structure?

Author response:
**Response to Question 1** 

Thank you for your insightful query. There are several key rationales for applying ODEs and continuous methods to static hypergraphs. 



**Continuous Representations**. By modeling the continuous changes in vertex and hyperedge representations over time, we enable the neural network's hidden layer representation to transform continuously [3]. This approach facilitates a more effective capture of the evolving nature of the representation, ensuring stable outcomes as the number of layers increases.



**Extended Interaction Range**. Furthermore, integrating ODEs and continuous methods addresses a critical limitation of general (hyper)graph neural networks, which often focus only on neighbors with low distances due to layer depth.  In general neural differential equations, the discretization method corresponds to a multi-step model [4]. Therefore, the model we construct based on ODEs allows for interactions across extended distances. This capability enables vertices to interact with long-distance neighbors, thereby broadening the interaction scope from merely local to encompassing global distances.



**Fine-Tuning Through Control Term**. A distinctive distinction of our model lies in the inclusion of a control term, designed for fine-tuning representations during the diffusion process. The control term is an auxiliary function, allowing for adjustments to the diffusion process, to suit the specific dataset. This added layer of adaptability not only enhances the model's performance but also provides flexibility and precision in hypergraph representation learning.



Our model integrates ODEs within the hypergraph framework. This integration results in a more nuanced representation of hidden layers, providing a smoother transition between them. This contrasts with general (hyper)graph neural networks, where features can exhibit more substantial variations between layers due to fewer intermediary layers. Second, the continuous diffusion results become stable as the number of layers in the model increases. It shows that, given sufficient depth, the model can reach an equilibrium state in system dynamics, providing consistent and reliable output.

Reviewer comment (Experiments):
Could you provide insights into the model's performance when the hypergraph convolution is replaced with simpler graph operators like GCN, as part of ablation studies? The functionality of hypergraph convolution in the model is unclear.

Author response:
**Response to Question 2**

We add an ablation study on whether the model uses hypergraph operators or graph operators in different hypergraph datasets. The ablation study include four methods, namely Graph Convolutional Network (GCN), Hypergraph Neural Networks (HGNN), HDS$^{ode}$ using graph operators, and HDS$^{ode}$. In the following table, "OOM" and "HDS$^{ode}$(graph op.)" represent "out of memory" and "HDS$^{ode}$ using graph operators", respectively.

$\newline$


| Model                  | Cora-CA             | DBLP-CA             | News20              | IMDB4k-CA           | IMDB4k-CD           | DBLP4k-CC           | DBLP4k-CP           |
| ---------------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- | ------------------- |
| GCN                    | $65.99_{\pm3.69}$   | $82.22_{\pm1.05}$   | $67.57_{\pm0.70}$   | $43.47_{\pm2.39}$   | $41.02_{\pm2.22}$   | $90.18_{\pm1.22}$   | $64.47_{\pm0.90}$   |
| HGNN                   | $67.58_{\pm1.83}$   | $82.83_{\pm1.09}$   | $76.58_{\pm0.94}$   | $43.21_{\pm2.39}$   | $41.08_{\pm2.43}$   | $93.46_{\pm0.77}$   | $67.99_{\pm2.12}$   |
| HDS$^{ode}$(graph op.) | $67.48_{\pm3.04}$   | $82.78_{\pm1.93}$   | OOM                 | $43.81_{\pm3.16}$   | $41.53_{\pm2.18}$   | $90.67_{\pm 0.83}$  | $66.75_{\pm0.79}$   |
| HDS$^{ode}$            | ${68.92_{\pm1.28}}$ | ${83.05_{\pm0.53}}$ | ${76.75_{\pm1.07}}$ | ${44.26_{\pm2.11}}$ | ${42.30_{\pm2.92}}$ | ${93.85_{\pm0.50}}$ | ${69.52_{\pm1.19}}$ |

$\newline$

According to the results in the above table, we found that in the hypergraph dataset, the hypergraph method achieves better results than the corresponding graph method (i.e., HGNN beats GCN, and HDS$^{ode}$ beats HDS$^{ode}$(graph op.), respectively), which indicates that it is necessary to use hypergraph operators in hypergraph correlation structures. 



As a generalization of graphs, hypergraphs have richer correlation expression capabilities than graphs. Such as in film background, hypergraphs adeptly capture the multifaceted connections among actors. Each actor can be represented as a vertex, and a hyperedge encompasses various correlations, such as actors co-starring in the same movie or originating from the same country. This hypergraph approach allows for a comprehensive representation of connections beyond pairwise interactions. Traditional graph models fall short since edges link only two nodes. They struggle to directly represent these group-based relationships without becoming complex or losing clarity.



In the graph operator, each vertex interacts with its adjacent vertices based on the edge connecting two vertices, while the hypergraph operator generates pair-wise interaction or group interaction based on the hyperedge that can connect two or more vertices. This is a key advantage of hypergraph operators. 

$\newline$

Thank you once again for your valuable time and consideration. We hope our response addresses your concerns. We also welcome any new questions you may have.

Reviewer comment (Presentation):
For the framework figure (Fig. 2), a more detailed and clearer introduction should be helpful.

Author response:
We thank the reviewer for your valuable comments and suggestions. We are currently revising the manuscript to provide a more detailed and comprehensible description of the framework. Additionally, we are reviewing the paper to identify and correct all typos to enhance the overall clarity and readability of our work.

Reviewer comment (Evaluation):
As shown in Fig. 1, the performance of HDS-ode increases during the first 8 layers and becomes stable then. Can the authors further explain how to control the diffusion speed during this procedure?

Author response:
Thank you for your question regarding the control of diffusion speed. Each layer in HDS$^{ode}$ includes a diffusion term and a control term. The hyperparameters $\alpha_v,\alpha_e$ in the diffusion matrix $\begin{bmatrix}
            -\alpha_v I& \alpha_v D_v^{-1}H \newline
            \alpha_e D_e^{-1}H^\top &-\alpha_e I
        \end{bmatrix}$ can be considered as the speed in the diffusion process. Taking vertex representations for explanation, $\alpha_v$ represents the proportion of vertex representations in each layer obtained from hyperedges in the diffusion process. 



Expanding the vertex representation of Eq. (7), we get $X_v(t+1)=(1-\alpha_v)X_v(t+\frac{1}{2})+\alpha_v D_v^{-1}HX_e$. Specifically,  $\alpha_v$ dictates the proportion of vertex representations derived from hyperedges during diffusion. For instance, with a lower $\alpha_v$ value, close to 0, the diffusion speed is minimized, and the vertex representation $X_v(t+1)$ remains closer to its previous state $X_v(t+\frac{1}{2})$. Conversely, as $\alpha_v$ approaches 1, $X_v(t+1)$ primarily consists of hyperedge representations $D_v^{-1}HX_e$.



In our experimental setup, $\alpha_v,\alpha_e$  are kept constant, implying a consistent diffusion speed across layers. When the number of layers increases (for example, reaching 8 layers), the representation of vertices and hyperedges is stable after diffusion, and the performance no longer increases significantly.

Reviewer comment (Theory):
The discussion of statibility seems wrong. That's why we say the authors may not well pick up knowledge of (linear/nonlinear) dynamical systems. Sec. 5.1 discussed the stability of HDS-ODE, where the statement of the first sentence in this section is basically wrong. The so-called "control" term also iterate over time. It cannot be simplified the stability analysis of HDS-ODE as simply a dissusion of linear system X_dot = A X. Let us use the eq.(3) to clarify the point simply. Supposing we accept the split of the general state-space equation X_dot = f(X, t) as eq.(3), it is obviously that the stability analysis is a general stability analysis of nonlinear system, besides AX(t) there also exists g(X(t))! If you said your discussion of HDS-ODE refer to the neural implementation, we can see that in eq. (6) the nonlinear term is still there, except it is in NN form, and then embeds into eq.(7) to complete the iteration t+1. Furthermore, these propositions on stability for your "simplified" linear systems are well-known in the field of electrical engineering.

Author response:
**Response to Weakness 2, Question 2**

Thank you for your feedback on the stability discussion in our paper. 



In our analysis, we primarily focus on the stability of the diffusion component, which is the main element of our model and a critical aspect of hypergraph neural networks in general. This is due to the diffusion component's significant role in determining the overall effectiveness of the hypergraph model. To clarify, the first sentence in Section 5.1 is intended to convey that diffusion in our model represents the teleportation of vertex and hyperedge representations over time. As this diffusion process is a main component of our ODE-based approach, its stability analysis is crucial. 



The control term in our model is designed to act as a secondary, fine-tuning term. Its primary function is to add minor perturbations to the main diffusion process, allowing the model to adapt more effectively to different data. The diffusion component dictates the primary dynamics of the network, the control term provides the minor necessary adjustments to enhance performance and adaptability without fundamentally altering the core stability characteristics of the model. That's the reason why we mainly analyze the stability of diffusion.



If the parameters in the function $g$ are too large, they amplify $X$, thereby affecting the main diffusion component. By controlling the parameter size of $W$ and $b$, we ensure that the transformations they induce in the representation space are bounded. In our neural network implementation, we use L2 regularization to keep the values of $W$ and $b$ within a lower range. This helps control the size of these parameters, thereby preventing potential instability that large weights may cause. This approach is a standard practice in neural network training, suitable for our system architecture, and helps ensure the boundedness of the model's output, maintaining stability.

Reviewer comment (Theory):
Why do you call these two terms in eq.(3) as the "control" term and the "diffusion" term. We are not familiar with the terminology in the "small" field (that is consisted of these ~5 papers in introduction). However, in the mature fields of "dynamical systems" in math, "stochastic analysis" in probability or mathematical finance and "control theory/engineering" or "cybernetics" in engineering, neither the name of "control" nor "diffusion" may be properly defined. The control term usually refers to external signals or extrogeneous variables (in econometrics) that can be designed or modified. Indeed, if you assume, eq.(3) is not an autonomous system in nature (if purely looking at (3) itself, it is), but the feedback system by using the nonlinear state feedback law g(X) as the control "u" for the linear dynamical system X_dot = A X + u. It somehow legitimate the name "control". However, it is still not recommended in this way since it is confusing. Referring to the diffusion term, we cannot see why it can be call as this name, since the diffusion term is h(X)dW in the SDE dX(t) = f(X, t) dt + h(X,t) dW, where W is the Brownian motion. The AX(t) term in eq.(3) is actually the term of f(X,t), so the drift term.

Author response:
**Response to Question 1**

Thank you for your insightful comments regarding the terminology used in our paper. In the field of graph and hypergraph, particularly when discussing (hyper)graph neural networks, "diffusion" is commonly used to describe the process by which features or representations teleport across the vertices of the graph or across the vertices and hyperedges of the hypergraph. This usage aligns with the intuitive notion of diffusion as a teleport process. In our model, the term "diffusion" refers to the $AX$ term in Eq. 3, representing the spread of representations across the hypergraph's vertices and hyperedges with a similar definition as existing methods [1-5]. 



In our model, the term "control" specifically refers to a fine-tuning step that complements the primary diffusion process. It acts as an auxiliary function, adjusting and controlling the diffusion $AX$ to align more precisely with the downstream goals. This control step is internal, and designed to refine the diffusion of representations across the hypergraph's vertices and hyperedges, enhancing the performance and accuracy of the diffusion. By integrating this fine-tuning control function, our model achieves a balance between diffusion of vertex and hyperedge representations and task-aware adjustments, leading to more reliable and accurate performance. 



We hope our response addresses your concerns. We also welcome any new questions you may have.



[1] Johannes Gasteiger, Stefan Weißenberger, and Stephan Günnemann. "Diffusion improves graph learning." *Advances in neural information processing systems*, 2019.

[2] Xhonneux, Louis-Pascal, Meng Qu, and Jian Tang. "Continuous graph neural networks." *International Conference on Machine Learning*, 2020.

[3] Yifei Wang, Yisen Wang, Jiansheng Yang, Zhouchen Lin. "Dissecting the diffusion process in linear graph convolutional networks." *Advances in Neural Information Processing Systems*, 2021.

[4] Ben Chamberlain, James Rowbottom, Maria I Gorinova, Michael Bronstein, Stefan Webb, Emanuele Rossi. "Grand: Graph neural diffusion." *International Conference on Machine Learning*, 2021.

[5] Peihao Wang, Shenghao Yang, Yunyu Liu, Zhangyang Wang, Pan Li. "Equivariant Hypergraph Diffusion Neural Operators." *International Conference on Learning Representations*, 2022.

Reviewer comment (Theory):
See Weakness point 2 on Sec.5.1. Please explain, in particular, the first sentence of Sec.5.1, which seems not correct.

Author response:
We deeply appreciate your feedback. We understand the importance of rigorous stability analysis in our work and take your suggestions seriously.



Although the rebuttal period provides limited time, we try our best efforts to address your concerns regarding the stability analysis. We describe the stability analysis currently available. If we put the control term and the diffusion term all together, the equation is $\dot{X}=AX+\sigma(XW+b)$. Here, $\sigma$ is the activation function, specifically implemented as the ReLU function, and $W,b$ are the learnable parameters in the control term. In particular, we write $W$ as the right multiplication of $X$ because it is a transformation of features, which is distinguished from the diffusion effect among vertices and hyperedges in $AX$. As the property of $\sigma(\cdot)=\max(0,\cdot)$, implies that if the overall control term reaches zero, the system's overall stability reverts to that determined solely by the diffusion term.



In cases corresponding to $\dot{X}=AX+XW+b$, we note the resemblance to a Sylvester System $\dot{X}=AX+XW$. The stability in this scenario hinges on two conditions, as demonstrated in prior works [1]. Firstly, $A,W$ must be continuous square matrices on $\mathbb{R}$. And secondly, there exists a positive constant $K$ such that $\int_{-\infty}^{\infty}\left\|\Phi(t) Y(t) P Z^*(t) \Psi^*(t)\right\| \leq K$, $\forall t \geq 0$, where $Y(t),Z(t)$ are fundamental matrix solutions of the linear system $\dot{X}=AX$ and $\dot{X}=XW$, respectively, with $\Phi(t), \Psi^*(t)$ being their corresponding bounded solutions. When these conditions are combined with $b=0$, the system maintains overall stability, albeit under strong constraints. We aim to identify broader conditions and bounds that assure system stability, addressing this challenge in a comprehensive and mathematically rigorous manner.



We hope to provide a more detailed and thorough stability analysis in our final paper. In the coming period, we will dedicate more time to expanding and refining our results to present a more comprehensive understanding of the stability aspects of our model. We believe these additional efforts will not only strengthen our paper but also provide valuable insights for our readers.



We are warmly open to further discussions on all aspects of our work to ensure its quality.



[1] Kanuri, K. V. V., S. Bhagavathula, and K. Murty. "Stability Analysis of Linear Sylvester System of First Order Differential Equations." *International Journal of Engineering and Computer Science* 9.11 (2020).