Paper: Graph ODE with Factorized Prototypes for Modeling Complicated Interacting Dynamics (ICLR-2024)

Reviewer comment (Novelty):
The mixture-of-experts architecture is not well-motivated. While the authors introduce the mixture as a way to enhance the expressivity of the model, it is unclear to me what is the advantage over a single message-passing function that also takes the node representation $\mathbf{u}_i$ and the global representation $\mathbf{g}$ as input.

Author response:
We are truly grateful for the time you have taken to review our paper and your insightful review. Here we address your comments in the following.

> Q1. The mixture-of-experts architecture is not well-motivated. While the authors introduce the mixture as a way to enhance the expressivity of the model, it is unclear to me what is the advantage over a single message-passing function that also takes the node representation and the global representation as input.

A1. Thanks for your comment. We have included a model variant PGODE w. Single, which takes the node representation and the global representation as input with a single message-passing function. The compared performance on the two datasets is recorded as below. From the results, we can observe that the full model performs better than PGODE w. Single, which shows that directly combining representation and the global representation cannot fully capture the complicated dynamics of every node. In contrast, our prototype decomposition can involve different GNN bases, which **model diverse evolving patterns** to jointly determine the individualized dynamics. This strategy can enhance the model expressivity, allowing for more accurate representation learning of hierarchical structures from a mixture-of-experts perspective. We have updated the manuscript accordingly.


| Dataset                | Springs (ID) | Springs (ID) | Springs (OOD) | Springs (OOD) | 5AWL (ID) | 5AWL (ID) | 5AWL (ID) | 5AWL (OOD)  | 5AWL (OOD)  | 5AWL (OOD)  |
|------------------------|------------------|------------------|-------------------|-------------------|------------------|------------------|------------------|-------------------|-------------------|-------------------|
| Variable     | $q$ | $v$ | $q$ | $v$ | $q_x$ | $q_y$ | $q_z$ | $q_x$ | $q_y$ | $q_z$ |
| PGODE w. Single | 0.208            | 0.434            | 0.248             | 0.481             | 3.010            | 3.741            | 3.143            | 3.523             | 4.691             | 3.839             |
| PGODE (Full Model) | **0.070**       | **0.262**       | **0.088**        | **0.291**        | **2.910**       | **3.384**       | **2.904**       | **3.374**        | **4.334**        | **3.615**        |

Reviewer comment (Experiments):
GNN-based discrete-time models are not compared in the experiments. It would be interesting to see how the proposed model compares to these models. For example, the NRI model (Kipf et al., 2018) (where the Springs and Charged datasets are from) could be a good baseline.

Author response:
> Q2. GNN-based discrete-time models are not compared in the experiments. It would be interesting to see how the proposed model compares to these models. For example, the NRI model (Kipf et al., 2018) (where the Springs and Charged datasets are from) could be a good baseline.

A2. We have included the baseline AgentFormer [1], NRI [2] and I-GPODE [3] in our performance comparison. The results of these comparisons are presented below and our method outperforms the compared methods, which verifies the superiority of our method. We have updated the manuscript accordingly. 


| Prediction Length | 12 (ID) | 12 (ID) | 24 (ID) | 24 (ID) | 36 (ID) | 36 (ID) | 12 (OOD) | 12 (OOD) | 24 (OOD) | 24 (OOD) | 36 (OOD) | 36 (OOD) |
|-------------------|------|------|------|------|------|------|------|------|------|------|------|------|
| Variable          | $q$  | $v$  | $q$  | $v$  | $q$  | $v$  | $q$  | $v$  | $q$  | $v$  | $q$  | $v$  |
| _Springs_           |      |      |      |      |      |      |      |      |      |      |      |      |
| NRI               | 0.103 | 0.425 | 0.210 | 0.681 | 0.693 | 2.263 | 0.119 | 0.472 | 0.246 | 0.770 | 0.807 | 2.406 |
| AgentFormer       | 0.115 | 0.163 | 0.202 | 0.517 | 1.656 | 1.691 | 0.157 | 0.195 | 0.243 | 0.505 | 1.875 | 1.913 |
| I-GPODE           | 0.159 | 0.479 | 0.746 | 3.002 | 1.701 | 7.433 | 0.173 | 0.498 | 0.796 | 3.193 | 1.818 | 7.322 |
| HOPE              | 0.070       | 0.176       | 0.456       | 0.957       | 2.475       | 5.409       | 0.076        | 0.221        | 0.515        | 1.317        | 2.310        | 5.996        |
| **PGODE (Ours)** | **0.035** | **0.124** | **0.070** | **0.262** | **0.296** | **1.326** | **0.047** | **0.138** | **0.088** | **0.291** | **0.309** | **1.337** |
| _Charged_           |      |      |      |      |      |      |      |      |      |      |      |      |
| NRI               | 0.901 | 2.702 | 3.225 | 3.346 | 7.770 | 4.543 | 1.303 | 2.726 | 3.678 | 3.548 | 8.055 | 4.752 |
| AgentFormer       | 1.076 | 2.476 | 3.631 | 3.044 | 7.513 | 3.944 | 1.384 | 2.514 | 4.224 | 3.199 | 8.985 | 4.002 |
| I-GPODE           | 1.044 | 2.818 | 3.407 | 3.751 | 7.292 | 4.570 | 1.322 | 2.715 | 3.805 | 3.521 | 8.011 | 4.056 |
| HOPE              | 0.614       | 2.316       | 3.076       | 3.381       | 8.567       | 8.458       | 0.878        | 2.475        | 3.685        | 3.430        | 10.953       | 9.120        |
| **PGODE (Ours)** | **0.578** | **2.196** | **2.037** | **2.648** | **4.804** | **3.551** | **0.802** | **2.135** | **2.584** | **2.663** | **5.703** | **3.703** |


| Prediction Length | 12 (ID)  | 12 (ID)  | 12 (ID)  | 24 (ID) | 24 (ID)  | 24 (ID)  | 12 (OOD)  | 12 (OOD)  | 12 (OOD)  | 24 (OOD)  | 24 (OOD)  | 24 (OOD)  |
|-------------------|------|------|------|------|------|------|------|------|------|------|------|------|
| Variable | $q_x$  | $q_y$  | $q_z$  | $q_x$  | $q_y$  | $q_z$  | $q_x$  | $q_y$  | $q_z$  | $q_x$  | $q_y$  | $q_z$  |
| _5AWL_    |      |      |      |      |      |      |      |      |      |      |      |      |
| NRI              | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM     |
| AgentFormer      | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM     |
| I-GPODE          | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM     |
| HOPE             | 2.326  | 2.572  | 2.442  | 3.495  | 3.816  | 3.413  | 2.581  | 3.528  | 2.955  | 4.548  | 5.047  | 4.007   |
| **PGODE (Ours)** | **2.098** | **2.344** | **2.099** | **2.910** | **3.384** | **2.904** | **2.217** | **3.109** | **2.593** | **3.374** | **4.334** | **3.615**     |
| _2N5C_    |      |      |      |      |      |      |      |      |      |      |      |      |
| NRI              | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM     |
| AgentFormer      | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM     |
| I-GPODE          | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM    | OOM     |
| HOPE             | 1.842  | 1.915  | 2.223  | 2.656  | 2.788  | 3.474  | 2.562  | 2.514  | 2.731  | 3.343  | 3.301  | 3.502  |
| **PGODE (Ours)** | **1.484** | **1.424** | **1.575** | **1.960** | **2.029** | **2.119** | **1.684** | **1.809** | **1.912** | **2.464** | **2.734** | **2.727**   |

Reviewer comment (Novelty):
Missing related work: https://openreview.net/forum?id=B1lGU64tDr, where GNNs are also used to model the dynamics of interacting objects and similar auxiliary losses are designed for representation learning.

Author response:
> Q3. Missing related work: https://openreview.net/forum?id=B1lGU64tDr, where GNNs are also used to model the dynamics of interacting objects and similar auxiliary losses are designed for representation learning.

A3. Thanks for your comment. We have added the discussion with [4] as follows: "R-SSM [4] models the dynamics of interacting objects using GNNs and includes auxiliary contrastive prediction tasks to enhance discriminative learning while our study focuses on dynamical systems under potential parameter shift and develop a graph ODE framework with high expressivity and generalization capacity."