Paper: TANGO: Time-Reversal Latent GraphODE for Multi-Agent Dynamical Systems (ICLR-2024)

Reviewer comment (Writing):
The current method description appears to be somewhat confusing and would benefit from revision. For instance, in Section 3.1, Equation (6) poses challenges in understanding aspects like the derivation of $z_1^{\text{fwd}}(t)$, and it's worth noting that $g(\cdot)$ is missing in Figure 2.

Author response:
We sincerely thank reviewers for valuable suggestions. We will revise our writing and Figure 1 based on your comments for audiences to better understand our work. Regarding the application of TANGO to real-world dataset, we have applied our method to a new motion capture dataset shown below and observed improvement compared with baselines.  The reason in the submission we only utilize simulated datasets is that most real-world datasets contain more noise and the underlying dynamics are unknown to us, therefore hard to study time-reversal symmetry at the beginning. The added motion dataset is one example that how this physical-informed learning can help, and we will further investigate how TANGO can be used for other real-world applications in the future.

Table 1: Evaluation results on MSE ($10^{-2}$) on MoJoCo Dataset (Huang et.al. Neurips 2020)

| $\text{Model \quad }$        | $\text{LatentODE \quad}$ | $\text{HODEN \quad}$  | $\text{TRS-ODEN}_\text{GNN} \quad$ | $\text{LGODE} \quad$   | $\text{TANGO} \quad$   |
|--------------|-----------|----------|-------------------------|---------|---------|
| MSE          | 2.9061     | 1.9855 |0.2609                  | 0.7610  | 0.2192  |

Reviewer comment (Experiments):
While the experiments are well-executed on simulated data, there's a notable absence of real-world data. This could potentially be attributed to the inherent challenges associated with collecting such data, as acknowledged in the paper.

Author response:
We sincerely thank reviewers for valuable suggestions. We will revise our writing and Figure 1 based on your comments for audiences to better understand our work. Regarding the application of TANGO to real-world dataset, we have applied our method to a new motion capture dataset shown below and observed improvement compared with baselines.  The reason in the submission we only utilize simulated datasets is that most real-world datasets contain more noise and the underlying dynamics are unknown to us, therefore hard to study time-reversal symmetry at the beginning. The added motion dataset is one example that how this physical-informed learning can help, and we will further investigate how TANGO can be used for other real-world applications in the future.

Table 1: Evaluation results on MSE ($10^{-2}$) on MoJoCo Dataset (Huang et.al. Neurips 2020)

| $\text{Model \quad }$        | $\text{LatentODE \quad}$ | $\text{HODEN \quad}$  | $\text{TRS-ODEN}_\text{GNN} \quad$ | $\text{LGODE} \quad$   | $\text{TANGO} \quad$   |
|--------------|-----------|----------|-------------------------|---------|---------|
| MSE          | 2.9061     | 1.9855 |0.2609                  | 0.7610  | 0.2192  |

Reviewer comment (Writing):
In the introduction, the paragraph preceding "Contributions" mentions the "time-reversal loss" not requiring additional labels beyond "ground-truth observations." Clarification is needed regarding what constitutes these "ground-truth observations."

Author response:
Thank you for bringing this up. Here, “ground-truth observations” refers to the observed training trajectories, such as particles’ positions and velocities. We want to express that in order to compute the time-reversal loss, we do not need additional data input compared with purely data-driven ones such as LG-ODE (Huang et.al. Neurips 2020). Sorry for the misunderstanding and we will clarify this in our revised revision.

Reviewer comment (Evaluation):
Notably, the results in Table 1 reveal that TANGO exhibits a lower Mean Squared Error (MSE) on the Damped Spring dataset compared to the Simple Spring dataset, which is somewhat surprising. Investigating the underlying reasons for this discrepancy may shed light on the method's performance characteristics and potential areas for improvement.

Author response:
We believe the main reason behind this observation is that the velocities of the particles in the damped spring system are gradually decreasing. Therefore, the numerical scale of MSE is smaller compared to the simple spring system. We will report the MAPE (Mean Absolute Percentage Error) as complementary to MSE to enable direct comparison between different dynamic systems.

Reviewer comment (Theory):
First, I would appreciate clarification on the precise calculation of the reversing operator $R$ for TANGO. While the reversing operator holds a significant role in addressing time-reversal symmetry, the current version of the paper lacks a detailed description and computation method for it (if this information is located in the appendix, I apologize; however, I would recommend including it in the main text of the revised version for clarity). Considering that the reversing operator is typically defined in the phase space $(q, p)$, as mentioned in footnote 3, it remains unclear how it can be computed within the latent space $z$. At first glance, I think there are possible approaches, although they all have some concerns: The first approach involves splitting $z$ into two components, $(z_0, z_1)$, with $z_0$ serving as a pseudo $q$ and $z_1$ as a pseudo $p$, defining $R: (z_0, z_1, t) \to (z_0, -z_1, -t)$. However, questions arise regarding the effectiveness of such a straightforward separation, as constraints similar to those found in variational Hamiltonian Monte Carlo literature might need to be imposed on $z_1$. Secondly, one might consider reversing the observations $y = (q, p)$ directly, i.e., $\hat{R}: (q, p, t) \to (q, -p, -t)$, and then assume $R \circ z = f_{dec}(\hat{R} \circ y)$. However, in this case, the reversing operation $R$ within the latent dynamics might not be an involution. Lastly, a simple identity operation, i.e., $z$, where $R: (z, t) \to (z, -t)$ might be employed. Currently, it appears that this operation is used in the paper. However, assuming the use of $R: (z, t) \to (z, -t)$ raises a new question, which I will discuss in the following section.

Author response:
>W1: The reversing operator in the latent space and its connection to time-reversal symmetry.

A1:  The implementation of the time-reversal symmetry in the latent space is based on $\hat{R}: (z,t) -> (z,-t)$, where we follow Huang, et.al. Neurips 2020, Huh et al NeurIPS 2020 in that we do not separately model q,p. This implementation is to approximate time-reversal symmetry instead of time reversibility. This is because we generate the forward and backward trajectories using the same ODE function, and then utilize the loss to make them close to each other. It differs from time reversibility where underlying dynamics (i.e. the ODE function) of forward and backward trajectories can be different, as long as the resulting trajectories are close to each other [1].

Reviewer comment (Theory):
Since there is no explicit definition of the reversing operator, I will assume that it solely reverses the dynamics in the context of the time direction, as described in equation (8) (i.e., solving the latent ODE in $-t$ direction). In this case, I have reservations about whether equation (9), as implemented in the paper, indeed enforces time-reversal symmetry regularization for the learned dynamics. It appears to me that equation (9) may promote time-reversibility (invertibility) rather than time-reversal symmetry. I believe it is crucial to rigorously differentiate between these two concepts within this work. Time-reversal symmetry implies that forward and backward time dynamics are indistinguishable. In contrast, time reversibility implies that these two dynamics can be distinguished but are one-to-one mappings of each other (i.e., the dynamics have a unique solution over a given time interval). For instance, an ideal oscillator would remain indistinguishable whether it evolves with forward or reverse time dynamics, thereby clearly exhibiting time-reversal symmetry. Conversely, a damped oscillator can be distinguished in this context. The forward time evolution reduces amplitude over time, while the backward time evolution increases it. This system lacks time-reversal symmetry but still poses time-reversibility (invertibility) because when a video of a damped oscillator is played in reverse, it returns to the initial frame due to the existence of a unique solution, i.e., well-defined dynamics. Taking this into account, equation (9) appears to be a loss function designed to promote time reversibility, as evident from the lower middle figure in Figure 2. It is worth noting that many deterministic ODEs, including those governing physical systems, are theoretically guaranteed to exhibit invertibility (uniqueness of solutions) according to the Picard–Lindelöf theorem. However, in practice, the actual solutions computed by numerical solvers like RK45 may not always be perfectly invertible, especially in the case of chaotic systems and homoclinic orbits. It is conceivable that the proposed regularization of TANGO compensates for such deviations from the limited precision of numerical solvers or sensitivity on initial conditions for chaotic systems (does this align with the findings of Theorem 1? I haven't verified the appendix). Alternatively, TANGO might learn smoother and more noise-resistant dynamics thanks to matching the forward and inverse trajectories. I do not intend to argue that TANGO lacks technical validity; it is reasonable to expect that TANGO's predictive performance can be improved by introducing the proposed regularization, regardless of whether the target dynamics are conservative, non-conservative, T-symmetric, or T-asymmetric. However, I find it challenging to assert that the added regularization arises from time-reversal symmetry based on the current presentation of the paper. Therefore, I remain somewhat unconvinced by the argument presented in this paper.

Author response:
>Q1: About R, $\phi_t$ ,reversibility and time reversal symmetry.

Firstly, we would like to restate the definition of Time-Reversal Symmetry in our paper 

“The system is said to follow the Time-Reversal Symmetry if it satisfies
$\frac{dR(\boldsymbol{Z}(t))}{dt}=-F(R(\boldsymbol{Z}(t)),$
where $R(\cdot)$ is a reversing operator”

Let's examine the state of two systems and their underlying dynamics when we apply $R \circ \phi_t \circ R \circ \phi_t = I$. One of these systems satisfies time reversal symmetry, while the other does not.

**the system that satisfies Time reversal symmetry**

Step1 $\phi_t:$

$z(\tau)→z(\tau+t)=z(\tau)+\int_\tau^{\tau+t} F(z(s))ds$ 

the dynamic is $dz/dt=F(z)$                        

Step 2 $R$:

$z(\tau+t)\rightarrow R(z(\tau+t))=z(-\tau-t)$                                

Due to the definition of a system that satisfies Time reversal symmetry as show above, the dynamic is $dR(z)/dt=-F(R(z))$   

$\Rightarrow dz(-t)/dt=-F(z(-t))$ (1)

 $\Rightarrow dz(t)/d(-t)=-F(z(t))$    
 $\Rightarrow dz(t)/dt=F(z(t))$    
Step3 $ \phi_t:$ 

$z(-\tau-t)→z(-\tau)=z(-\tau-t)-\int_{-\tau-t}^{-\tau} F(R(z(s)))ds=z(-\tau-t)+\int_{-\tau-t}^{-\tau} F(z(s))ds$.           -

 the dynamic remains $dR(z)/dt=-F(R(z))$

Step 4 $R$:

$R:z(-\tau)\rightarrow z(\tau)$                                                                     

$dR(R(z))/dt=-(-F(R(R(z))))$ ($R$ is an involution operator, so $R \circ R=I$)

$\Rightarrow dz/dt=F(z)$     

Thus we can see that both the state and the underlying dynamic were the same as the initial state.
 
However, for previous pure-data driven neural networks that **don't adhere to Time Reversal Symmetry**, Eq(1) — which defines Time Reversal Symmetry — **doesn't exist**. Consequently, we cannot derive this process in the same manner as described above. This implies that $R \circ \phi_t \circ R \circ \phi_t = I$ **does not hold true for these systems**.

Thus, to enforce our model satisfy Time-Reversal Symmetry, we equivalently derive that two trajectories should converge based on $R \circ \phi_t \circ R \circ \phi_t = I$ form Lemma 1 and design our loss function based on this equivalent definition of time reversal symmetry.

Though the form of the loss might have led to some misunderstanding that we are using reversibility, our loss function actually derived from the equivalent definition based on time reversal symmetry (from Lemma1). And in our previous response, we have explained that it differs from time reversibility, where the underlying dynamics of forward and backward trajectories can differ as long as the resulting trajectories are close. This should adequately clarify that our model's training is guided by time reversal symmetry.”

Reviewer comment (Theory):
What is the precise definition of the reversing operator employed in TANGO? More broadly, is there a standardized or canonical approach to defining the reversing operator for latent-based dynamics?

Author response:
> Q1:Is it meaningful to satisfy the time-reversal symmetry when the reversing operator is an identity
> 

A1: Thanks for your suggestions. However, we would like to clarify that **R is not an identity**. In our response,  the reversing operator in the latent space is defined as $\hat{R}: (z,t) \rightarrow (z,-t)$, i.e. $R(z(t))= z(-t) = z(T-t)$, where T denotes the total trajectory length. It is **not an identity transformation** $z\rightarrow z$, as z is a time-dependent variable and the reversing operator R would reverse the time index t with a negative sign.

The governing differential equations of the system are as follows:

$\frac{dz(t)}{dt}=f(z(t))$

$\frac{dR(z(t))}{dt}=-f(R(z(t))) \rightarrow \frac{dz(-t)}{dt}=-f(z(-t))$

Now let's analyze the definition of time-reversal symmetry: $R \circ \phi_t = \phi_{-t} \circ R$.

For the left side: $R \circ \phi_t (z(\tau)) = R(z(\tau + t)) = z(-\tau - t)$. (1)

For the right side: $\phi_{-t} \circ R(z(\tau)) = \phi_{-t} \circ (z(-\tau)) = z(-\tau - t)$. (2)

It's evident that the left side is equal to the right side.

Furthermore, considering $R \circ R(z(\tau))=R(z(-\tau))=z(\tau)$, thus $R \circ R=I$. Therefore, Lemma 1 remains valid, as demonstrated below:

$R \circ \phi_t = \phi_{-t} \circ R$. （3）

$R \circ \phi_t \circ R= \phi_{-t} \circ R \circ R$.

$R \circ \phi_t \circ R = \phi_{-t} $.

$R \circ \phi_t \circ R \circ \phi_t= \phi_{-t} \circ \phi_t = I$.

Since the reversing operator  $\hat{R}: (z,t) \rightarrow (z,-t)$ is not an identity transformation, directly eliminating R from both sides of the Eq.(3) to get $\phi_{t} = \phi_{-t}$  is not applicable.

Thus, it is meaningful to satisfy the time-reversal symmetry based on  lemma1, where the reversing operator $\hat{R}: (z,t) \rightarrow (z,-t)$ is not an identity transformation.

Reviewer comment (Theory):
Does the suggested reversal loss function result in time-reversal symmetry? Alternatively, does it promote time-reversibility (= invertibility, solution uniqueness, reduction of numerical divergence, ...), as previously mentioned?

Author response:
> Q2: How the reversal loss (8 - 9) is derived from (and related to) Lemma 1.
> 

A2: The definition of time-reversal symmetry is $R \circ \phi_t = \phi_{-t} \circ R$. As previously demonstrated, we have proved that $\hat{R}: (z,t) \rightarrow (z,-t)$ satisfies Lemma 1. Hence, the definition of time-reversal symmetry $R \circ \phi_t = \phi_{-t} \circ R$ is equivalent to $R \circ \phi_t \circ R \circ \phi_t = I$. This implies that if we move t steps forward, then turn backward and move for t more steps, it shall restore back to the same state.
And it's important to note that $\hat{R}: (z,t) \rightarrow (z,-t)$ **is not an identity transformation**, so we can not directly eliminating R to get $\phi_{t} \circ \phi_{-t} =I $.

In Eq. (9), $\hat{y_i}^{fwd}(\cdot)$ represents the forward steps, while $\hat{y_i}^{rev}(\cdot)$ represents the steps taken during the reverse trajectory, and we expect they should be the same, which indeed adheres to the equivalent definition in Lemma 1.

Moreover, note that $\hat{y_i}^{fwd}(\cdot)$ is not equal to $\hat{y_i}^{rev}(\cdot)$, i.e. $\hat{y_i}^{fwd}(t) \neq \hat{y_i}^{rev}(t)$. They are **two different notations**. We use them as distinct markers to denote the forward and reverse trajectories. Thus, Eq. (9) does not appear as the form of $||y(t) - y(T-t)|| $, indicating that the implementation is not derived from $\phi_t \circ \phi_{-t} = I$. You can further refer to Figure 7 in the paper's appendix. The left segment of the figure details the implementation of our reversal loss.

Thank you again for your valuable suggestions and your time in reviewing our paper. We hope that your concerns are adequately addressed and may consider raising the score.

Reviewer comment (Experiments):
It would be valuable to explore the influence of the solver choice, particularly for LGODE, TANGO, TANGO (gt-rev), and TANGO (rev2) models, through ablation studies. For instance, what would be the outcome when employing a higher-order solver in lieu of the conventional RK45? Or, how would lower-order solvers like Euler impact the results? How about Leapfrog? Assuming that the proposed loss function indeed promotes time-reversibility (invertibility), it is plausible that the effectiveness of TANGO could diminish as the solver's precision increases.

Author response:
>Q1: Regarding performance when using different solvers.

A1: We appreciate the reviewer acknowledging the potential of our method of compensating numerical errors brought by ODE solvers (highlighted by Theorem 1). We add the following experiments to show the performance differences when using the Euler solver compared with RK4. As shown in the following table, TANGO achieves higher improvement when using RK4 compared with Euler, indicating that it adequately minimizes the higher-order Taylor expansion term to decrease numerical errors. We would like to mention that Leapfrog is not a suitable solver in our setting. As our motivation is to softly satisfy time-reversal symmetry so that it can be applicable to various dynamical systems. Leapfrog strictly enforces time-reversal symmetry. It can harm the performance of systems that do not strictly obey time-reversal symmetry such as damped/forced springs shown in Table 1. 

Table 1: Evaluation results on MSE ($10^{-2}$) on Different Solvers.

|           | $Simple$|$Spring\qquad $   | $Forced$|$Spring\qquad $       | $Damped$ |$Spring\qquad $        | $Pendulum $ | $\qquad $      |
|-----------|-------------------|---------|-------------------|-------|-------------------|-------|-------------|-------|
|           | Euler             | RK4   | Euler             | RK4   | Euler             | RK4   | Euler       | RK4   |
| LGODE     | 1.8443            | 1.7429| 2.0462            | 1.8929| 1.1686            | 0.9718| 1.4634      | 1.4156|
| TANGO     | 1.4864            | 1.1178| 1.6058            | 1.4525| 0.8070            | 0.5944| 1.3093      | 1.2527|
| improve % | 19.4057           | 35.8655| 21.5228           | 23.2659| 30.9430           | 38.8352| 10.5303     | 11.5075|

Reviewer comment (Evaluation):
In Figure 5, it is noticeable that HODEN, a model known for strict energy conservation, exhibits energy divergence in the damped spring system. What could be the reason for this occurrence?

Author response:
>Q2: Why HamitonianNN fails?

A2: HamitonianNN is designed to preserve energies for each single agent. In multi-agent dynamical systems, the energy is only conserved as a whole, instead of each individual’s energy being conserved. Therefore, it has inferior performance.

Reviewer comment (Writing):
Some minor typos, for example, page 3: real *worldsystems*, page 4: $\tau_1,\tau_2 \in \mathbb{R}^\textit{4}$, Theorem 1: The *reversal* loss $\mathcal{L}_{pred}$, ...

Author response:
>Q3: Typos.

A3: We sincerely thank the reviewer for reading our paper carefully and we will correct them in our revised version.

Reviewer comment (Novelty):
As highlighted in the “Strengths” part, the motivation of the work is compelling and the idea of addressing reversibility through a simple regularization term is interesting. Unfortunately these are not novel contributions of this paper. Rather they are adapted from Huh et al, NeurIPS 2020, essentially adding the graph NN dimension (which is straightforward extension) and the simplified reversibility regularization with associated theoretical analysis (which is a less straightforward one). Hence my issue here is that the work is perhaps a bit too incremental.

Author response:
A1:  Thanks for your comments. We would like to emphasize our contributions  1.) Our work is **the first** to establish the theoretical connection between the **physical meaning** of the time reversal symmetry loss with its **numerical benefits** in improving the accuracy in general, without requiring the systems to follow any energy constraint (Theorem 1). Therefore, we broaden the application of the proposed loss to more real-life dynamical systems. 2.) Secondly, time-reversal symmetry has different definitions that are equivalent. The loss functions in our paper and in Huh et.al. Neurips 2020 are built upon definitions in Lemma 1 and Eqn (5) respectively. Note that the loss functions are only approximations of time-reversal symmetry where they try to satisfy Lemma 1 and Eqn (5). We **theoretically** show in Appendix A.3 that our reversal loss is guaranteed to have a lower maximum error compared to their approximations. We also **empirically** show our loss is achieving better performance: In Table 1, $TANGO_{L_{rev}=\text{rev2}}$   is a model variant by replacing our reversal loss with the one proposed in Huh et al NeurIPS 2020. TANGO consistently outperforms $TANGO_{L_{rev}=\text{rev2}}$  across systems, showing the superiority of our approximation.

Reviewer comment (Reproducibility):
The empirical analysis lacks sufficient details for reproducibility. I am missing in the main body (as well as in the appendices) a clear indication of the presence of a validation set, on which for instance one needs to identify the proper \alpha regularization weight. The empirical analysis itself is not very compelling as it is limited to few simple simulated physical systems. While this is somewhat consistent with some published earlier literature, in more recent works there is the tendency to validate models on more compelling and complex setups, e.g. involving simulation of MuJoCo dynamics or CMU human trajectory data.

Author response:
A3: Thanks for your insightful suggestions. We will add the validation set details in Appendix D.4 where 10% of training samples are used as validation sets in our experiments. Towards the dataset, we use three different spring systems to illustrate the different combinations of energy conservation and time reversibility shown in Figure 1. We additionally use a challenging pendulum dataset to show our model performance is especially better compared to baselines. Although the data is simulated, it contains noise and is incompletely observed and irregularly sampled, to some extent reflecting the challenges that real-world data may face. 
For the MuJoCo human motion dataset, it differs from the simulated dataset in that it does not have explicit physical inductive bias. Since our work is positioned as physics-informed NN, time-reversal symmetry is not observed in MuJoCo so we believe our architecture design will not be very useful for it. We conducted experiments on the dataset from Huang.et.al. Neurips 2020 and showed the results below. we can see that TANGO outperforms existing baselines though the margin is relatively small.

Table 1: Evaluation results on MSE ($10^{-2}$) on MoJoCo Dataset (Huang et.al. Neurips 2020)

| $\text{Model \quad }$        | $\text{LatentODE \quad}$ | $\text{HODEN \quad}$  | $\text{TRS-ODEN}_\text{GNN} \quad$ | $\text{LGODE} \quad$   | $\text{TANGO} \quad$   |
|--------------|-----------|----------|-------------------------|---------|---------|
| MSE          | 2.9061     | 1.9855 |0.2609                  | 0.7610  | 0.2192  |

Reviewer comment (Novelty):
Authors are invited to expand the empirical and theoretical comparison to other works related to reversible architectures (see the sample provided in the Weakness sections)

Author response:
Thanks reviewer for the acknowledgment of our response.

We'd like to further clarify: 

In our response, we didn't claim reversible NN are **irrelevant** to our work. Instead, both of these works are motivated by the nature of dynamic system: reversible NN treat each RevNet unit as a transforming operator (which is reversible, i.e., input can be induced from output), and thus its forward pass can be treated as ODE without storing past activations. Definitely this line of work can be used to replace neural operator in any Neural-ODE for dynamic system modelling.

However, we'd like to clarify that simply replacing with a RevNet didn't resolve the time-reversible problem that we're studying. The core of RevNet is that input can be recovered from output via a reversible operation (which is another operator), similar as any linear operator $W(\cdot)$ have a reversed projector $W^{-1}(\cdot)$. In contrary, what we want to study is that the **same operator** can be used for both forward and backward prediction over time, and keep the trajectory same. (see our **eq 6 and 8, we are using the same $g(\cdot)$, instead of $g^{-1}(\cdot)$**, and we don't need and assume operator $g(\cdot)$ to be reversible, instead, we expect it can predict same trajectory when predicting the backward via reversing the time)

In summary, though both reversible NN and time-reversal share similar insights and intuition, they're talking about very different thing: reversible NN make every operator $g(\cdot)$ having a $g^{-1}(\cdot)$, while time-reversible assume the trajectory get from $\hat{z}^{fwd}=g(z)$ and $\hat{z}^{bwd}=-g(z)$ to be closer. **Making $g$ to be reversible cannot make the system to be time-reversible.** We totally agree they are orthogonal to each other and very likely combining them together can achieve better performance, which we might leave for future exploration. We will add the above discussion into our paper.


We have added the discussion in our revised draft and hope our clarification resolves your concern.  Please let us know if you have further concerns and questions

Reviewer comment (Experiments):
Can the Authors clarify the model selection setting of the work, in particular as pertains the optimization of the model hyperparameters?

Author response:
A: TANGO follows the GraphODE architecture as in Huang et al. 2021, and we use the default architectural hyperparameters detailed in Appendix D.2. For training hyperparameters, we search from the following ranges: lr ∈ {1e-3, 5e-4, 1e-4, 5e-5, 1e-5, 5e-6}, batch_size ∈ {256, 512}. For spring datasets, we search $\alpha$ from a range of smaller values {0.01, 0.02, 0.1, 0.5, 1, 2, 5, 10}. For the Pendulum, we search $\alpha$ from a range of larger values {5, 10, 50, 100, 1000}.