Paper: Invariance-based Learning of Latent Dynamics (ICLR-2024)

Reviewer comment (Theory):
The definition of a system with realization-specific and realization-invariant properties is not clear. In chapter 3, I understand the authors try to explain its difference to the ode like systems and want to extend the ODE-like system setting to a more general settings with RS and RI properties, but I cannot find a specific example which is included in the RS-RI setting but excluded in the ODE setting. Can the author provide a useful example? It seems like all the experiments in the paper can be treated as ODE-like dynamical system, so I was wondering what is the major difference here?

Author response:
Q1: Definition of RI and RS properties / Real-world motivation\
Thank you for this question. Indeed, as you point out, our approach shares similarities with (neural) ODE approaches, but seeks to provide much more flexibility to bypass limits on the complexity of the trajectories that ODEs can model. Please see the overall response for a detailed description of the novelty of our contribution.

Our long-term real-world motivation comes from modelling of trajectories in the biomedical domain: 
Here, underlying processes are too complex to permit explicit description, but we think the invariances we identify can still hold.  Based on our experience (neural) ODE approaches are typically not well suited to capture these complex, high-dimensional systems including strong system delays, discontinuities and distribution shifts (e.g. under  treatment), and therefore require the development of more general and flexible dynamics models.

Reviewer comment (Novelty):
There is a missing literature review regarding invariant learning and its application in data-driven dynamical system learning. There are a few branches of related work worth mentioning and they are very close to the topic in this paper: Physics/Conservation invariant: "Hamiltonian neural networks." Advances in neural information processing systems 32 (2019). Lagrangian Neural Networks. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations. ConCerNet: A Contrastive Learning Based Framework for Automated Conservation Law Discovery and Trustworthy Dynamical System Prediction. International Conference on Machine Learning 2023. Symmetry invariant: Incorporating Symmetry into Deep Dynamics Models for Improved Generalization, ICLR 2021. Actually the Hamiltonian Neural network paper provides an image-based pendulum example which is similar to the experiment in this paper.

Author response:
Q2: Extension of related work\
We added a new subsection to Section A - Extended Related Work in the Appendix according to your suggestion and included the mentioned work, amongst others. Thank you for this suggestion.

Reviewer comment (Novelty):
Regarding the novelty of the model structure, I feel like it still belongs to the encoder-latent dynamics-decoder model class like many other prior works. Despite the author claimed the model decomposes the realization-specific (RS) and realization-invariant (RI) information, there is no evidence proving the claim both experimentally and theoretically.

Author response:
Q3: Contribution of work / Guarantee of decomposition\
Thank you for this feedback, but we politely disagree with the reviewers opinion. Please see the overall response for a precise description of the fundamental differences of our method to other approaches and discussion re: decomposition in RI dynamical model and RS trajectory representation.

Of course, we agree that specific model components like convolutions, attention, and VAEs are not new, but combining these ideas under our theoretical view and the presented graphical model results in a novel and demonstrably powerful inference approach.

Reviewer comment (Evaluation):
In the performance comparison figure 4, is the X axis the time axis? The results surprised me because the prediction from the first 3 models is bad starting from the beginning of the prediction, which means the error mostly comes from decoder reconstruction rather than dynamics prediction. Especially for ODE2VAE, VAE should be able to reconstruct the pendulum image fairly well.

Author response:
Thank you for mentioning this detail. You are right, the x-axis depicts the temporal evolution of the system. Regarding your concern, there is a simple explanation: In Fig. 4, the visualization of the predicted trajectory starts after $25\%$ of the entire length meaning that only future time predictions are visualized without the initial reconstruction of the input. Unfortunately, we missed to provide this detail in the figure caption, but amended it now for clarification. Apologies for the inconvenience.

Q5: Initially, we started to display prediction after 25\% since we put focus on long-horizon future prediction.  However, we updated all figures so that  they also display system states right from the beginning of the prediction. Please see Figs. 4 and K.1 – K.6 in the Appendix. The updated figures show that the baseline algorithms (except ODE-RNN) reconstruct reasonably the first few prediction time steps but diverge for long-term prediction.

Reviewer comment (Evaluation):
In Figure 5, the MSE from the proposed method is 10-100 times smaller than the prior works. This is another surprise as such a difference usually happens between white box model and black box model. Could there be bugs in your code, or is there any unfair comparison? I’m not trying to be mean but it does not look reasonable for me unless formal claims/evidence is provided.

Author response:
Q6: Analogy to black/white-box model / masssive emperical performance boost\
This is a great observation and your analogy of black and white-box model describes perfectly the difference between our approach and neural ODE based systems. Based on the conditional independence between latent states as outlined in the global answer we can directly predict the latent state $z_{t_{t_q}}$ without needing any information from previous states.  As you correctly point out,  this is equivalent to a black-box model for state transitions. In contrast, other approaches rely on the Markov assumption, meaning that they are modeling explicitly the transition between consecutive states. This is effectively a white-box description of state transition which we mitigate with our novel approach.
    
For all baseline methods we used the code provided in the respective official repositories. Therefore, we do not think it is likely that bugs underlie the results.  Moreover, the test results in Appendix J including Figs. J.1 - J.6 nicely illustrate that the baselines are indeed able to reconstruct and predict future trajectory samples, but simply not at the same level of accuracy compared to LaDID. In addition, previous studies reported in literature outline the same weaknesses of the baseline methods and show almost identical results of future prediction of similar test cases, see for example Figs. 13, 22, 23 \& 24 in "Latent Neural ODEs with sparse Bayesian Multiple Shooting", Iakovlev et al., ICLR 2023.\
These arguments support our view that the empirical results reflect real behaviour. Rather, we think the performance gap is due to the effectiveness of the decomposition we propose and study.

Reviewer comment (Reproducibility):
In the latent dynamics model (part v in figure 1), how does the query time t work? Does this model need to solve it iteratively like ODE-like solver to time t or the neural network directly takes t as input and outputs the latent state at time t.

Author response:
Q7: Encoding of query time t\
Thank you for this question. We represent each queried time step $t_q$ as a relative time embedding using a sin/cos encoding. Therefore, our dynamics model is continuous in time and, more importantly, can handle irregular time grid by default. Additionally, we are not forced to solve for intermediate time steps.

Reviewer comment (Theory):
My prominent concern in the paper pertains to the insufficient explanation for why their model outperforms others. The main contribution of the paper lacks clear description and justification. As pointed out by the authors, both LaDID and MSVI (Iakovlev et al., 2023) share several similarities, such as using CNN encoders/decoders and the same transformer. The primary distinction between these models lies in replacing neural ODE with MLPs to learn $f_{\phi_{dyn}}$ and introducing a smoothness loss to discourage abrupt transitions between latent subpatches. It remains unclear how these modifications enhance the process of learning latent dynamics in comparison to ODE-based methods.

Author response:
Q1: Unclear how modifications enhance learning of latent dynamical systems\
Please find a fuller account of how our approach enhances the learning of latent dynamics in the overall response above. We clarify there why our approach is different from e.g. MSVI, and explain the invariances in more detail. We emphasize that they are general properties that we think hold for a variety of systems rather than specific invariances derived from equations for a particular system.

Reviewer comment (Writing):
The paper's writing can be improved, and some content from the Appendix may be integrated into the main text.

Author response:
Q3: Improved writing, add parts from App. to MS\
We are highly motivated to improve the clarity of our work and would appreciate any direct feedback on unclear sections. However, we are constrained by the strict page limit, which limits the amount of additional content we can include.

Reviewer comment (Writing):
The descriptions of the experiments are somewhat confusing and could benefit from increased clarity and explanation.

Author response:
Q4: Improved Exp. Description\
Due to the page limit, we have added the Sec. J in the Appendix to provide a more comprehensive description of the experiments and hope that the updated Appendix addresses your concerns.

Reviewer comment (Theory):
If I understand correctly, the paper addresses only the initial state as the specific (variant) trajectory factor, right? What about other sources of variability, like measurement noise?

Author response:
Q5: Measurement noise\
To demonstrate the robustness to noise, we provide evaluation metrics for various noise levels added to the mean of our experiments, e.g. Gaussian noise ($\sigma = [0.1, 0.2, 0.3, 0.4]$) and salt-and-pepper noise ($\rho = [10\\%, 35\\%, 50\\%, 70\\%]$). The results are presented in Sec. M1 and M2 of the Appendix. We note also that within our loss-based framework, measurement and process noise are, in a way, handled under a unified overall objective.

Reviewer comment (Writing):
In section 4.1, Model, the main text says *"To obtain a latent trajectory,  ... we choose as a set of different sine and cosine waves with different wave length."* Could you explain it?

Author response:
Q6: Encoding of time queries\
We stack the sin/cos of the first 16 integer frequencies contained in the observed time interval similar to a Fourier series yielding a 32-dimensional relative time encoding.

Reviewer comment (Evaluation):
Could you please explain the choice of using MSE/RMSE for evaluation? It seems that capturing the overall structure of dynamics (direction, shape, etc.) might be more crucial than the absolute amount of misprediction.

Author response:
Q7: Why MSE for evaluation?\
We agree that time-averaged RMSE values do not capture all relevant information and can easily yield misleading interpretation. Therefore, we consider three metrics including the time-averaged nMSE, the RMSE over time and the visualization of the pixelwise $L2$-error (see Figs J1-6) allowing us to precisely study the accuracy of the predicted dynamics.

Reviewer comment (Experiments):
Can LaDID effectively handle time-variant dynamical systems? If yes, how?

Author response:
Q8: Can LaDID effectively handle time-variant dynamical systems?\
Since we have not conducted experiments for such systems, we are hesitant to make claims about the effectiveness of LaDID in this context. However, we are optimistic that LaDID could be effective for some such systems, provided suitable higher-level regularities exist. Here, a key point is that LaDID does not pre-suppose any particular kind of model but rather the high-level properties detailed above, which can hold for some time-varying systems. The observed generalization performance in Section 6.2 suggests that LaDID may have the potential to handle time-varying systems. Further investigation will be required to address this question in future work.

Reviewer comment (Presentation):
In the graphical model (Figure A1), $\psi^r$ is out of the box "$N$". Then, how is $\psi^r$ a function of "$n$" in Eq. 3 and Eq. 9?

Author response:
Q9: Notation in graphical model vs. eq. 3 and 9\
Thank you for bringing this to our attention. It is important to distinguish between the conceptual representation in Fig. A1 and the implementation of the LaDID model. In Fig. A1, we illustrate the overarching concept, indicating that for each trajectory, we generate a unique $\psi^r$ and subsequently roll out the latent trajectory. The plate notation $N$ represents the length of the requested trajectory points. In eq. 3 and 9, the variable $N$ denotes the number of subtrajectories, with the understanding that the concept outlined above holds true for each subtrajectory. We recognize the potential confusion and updated the notation in Figure A1 and its caption for clarity.

Reviewer comment (Writing):
The text mentions *" ... mean-aggregation, which can be changed based on the task at hand."* Could you elaborate on the role and task-dependent nature of mean-aggregation?

Author response:
Q10: Mean-aggregation and possible alternatives\
In the proposed implementation we choose the mean for sake of simplicity, but many different aggregations appear possible. One counterpart which we want to investigate in future work is the replacement of the mean by a singular value decomposition. Since the derived embeddings of the $k$ input samples are low-dimensional, a SVD may be cost efficient in extracting coherent data structures across all embeddings. Another promising replacement may be the introduction of a RNN where we want to use its final hidden state as general initial condition representation.

Reviewer comment (Theory):
In Eq. 9, within the representation prior term, none of the variables appear to be a function of "$n$." Can you clarify this discrepancy?

Author response:
Q11: Dependence of prior on subtrajectory\
All priors are computed per subtrajectory and thus a function of $n$ which we assumed to be clear from the context, but for clarity, we have added the dependence to the equations.

Reviewer comment (Presentation):
What does "$D$" represent in Figure A1?

Author response:
Q12: Graphical model - plate notation D\
In Fig. A1, D refers to the total number of the training samples. We amended the figure caption to clarify this aspect.

Reviewer comment (Experiments):
Beyond classical scientific examples, can LaDID effectively learn developmental trajectories involving bifurcations and other complex phenomena?

Author response:
Q13: Application to bifucations and other phenomena\
We appreciate your curiosity in this regard but at the stage, we lack evidence to make statements about LaDID's effectiveness in learning such dynamics. We plan to conduct thorough investigations into the performance of LaDID in such scenarios in future.