Paper: Meta-Learning Nonlinear Dynamical Systems with Deep Kernels (ICLR-2024)

Reviewer comment (Novelty):
The use of terms such as “multi-task modeling” and “meta-learning” may require some careful considerations. The task in the manuscript is defined as each data instance. For example, in the dynamics modeling context, a single task here corresponds to learning a dynamics function (or inferring a parameter as in system identification) from a set of measurement, which is collected from simulating a set of ODEs with a specific parameter. This way of defining “task” may be considered as a non-standard way in “multi-task learning” and, thus, a proper definition would be needed. Relatedly, the use of the terminology “meta-learning” also seems to require proper justification. Without these, the current method can be seen as a single framework, which is a mixture of parametric and non-parametric ML algorithms, where the parametric part of the framework is being trained on multiple input instances (which are denoted as tasks in the paper) via likelihood maximization with a regular gradient descent algorithm (without specifically learning meta-learned model parameters).

Author response:
You are correct in your understanding of “task” under the meta-learning framework in our paper: it is learning the latent force and outputs of a system of parameterized equations. The task representation, or meta-parameters in meta-learning terminology, are inferred exactly via our framework. This is a key strength, since closed-form meta-losses enable gradient updates of the meta-parameters much more effectively. We also demonstrate how we perform far from the meta-train set. We believe this distinction and the specific advantages it brings to latent force inference are critical to understanding the contribution of our work, and we would further emphasize that our primary contribution is centered on latent force inference rather than meta-learning per se.

Reviewer comment (Reproducibility):
The assumption on the availability of the latent force seems to be less practical. As shown in one of the benchmarks (the ODE benchmark), it does seem that latent force needs to be inferred from another software package (Alfi). The paper does not provide much information on how reliable this process is or if there are other alternatives for collecting latent forces (if they are not readily available). Based on this limited information, it is less convincing if the proposed method is practical. A side note is that the software, Alfi, seems to be based on the algorithm, appeared in a preprint, which has not been published; this makes it ever harder to assess the practicality of the proposed method.

Author response:
Regarding the assumption of the availability of latent forces. Please note that LFMs are specified as a generative process involving differentiation equations and (typically) GP-distributed latent forces. Therefore, these latent forces can be sampled, the equations solved, yielding the training dataset for a DKLFM. The latent forces are thus only accessible at training time. At inference time, like any LFM, the latent force is not available to the model.  

We note that there may be some misunderstanding with regards to the use of Alfi, the software package we use to simulate a training dataset given the differential equations. We do not use Alfi to infer latent forces at any point. Moreover, Alfi is not based on an algorithm presented in this paper, and we don’t consider the publication status of Alfi to impact the utility of this method; indeed, if a different method was used to construct the training set, it would not impact the DKLFM.

Reviewer comment (Experiments):
An empirical side of the paper is weak. The proposed method has been tested on relatively simple sets of benchmark problems, which does not seem to provide much insight on the effectiveness of the method or the scalability of the method. It does not seem that there is no place where the scalability of the method is discussed. Finally, the output MSE of the method on the second benchmark problem seems to be very poor.

Author response:
While we acknowledge that the first experiment on gene interactions may be considered a simple baseline task (albeit a necessary one), we would argue that the PDE problem and periodic solution to the Lotka Volterra equations are more complex problems for the latent force model framework to solve. Indeed, LFMs are typically evaluated on simpler tasks than those discussed in this paper.  

Moreover, with regards to scalability, we would like to highlight Section 4.3 which specifically discusses scalability of our method. Our model’s output MSE beats that of the two comparison methods for the ODE task. For the PDE task, it is important to view the metrics in combination. It takes significantly longer to obtain a better MSE with other methods. We believe these factors are key to understanding the impact of our approach for large datasets.

Reviewer comment (Experiments):
The authors place their work within the Meta-Learning field, but they do not compare the proposed model with any other Meta-Learning method. Moreover, the lack of even mentioning the Meta-Learning approaches within the Related Works section. Unfortunately, it is an important issue, because the Meta-Learning field is known from considering many GP-based approaches. To mention just a few: [1], [2], [3], [4], and [5]. I strongly suggest to compare the proposed method with other Meta-Learning approaches. It could also be a good source of new ideas for the future work.

Author response:
Regarding Q1: we acknowledge your observation regarding the positioning of DKLFM in the context of meta-learning, and we would also like to thank you for sharing the meta-learning references. In response, we would like to clarify that our primary contribution is centered on latent force inference rather than meta-learning per se. Using meta-learning terminology,  the meta-parameters here are the task representations. The exact inference over these parameters is a key strength, since closed-form meta-losses enable gradient updates of the meta-parameters much more effectively. We also demonstrate how we perform far from the meta-train set. We believe this distinction and the specific advantages it brings to latent force inference are critical to understanding the contribution of our work.

Reviewer comment (Experiments):
If I understand correctly, the authors for each experiments used the same kernel - RBF. However, in the Deep Kernel setting, we can utilize any kernel function we would like - even the scalar product in an embedding space is able to imitate many other kernels. From my perspective, the lack of comparison across various kernels (e.g., family of Matern kernels, spectral or cosine kernel) is another important issue. Even if the RBF kernel would be the best for all of the presented experiments, I would like to see any justification or ablation study proving that.

Author response:
Regarding Q2, and the issue of using only the RBF kernel. We recognise that this was not made clear in the manuscript. To clarify, our paper does describe scenarios of periodic nature. Specifically, at the top of page 4 and in the middle of page 5, we outline where one might wish to use a periodic kernel. Furthermore, the practical application of the periodic kernel is demonstrated in one of our key experiments, as illustrated in Figure 4b. We apologize that this was not made clearer in the manuscript, and we have revised the manuscript to clarify this.

Reviewer comment (Evaluation):
I am not particularly sure if the MSE is the best metric to compare between the solutions. The interesting will be to see how it looks like in different measures, like NLL/ELBO, since it should be available in the GP setting.

Author response:
While the output MSE for DeepLFM is superior to DKLFM, our evaluation is not focussed on that singular metric. For use cases we have discussed in the paper, we believe that inference time is just as important, and sometimes is a dealbreaker for usability.

Reviewer comment (Experiments):
I am not certain if I understand correctly the comparison between Alfi, DeepLFM and DKLFM. What is the size of datasets on which each result in those tables are computed? If I understand it looks like this: the results for Alfi and DeepLFM are from the set of 20 examples and the results for DKLFM is from the set of 256 examples? Moreover, regarding the computation times if they are averaged across all examples? Please if you could elaborate more on this? If the results for 2 methods are from the smaller set of examples than for the last method, it will not be a fair comparison.

Author response:
Question about dataset sizes and the results tables: Table 1’s caption is worded poorly. The MSE is averaged over the same 20 test instances for each model, including the DKLFM. Computation time is also the average across the 20 test instances. We have also clarified this caption in the rebuttal revision.

Reviewer comment (Novelty):
Comparing with other Meta-Learning methods and placing the DKLFM in a right place in this field (adding needed related works).

Author response:
Regarding Q1: we acknowledge your observation regarding the positioning of DKLFM in the context of meta-learning, and we would also like to thank you for sharing the meta-learning references. In response, we would like to clarify that our primary contribution is centered on latent force inference rather than meta-learning per se. Using meta-learning terminology,  the meta-parameters here are the task representations. The exact inference over these parameters is a key strength, since closed-form meta-losses enable gradient updates of the meta-parameters much more effectively. We also demonstrate how we perform far from the meta-train set. We believe this distinction and the specific advantages it brings to latent force inference are critical to understanding the contribution of our work.

Reviewer comment (Experiments):
Please, compare the DeepLFM with DKLFM in a similar regime to the one presented in Figure 6.

Author response:
Regarding Q3 and Q4: in our paper, we emphasize that DKLFM is set apart from DeepLFM in two fundamental aspects. Firstly, DKLFM is multi-task, whereas deepLFM only operates on a single instance. Secondly, DKLFM infers the latent forces in addition to the output functions. In contrast, DeepLFM only yields output functions. Without significant modification to the DeepLFM codebase, we would not be able to get DeepLFM to perform favorably in terms of time taken for a batch of instances, since it is currently set up for single-instance inference. The exact solutions deriving the training data are from Alfi.

Reviewer comment (Evaluation):
It will be really helpful if you will be able to add the comparison on other metric like NLL or ELBO.

Author response:
We are currently working on adding NLL and ELBO to the table of results.

Reviewer comment (Theory):
I have a question if in more complex examples the incorporation of the GPs is a good idea? I mean that the GPs are really powerful method, but it has the strong assumptions regarding the Normal distribution, which might reduce the model flexibility in a more complex (e.g., non-gaussian) distributions. It could be a good idea to incorporate the flow-based approaches which are able to map one distribution into another. Or just use the neural ODE. Out of the topic is the question what Alfi use as a particular ODE solver? Maybe it will be able to rewrite this setting into neural ODE.

Author response:
You correctly point out the inductive bias that GPs bring to a model. In our view, the utility and drawbacks of this are very much application-dependent. Incorporating differential equations is a similarly strong bias, which could be replaced with a neural ODE. However, this takes us back to the discussion around what level of dynamical bias is needed in a model. In some applications, such as biophysical models, where parameters have real meaning, this bias is useful. In addition, where we know that our latent forces should be smooth, then a GP is an appropriate method to use.