Paper: Interpretable Meta-Learning of Physical Systems (ICLR-2024)

Reviewer comment (Novelty):
**The proposed method lacks novelty and are not situated in the right context**. The authors claim to propose a novel “model agnostic meta-learning architecture." However, the proposed approach is not model agnostic (because it requires specifying the width of the last layer dimension) and more importantly is not novel and not situated in the most relevant context.

Author response:
We first address the main concerns regarding the novelty and the context of our approach. The rest of the questions is addressed in a second message, along with an additional experiment that we conducted for further comparison with existing work.

## Novelty of our method

We agree that the representation learning architecture with a shared feature map and task-specific linear weights is not novel. We will make it clearer by better referencing (Caruana, 1997) and adding the references below. Our contribution demonstrates the effectiveness of this representation learning architecture in a challenging setting with very few tasks, each tasks containing a large number of points. This framework is motivated by the interpretable learning of physical systems. 

In (Wang *et al.*, 2021), representation-based multi-task learning is compared with gradient-based meta-learning algorithms such as MAML and ANIL. As in our paper, both architectures are presented as methods for generalizing to new tasks, by minimizing a regularized multi-task loss function. This work theoretically demonstrates that representation learning can achieve performances comparable to those of meta-learning algorithms, within the limit of large neural networks. These theoretical conclusions are in line with the practical conclusions of our paper in the context of physical systems, which however have a different framework and scope.

While the scope of these works is indeed similar to ours, we would like to point out some important differences. Our approach lies in the context of learning physical systems, where data collection is costly and computational resources can be limited. Typically, the number of training tasks is small (the number of tasks $T$ can be smaller than $4$ for example, as in our experiments), and the neural networks used are limited in width and depth to avoid overfitting. Consequently, the theoretical guarantees of (Maurer *et al.*, 2016), Theorem 2, and (Wang *et al.*, 2021), Theorem 1, where the number of tasks and the network depth tend to infinity, do not apply to our use cases. We also point out that we focus on regression tasks, while the experiments carried out in (Wang *et al.*, 2021) focus on classification. Furthermore, in the few-shot image classification experiments in the latter work, $T$ is very large ( $\binom{5}{64} \simeq 7.10^6$) and the number of training points per task is very small (typically smaller than 10), as opposed to our setting where $T$ is small and the number of training points $N_t$ is not restricted ($N_t$ of order 10000 for the training grid in the capacitor experiment).

Unlike image classification, we show that the linear heads in the representation learning architecture can exploit the structure of the problem to provide a low-cost, high-performance algorithm for learning an interpretable physical model. In image classification, the algorithm has access to a very large database, which compensates for the lack of structure in the learning problem. In our submission, we show that, in a very different framework with less data and less compute, complex functions can still be learned efficiently with an interpretable model by exploiting the linear structure of the problem. Hence, like (Wang *et al.*, 2021), we demonstrate that representation learning can match more complex meta-learning architectures, but our study is carried out in a different setting, with different motivations and scope, and with the crucial benefit of identifying the underlying physical parameters.

Reviewer comment (Novelty):
**Novelty**: It could be argued that the proposed method is not meta-learning but applying multi-task representation learning in a learning-to-learn problem. A more general version of the authors’ proposed approach have been theoretically studied in terms of its generalization performance by Maurer et al in 2016 (page 7, section 2.2 Bounding the Excess Risk for Learning-to-learn). More recently, Wang et al 2021 have proposed almost exactly the same approach by first multi-task learning a shared feature extractor with linear heads and then performing last linear layer fine-tuning over a new task’s data at test time (section 3.4 Fine-tuning for Test Task Adaption). Besides, Wang et al has also studied the computational time savings of this MTL approach compared to last-layer MTL methods. Thus it is not correct for the authors to claim their methods as a novel approach.

Author response:
We first address the main concerns regarding the novelty and the context of our approach. The rest of the questions is addressed in a second message, along with an additional experiment that we conducted for further comparison with existing work.

## Novelty of our method

We agree that the representation learning architecture with a shared feature map and task-specific linear weights is not novel. We will make it clearer by better referencing (Caruana, 1997) and adding the references below. Our contribution demonstrates the effectiveness of this representation learning architecture in a challenging setting with very few tasks, each tasks containing a large number of points. This framework is motivated by the interpretable learning of physical systems. 

In (Wang *et al.*, 2021), representation-based multi-task learning is compared with gradient-based meta-learning algorithms such as MAML and ANIL. As in our paper, both architectures are presented as methods for generalizing to new tasks, by minimizing a regularized multi-task loss function. This work theoretically demonstrates that representation learning can achieve performances comparable to those of meta-learning algorithms, within the limit of large neural networks. These theoretical conclusions are in line with the practical conclusions of our paper in the context of physical systems, which however have a different framework and scope.

While the scope of these works is indeed similar to ours, we would like to point out some important differences. Our approach lies in the context of learning physical systems, where data collection is costly and computational resources can be limited. Typically, the number of training tasks is small (the number of tasks $T$ can be smaller than $4$ for example, as in our experiments), and the neural networks used are limited in width and depth to avoid overfitting. Consequently, the theoretical guarantees of (Maurer *et al.*, 2016), Theorem 2, and (Wang *et al.*, 2021), Theorem 1, where the number of tasks and the network depth tend to infinity, do not apply to our use cases. We also point out that we focus on regression tasks, while the experiments carried out in (Wang *et al.*, 2021) focus on classification. Furthermore, in the few-shot image classification experiments in the latter work, $T$ is very large ( $\binom{5}{64} \simeq 7.10^6$) and the number of training points per task is very small (typically smaller than 10), as opposed to our setting where $T$ is small and the number of training points $N_t$ is not restricted ($N_t$ of order 10000 for the training grid in the capacitor experiment).

Unlike image classification, we show that the linear heads in the representation learning architecture can exploit the structure of the problem to provide a low-cost, high-performance algorithm for learning an interpretable physical model. In image classification, the algorithm has access to a very large database, which compensates for the lack of structure in the learning problem. In our submission, we show that, in a very different framework with less data and less compute, complex functions can still be learned efficiently with an interpretable model by exploiting the linear structure of the problem. Hence, like (Wang *et al.*, 2021), we demonstrate that representation learning can match more complex meta-learning architectures, but our study is carried out in a different setting, with different motivations and scope, and with the crucial benefit of identifying the underlying physical parameters.

Reviewer comment (Experiments):
**Context**: In the background section, the authors situate their methods among Gradient-based Meta-learning methods. However as the authors have shown in Table 1, CAMEL doesn’t perform gradient adaptation, so it’s not a gradient-based method. Instead, because CAMEL adapts only the last layer of the learned model, a more appropriate way to situate the current method is to compare it among last-layer meta-learning methods, which includes not only ANIL (which the authors have compared against) but more importantly MetaOptNet (Lee et al 2019) and R2D2 (Bertinetto et al., 2019). Here both MetaOptNet and R2D2 solve the last linear layer exactly (using a convex solver or closed-form expression) and use implicit function theorem to propagate the gradients to the earlier feature extractors. Because this paper focuses on the regression task, the authors should compare additionally against R2D2 in the experiments. (One can think of R2D2 roughly as ANIL where the last layer adaptation are solved exactly instead of using a fixed number of gradient descent steps.)

Author response:
## Context

We thank you for the references and we will make sure to add them to the text body, and to better situate the context of our meta-learning algorithm.

The R2D2 algorithm indeed has in common with CAMEL the adaptation of the network head for each task. Using the formalism of our paper, R2D2 computes task specific weights $w_t = A(\pi, D_t)$ with $A(\pi, D_t)$ solving a Ridge regression of the $N_t$ points of $D_t$, for each training step and for each task. 

R2D2 should perform well for multi-task learning thanks to the complete resolution of the regression at each learning task. However, implementing it in our setting where $N_t$ is very large is not straightforward, as Ridge regression requires the inversion of a quadratic-sized system into the number of the features or of the regression points (which can be in the tens of thousands in our experiments as we mentioned above), hence adding a substantial amount of gradient computations. We have implemented this method and compared it to CAMEL in our experiments, and we report our results in a separate message.

Reviewer comment (Writing):
**Notation needs to be more consistent and clearer.** For example, both $f_t(x; \pi)$ (page 3) and $f_t(x)$ (page 4) are used. Besides, the ground truth function is denoted $v_{\star}$. However, it has $n$ output dimensions, while the function $v$ has $r$ output dimensions.

Author response:
We now answer the other questions. We thank you for pointing out notation and clarity issues; we will fix them in an updated version of the submission.

Reviewer comment (Experiments):
How many steps of gradient adaption is used for ANIL? It is important to note that one can apply more steps of adaptation during evaluation than training as it can potentially improve ANIL's performance.

Author response:
``How many steps of gradient adaption is used for ANIL? ``

In the training inner loop, we used only one adaptation step. Adding more might make the model more expressive, but the computational cost would be even larger. At inference time, we have used a number of steps varying between 10 and 50, depending on the experiment. We chose the number of adaptation steps (and the stepsize) carefully so that the adaptation loss converged to is minimal value as in (2.5).

Reviewer comment (Writing):
Can the authors explain for example 1 and 4, what is the learning goal here? I saw the authors mention $x=(q, \dot q, \ddot q)$ and $y=u$, but can the authors explain why do we want to predict $u$ (which I'm assuming is a function of time) in the first place?

Author response:
``Can the authors explain for example 1 and 4, what is the learning goal here?``

We indeed want to predict the action $u$ as a function of the state, velocity and accelerations. The goal here is to track a reference trajectory, hence we are given the target state and its derivatives. Since the problem is "what control law $u$ should be input to the system to reach the desired trajectory?", a model that predicts $u$ as a function of the trajectory information can be used in combination with proportional controllers to design a feedback controller for trajectory tracking, which is called inverse dynamics control. This type of control is very common in robotics. More information can be found in Appendix B.2 of our submission, and in (Spong *et al.*, 2020).

Reviewer comment (Experiments):
Every experimental meta-learning setting in this paper seems to be newt. This would have been helpful to help better-ground the results in the community, as otherwise it is difficult to understand if the performance improvements are coming from improper implementations of other methods or from a fundamental modeling advancement. For example, MAML with a training step size of 30 and adaptation size of 10 in Table 2 is very arbitrary and unlikely to be the best MAML hyper-parameters here. The paper would be significantly more convincing if the CAMEL model was evaluated in the exact experimental settings from CoDA or [Wang et al., NeurIPS 2022](https://arxiv.org/abs/2102.10271), which also meta-learn physical systems with varying amounts of contextual information.

Author response:
## 1. Experimental comparison

### Experimental setting

In our paper, we have compared CAMEL with a number of baselines, including gradient-based meta-learning algorithms MAML and ANIL and the multi-environment context-informed network CoDA. 
We agree that our experimental conditions are not exactly the same as those of (Wang *et al.*, 2022) and (Kirchmeyer *et al.*, 2022). This is because **we do not address the same learning problem**. These papers focus on the problem of learning a function $f_\star$ representing a dynamical system by fitting the trajectories of the system. As a consequence, the training loss function is expressed in terms of the trajectory obtained by integration of the model, typically using Neural ODEs (Chen *et al.*, 2018). Although trajectory prediction is very rich and interesting, the trajectory integration step adds up a layer of complexity to the multi-task learning problem.

We, on the other hand, chose to focus on the more fundamental problem where $f_\star$ is learned by regression. This is not only motivated by the fact that many physical systems are static (for example, the electrostatic field falls within this category, and could not be learned with a trajectory prediction approach), it is also the most general and natural framework for assessing the learning performances (including its expressivity and its computational cost) of a given architecture for learning a target function $f_\star$, regardless of the additional problem of integrating the dynamics into a trajectory.

Therefore, we decided to run independent experiments on a number of systems representing various settings of regression, for which we compared our approach as fairly as possible to the other existing architecture, which we have reimplemented for static regression and carefully tuned. As we point out in the conclusion (Section 7), CAMEL can be extended to the setting of trajectory prediction for which identification of the physical parameters is also an important question. We are currently working on this extension and the results should be ready within weeks, in which case we can add them to an updated version of the paper.


### Choice of hyperparameters

While we agree that comparing algorithms on different initial conditions may induce a bias related to the choice of hyperparameters, we would like to make it clear that we have conducted our experiments as fairly as possible. Hyperparameters such as the learning rates have been chosen on a test set in such a way as to give the best possible performance, for all architectures. 


**Additional experiment**


 We present a new in-domain evaluation experiment showing that our choice of hyperparameters give similar performances for MAML and CAMEL in in-domain evaluation. The algorithms trained on multi-task data are evaluated **not on a new out-of-domain task, but on the training tasks**, albeit with different data points from the training set. 

Evaluation mean squared error
|experiment|in-domain |out-of-domain (results presented in the submission)|
|---|---|---|
|MAML |8.1e-5 ± 2e-5|1.6e-1 ± 5e-2|
|CAMEL|3.2e-5 ± 1e-5|1.0e-4 ± 5e-5|


This experiment demonstrates that in a case more favorable to MAML's meta-learning, its performance is much closer to that of CAMEL and CoDA.


We would also like to point out that the quantities reported on the right in Table 2 are not gradient step sizes, they are computation times, for training and during inference (a step size of order 30 or 10 would indeed be probably bad).

Reviewer comment (Theory):
The term "interpretable meta-learning" in the title and throughout the rest of the paper is a big statement without the right qualifications. The paper gets to defining what interpretability means here in proposition 1 of recovering true parameters when they exist and fit into the modeling structure they have set up, but I this is not what I had in mind when I first saw the term "interpretability". It also seems like there is still a large amount of uninterpretability in the system, as the model in equation 3.2 still has 2 neural networks.

Author response:
## 3. Interpretability

We agree that "interpretability" is a big word, and that our approach provides only a partial answer to the question of model interpretability.  It is, however, a first step towards this goal, since our method allows the physical parameters of systems that vary from task to task to be identified, with applications to zero-shot adaptation and to online system identification in robotics for example. 

Identifying all the parameters of the learning model seems out of reach when using deep neural networks. An interesting and realistic research direction would be the use of invertible neural networks in $v(x;\theta)$, so as to identify not only the physical parameters but also the feature map shared across tasks.

If necessary, we propose to change the title of our submission to "Towards Interpretable Meta-Learning of Physical Systems".

Reviewer comment (Reproducibility):
The experimental results rely heavily on Table 2; however, the experimental conditions for these results are not made clear. Specifically, the training details for MAML, ANIL, CoDA, and CAMEL are not fully detailed. The "Training" and "Adaptation" columns are not described, nor are their units present.

Author response:
Our experimental results indeed lack some details. The number of experimental trials for evaluation is mentioned in the text body for Section 5.1 and in B.1.2 for the capacitor, but we have omitted to specify it for the other experiments. The standard deviations are reported in Table 3, in Appendix B.4. We shall update this section with the number of trials, and also report the latter in the legend Table 2. Regarding the computational time, we shall include additional details regarding the protocol that we used and the statistics of the measured time. Once measured and averaged, the unit was chosen arbitrarily as the lowest measured time (in this case, that of CAMEL) for a clearer comparison. The time for training CAMEL for 10000 gradient steps on a single CPU in the capacitor experiment was typically of the order of the minute.

Reviewer comment (Theory):
The argument that CAMEL is an interpretable method relies on the idea that the weight vector \( w_t \) learned on the training meta-dataset can correspond to physical parameters of a system, as explained in Proposition 1, which assumes that the training loss is taken to 0. However, the argument that CAMEL allows for such interpretation is not sufficiently demonstrated.

Author response:
In the case of the point charge system, we demonstrate the interpretability of the learned parameter by computing the identification error, *i.e.* by showing how close to the physical parameters (in this case the electric charges) the learned parameters are, which relies on CAMEL's ability to learn the system. We would like to point out that we further demonstrate the interpretability of the learned model on two major and novel applications. 

Firstly, the interpretation of the learned contexts allows for a zero-shot adaptation scenario (in the last paragraph of Section 4.), where the adapted model uses information about the physical parameters rather than measurements of the target function. This application is presented in both the charge experiment and in the capacitor experiment.

Secondly, for robotic systems, the interpretability of the learned task parameters allows for the identification of the robot's inertial parameters, which is valuable in adaptive control. This is illustrated in Section 5.2.

Reviewer comment (Evaluation):
The example of learning an electric point charge system in section 5.1 and expanded in section B.5 relies heavily on the fact that CAMEL is able to learn the dynamics of the system, and not on interpreting the resultant parameters. Demonstrating how the learned physical parameters allow for interpretation of the model and physical system, beyond CAMEL’s performance, would strengthen this argument.

Author response:
In the case of the point charge system, we demonstrate the interpretability of the learned parameter by computing the identification error, *i.e.* by showing how close to the physical parameters (in this case the electric charges) the learned parameters are, which relies on CAMEL's ability to learn the system. We would like to point out that we further demonstrate the interpretability of the learned model on two major and novel applications. 

Firstly, the interpretation of the learned contexts allows for a zero-shot adaptation scenario (in the last paragraph of Section 4.), where the adapted model uses information about the physical parameters rather than measurements of the target function. This application is presented in both the charge experiment and in the capacitor experiment.

Secondly, for robotic systems, the interpretability of the learned task parameters allows for the identification of the robot's inertial parameters, which is valuable in adaptive control. This is illustrated in Section 5.2.

Reviewer comment (Novelty):
Including a discussion of Physics Informed Neural Networks (PINNs), which share the same motivation and have been used to present interpretable results, would further improve the interpretability argument.

Author response:
We will make sure to mention physics informed neural networks (Raissi *et al.*, 2019), for which interpretability is ensured by learning the solution of a PDE, while enforcing the physical constraints directly in the loss function.

Reviewer comment (Theory):
The authors appear to only study the case in which \( c = c_* = 0 \), which is a large simplification. If Equation 3.1 is valid, this means \( h(x;\theta_0) = 0 \), which is a notable departure from other meta-learning methods. The authors should further justify this simplification and potentially explore the case where \( c \neq 0 \).

Author response:
Regarding the case where $c, c_\star \neq 0$, it can be handled by augmenting  $\varphi$ and $v_{\star}$, and $w$ and $v_\star$ with an additional dimension, with the corresponding component of $v$ and $v_\star$ equal to $1$. The conclusion of Proposition 1 then applies wit the assumption of $\{v_\star(x^{(i)})\}$ spanning $\mathbb{R}^{n+1}$. Hence, the augmented physical parameters can be recovered up to a linear transform, meaning that $\varphi$ can be recovered with an affine transform. We have tackled the interpretability in the case $c, c_\star \neq 0$ experimentally in the capacitor system, where $c_\star$ is non zero a fortiori because the electrostatic field is linearized around a nonzero value, as is pointed out in the review. 

We will clarify this explanation in a revised version of the submission.

Reviewer comment (Presentation):
There is a minor formatting issue in which works are cited in text when they should be cited in parentheses, although some citations are correct. For instance, in a paragraph in section 4, the Ljung citation is correct, but the Nelles citation is not:  
"System identification and model identifiability are key issues when learning a system (Ljung, 1998). Although deep neural networks are becoming increasingly popular for modeling physical systems, their complex structure makes them impractical for parameter identification in general Nelles (2001)."

Author response:
We thank you for pointing out these two presentation weaknesses. We will correct them in an updated version of the submission.