Paper: Two-shot learning of continuous interpolation using a conceptor-aided recurrent autoencoder (ICLR-2024)

Reviewer comment (Evaluation):
The proposed controlled recurrent auto-encoder allows to be able to generate continuous patterns evolution between two initial pattern of time series. However this is quite hard to assess the benefits of the proposed approach since no comparison to other technique is proposed nor any quality metrics. Only subjective evaluation is reported.

Author response:
Thank you for your critical feedback. We have made several amendments to our manuscript to address your concerns, particularly regarding the evaluation methodology.

We agree with  the need for objective evaluation metrics, and we have devised a quantitative metric in the revised manuscript. In this context, we would not like to use a third pattern to evaluate our method as the interpolated temporal patterns are based on probabilistic interpolation related to the training data. A third example of a parametric family could contain features that are absent or only weakly present in the training data and therefore can not be inferred by the CARAE mechanism. Using such third patterns for evaluating our method could lead to false understanding of what our method aims to achieve. To quantitatively measure the performance of the interpolated temporal patterns we have developed a similarity metric specifically tailored to our task. This metric provides a probabilistic measure and a temporal correlation measure of the generated patterns' fidelity to the training data, addressing the subjective nature of our initial evaluations. Further details on this metric, including its formulation and application, are now included in the appendix and referenced in the manuscript. Finally, we would also like to note that auto-encoders are often judged on their output re-generation with subjective judgments [1]. Motion modeling also [2] because one of their main goals is to develop plausible motion for simulated characters. 
To provide further clarity on the role of conceptor loss and its regularization term, we included in the appendix an ablation study of the regularization parameters beta_1 and beta_2. This addition aims to elucidate the identifiable effects of conceptor loss on controlled neural networks, also illustrating the applicability of the newly introduced evaluation metrics. Both regularization parameters show different influences for the two evaluated tasks. Thereby, we find that the usage of both is needed to significantly improve the generalization during the interpolation. 

In summary, these revisions and additions address your concerns by enhancing the objective evaluation of our approach and justifying our methodological choices. We believe these changes strengthen the manuscript and look forward to your further feedback.

Reviewer comment (Presentation):
References to equation and figures needs to be revisited since when cited it is often omitted to mention if it refers to a table, image or other.

Author response:
Minor remarks:

We adapted the references in the revised manuscript making it more clear to which figure/table/equation we refer.

Reviewer comment (Experiments):
As quickly mentioned in footnote of page 6, SPD matrices are to be associated to a specific manifold. So linear interpolation used in equation 10 could be discutable. But also the distance metric used in equation 8. It would be interesting to consider also here some classical metrics and interpolation technique of SPD matrices to see the impact of distance metric and interpolation technique there. See also [1,2] for additional paper on SPD matrices metrics.

Author response:
In response to your remarks on SPD matrices, we would like to discuss our choice of the linear interpolation. We also discussed alternative metrics like log-euclidean and interpolation methods for SPD matrices, as mentioned in the referenced literature. However despite its simplicity, the linear interpolation approach preserves the SPD matrix properties [3], ensuring that the interpolated conceptors are still valid soft projection matrices. Furthermore, we decided to focus on linear interpolation due to its computational efficiency. Other metrics include the usage of matrix logarithm and exponential that are computationally expensive for the forward and backward-pass and could potentially introduce instabilities in the training.  Furthermore, these functions are often not implemented in standard deep learning libraries making their usage more complicated. Nevertheless, as it is computationally more feasible, we tested the usage of the log-euclidean and the shape-and-orientation (SAO) metric during the inference while relying on the linear notion for the training. The log-Euclidean and SAO metrics are developed to tackle known challenges in statistical operations involving semi-definite matrices. One such problem is the "flattening effects" [4]. This refers to the deformation and shrinkage of singular values that occur during linear interpolation. However, in our study, we found that these metrics did not significantly impact the inference process. Whereas this might be related to the particular task at hand we agree and mention in the paper that a more rigorous investigation of the influence of the metric might further improve our results. Due to the limited time, we started this investigation of the metrics for inference only and attached an initial discussion to the appendix as section F.

Reviewer comment (Experiments):
No discussion on the impact of $\beta_1, \beta_2$ parameter is provided. What about for exemple using only pattern proximity (e.g. $\beta_1=0$) ?

Author response:
To provide further clarity on the role of conceptor loss and its regularization term, we included in the appendix an ablation study of the regularization parameters beta_1 and beta_2. This addition aims to elucidate the identifiable effects of conceptor loss on controlled neural networks, also illustrating the applicability of the newly introduced evaluation metrics. Both regularization parameters show different influences for the two evaluated tasks. Thereby, we find that the usage of both is needed to significantly improve the generalization during the interpolation.

Reviewer comment (Evaluation):
Could there be objective metric to evaluate the performance of proposed approach? For example what about having a 3rd pattern to be used for evaluation?

Author response:
We agree with  the need for objective evaluation metrics, and we have devised a quantitative metric in the revised manuscript. In this context, we would not like to use a third pattern to evaluate our method as the interpolated temporal patterns are based on probabilistic interpolation related to the training data. A third example of a parametric family could contain features that are absent or only weakly present in the training data and therefore can not be inferred by the CARAE mechanism. Using such third patterns for evaluating our method could lead to false understanding of what our method aims to achieve. To quantitatively measure the performance of the interpolated temporal patterns we have developed a similarity metric specifically tailored to our task. This metric provides a probabilistic measure and a temporal correlation measure of the generated patterns' fidelity to the training data, addressing the subjective nature of our initial evaluations. Further details on this metric, including its formulation and application, are now included in the appendix and referenced in the manuscript. Finally, we would also like to note that auto-encoders are often judged on their output re-generation with subjective judgments [1]. Motion modeling also [2] because one of their main goals is to develop plausible motion for simulated characters.

Reviewer comment (Experiments):
What about the impact of $\beta_1, \beta_2$ hyper-parameters? Could there be a specific ablation study?

Author response:
To provide further clarity on the role of conceptor loss and its regularization term, we included in the appendix an ablation study of the regularization parameters beta_1 and beta_2. This addition aims to elucidate the identifiable effects of conceptor loss on controlled neural networks, also illustrating the applicability of the newly introduced evaluation metrics. Both regularization parameters show different influences for the two evaluated tasks. Thereby, we find that the usage of both is needed to significantly improve the generalization during the interpolation.

Reviewer comment (Experiments):
How does the proposed technique performs with respect to other approaches?

Author response:
We have now addressed your suggestion for quantitative evaluation using a third pattern to benchmark the performance of our CARAE framework, taking inspiration from the MIOFlow paper that you added as reference.

In our revised approach, detailed in Appendix I, we compared the performance of CARAE against a baseline method of simple linear interpolation between two distinct temporal motion patterns: walking and running. Our goal was to generate an intermediate time series that would ideally represent a jogging motion (the “correct” intermediate between walking and running from the dataset), and then measure the accuracy of these generated patterns against the ground truth jogging data available in our dataset.

We used MSE and JSTCC the as the metric for quantification. The results, as presented in Appendix I, demonstrate that CARAE achieves a lower MSE in comparison to the direct linear interpolation approach when evaluated against the ground truth jogging data. This finding substantiates the efficacy of CARAE in generating more accurate intermediate temporal patterns and also provides a clear, objective measure of its performance. This establishes a tangible benchmark for future research, offering a solid foundation upon which to evaluate, measure, and enhance our proposed framework, task, and solution.

We believe that this additional experiment and its results effectively address your concerns regarding the lack of quantitative evaluation and comparative analysis in our initial submission. The ability of CARAE to outperform a straightforward linear interpolation method in this context highlights its potential as a more nuanced and effective tool for interpolating between temporal motion patterns.

Reviewer comment (Novelty):
Outdated References: The reference to Jaeger (2014) being eight years old may indicate that the paper doesn't incorporate more recent developments in the field. In rapidly evolving fields like machine learning, research can become outdated relatively quickly.

Author response:
We thank you for your feedback, which we would like to address as follows

1. Acknowledging the Relevance of Older References:  
We would like to point out that the age of a reference does not undermine its validity. In fact, older concepts often find renewed significance in contemporary research. Additionally, linear subspace characterization linear subspace control did have recent successes by being effectively combined with backpropagation to address catastrophic forgetting issues, as highlighted in recent studies [1], [2]. Also, the concept of low-dimensional dynamics has recently gained traction in computational neuroscience, serving as a foundation for rich theoretical and experimental work to understand the dynamical aspect of the brain [3]. Our research proposes a novel interpretation of these phenomena by connecting it to the rich field of learning representation and autoencoding in Deep Learning.

2. Differentiating from Jaeger’s Work and Aligning with Current SOTA Research:  
Our approach represents a significant departure from Jaeger’s initial work [4], where linear subspace control was not linked to learning with RNNs but to store memories akin to Hopfield Networks but for time series. Instead, our methodology is more closely aligned with contemporary SOTA research in meta-learning [5], where some meta-parameters (equivalent of the conceptor) are multiplicatively interacting with a base model to improve the learning data-efficiency. These recent studies recognize and utilize the potential of linear subspace control in developing advanced learning models.

Reviewer comment (Experiments):
Why use RNN rather than a more recent and powerful transformer architecture?

Author response:
We thank you for your feedback, which we would like to address as follows

3. Emphasizing the Role of RNNs in Deep Learning:  
In the realm of deep learning, it's crucial to note that RNNs stand on par with other architectures like transformers on data-sparse domains. This includes the distillation and compression of locomotion policies into multi-task policy networks [6, 7]. Finding good compression principles is recognized as promising to be more data-efficient. Moreover, various RNN architectures are increasingly challenging transformers in terms of reducing computational costs and offering unbounded context length [7]. Additionally, some RNNs architectures have been shown to be very close to attention-based computations  [8].

Reviewer comment (Experiments):
Why is there not a comparison with related works or a baseline?

Author response:
We thank you for your feedback, which we would like to address as follows

4. Extending the CARAE Architecture:  
As mentioned in our responses to other reviewers, we have expanded our work in the appendix. Here, we demonstrate that the CARAE (Conceptor-Aided Recurrent AutoEncoder) architecture can be extended to scenarios involving more data, higher-dimensional bottlenecks, and other commonly used architectures like the LSTM. Furthermore, we adapted the CARAE to static Feed Forward Networks (FFWs) like MLPs applied to time series prediction, theoretically allowing for application to transformers.

Reviewer comment (Novelty):
The major weakness of the work is a lack of justification for its practical significance, which makes it difficult to judge the work's contribution.

Author response:
We would like to thank you for engaging with us. However, we challenge the assumption that our novel representational learning algorithm lacks practical importance. Contrary to your references, our work focuses on the smooth interpolation between learned motions, not just generating motion patterns from sequences. This is not the main aim of the work we present here. Other works, like [1], focus on learning velocity control in human motion using reinforcement learning. However, these approaches, lacking a robust representational learning mechanism, often require extensive training and computation, often resulting in unnatural motion patterns. In contrast, our work introduces a method for learning continuously interpolatable abstract representations using a small RNN trained on sparse data. In the following, we will address the raised points more in detail:

**First method that shows interpolatable generalization relying on two-shot learning**: We focussed on the two-shot learning setting to showcase the unique generalization capability. Thereby, we present that our framework outperforms other e.g. classical recurrent auto-encoders that are notoriously hard to train and require a lot of data. Thereby we are less constraining, as there is no need to encode all the input into a low-dim vector space as done e.g. in transformer-based architectures. Furthermore, in the last revision, we introduced how our setup is extendable to few shot learning which further extends the applicability of our method. 

**Relevance for applications in RL**: Two-shot and few-shot learning are central challenges in machine learning. For instance, the recent resurgence of meta-learning is motivated by the drastic difference between humans and machines in terms of sample efficiency (cf. omniglot [5], finn). In RL, this need is more critical, as current algorithms still lag behind human-level sample efficiency. Addressing the reviewer’s question about related works, we would highlight the work of Wayne et al[2], Mocapact[1], and more recently Gehring[2]. These start from a few policies learned by imitation (walking, running, jogging) and aim to distill them into a single neural network to accelerate further reinforcement learning on downstream tasks like following a speed target (cf. Figure). Despite showing minor improvements like faster subsequent learning and better pose, they still rely on significantly more training to follow intermediary speed targets. In our work, we accomplished a more complex task by successfully enabling the model to follow intermediary speed targets without requiring additional training (cf. video in Supplementary Materials), a development that shows a promising direction for reinforcement learning applications.

Reviewer comment (Novelty):
It is hard for me to judge the practical significance of the problem settings of the work, learning a continuous spectrum of representations that can interpolate between two training examples. The work mentioned related works of interpolatable representation learning in different areas, including locomotion modelling, robotics, and reinforcement learning. None of the mentioned work studies problems settings even close to the proposed setting. The proposed setting also restricts the approach to a two-shot learning setting and it is not clear how the approach can be extended to few-shot learning settings with more training examples and if the proposed approach would still have any practical value when larger amount of training data is available, which is not unusual in practice.

Author response:
1. 1.  We mention related works of interpolatable representation learning in different areas, including locomotion modeling, robotics, and reinforcement learning because they deal with certain challenges, like sparsity of data while aiming for stable periodic solutions that can be interpolated. We would like to argue that the CARAE framework, although not yet applied to RL learning tasks, shows promising results in the motion modeling using the MoCap data set. Thereby, we address and overcome several of the discussed challenges that yet could only be solved by massive computational effort.

    2.  While our current focus is on two-shot learning, we see the potential for expanding CARAE to few-shot learning scenarios. Thereby, the loss function needs to be reformulated to include several conceptors, one per learned temporal pattern. The CARAE framework even enables us to encode into the loss function whether the patterns should be all-to-all interpolated or whether they should line up along a single dimension. We briefly outline potential extensions of our method in the manuscript's conclusion and in appendix E, discuss how CARAE could adapt to and benefit from additional training examples, thereby enhancing its practicality and applicability in a broader range of data-constrained environments.

Reviewer comment (Experiments):
The proposed CARAE is based on an artificial RNN model with specific architecture designs, which is different from more commonly used RNN architectures like LSTM or GRU. Extending CARAE to these more common RNN architectures or more recent transformer architecture could significantly improve its practical value.

Author response:
2. We appreciate your concern about extending CARAE to more commonly used RNN architectures like LSTM and to transformer architectures. As CARAE is an architecture-independent mechanism for manipulating neural dynamics through conceptors, we did not expect any problems with this. Indeed, in response to your suggestion, we have extended CARAE to the commonly used LSTM architecture. Furthermore, we have outlined a version of the CARAE that relies on a feed-forward neural net. Both architectures are able to learn continuous interpolation of temporal patterns as we present in the appendix C. These extensions demonstrate CARAE’s versatility and significantly broaden CARAE's practical applicability.

Reviewer comment (Writing):
In Sec. 5, the work uses *two data sets* and *two distinct examples* to call \( u_1 \) and \( u_2 \) which can be confusing. It would be much more clear if \( u_1 \) and \( u_2 \) are referred to as two training examples consistently in the work.

Author response:
Minor weaknesses: 

1. We thank  you for the careful reading and have streamlined the formulation in section 5 to avoid any confusion.

Reviewer comment (Evaluation):
The work only presents qualitative results and training loss curves. The lack of a systematic quantitive evaluations makes the proposed approach and claimed contributions less convincing.

Author response:
Minor weaknesses: 

2. In the appendix, we added the introduction of a similarity measure (see revised manuscript) that evaluates the relation of our inferences to probabilistic features and temporal correlations of the training datasets. Based on this metric, we performed an ablation study of the two regularization parameters beta_1 and beta_2. The metric thereby adds a quantitative measure to evaluate the inferences beyond the loss terms we design, allowing for a more systematic investigation of our proposed method.

Reviewer comment (Writing):
In Sec. 2, the work identifies 4 challenges in the ability of RNN to generalize to different temporal dynamics. **Inferences** is explicated tackled with in Sec. 3. Does CARAE address the other three challenges? If so, how are they addressed?

Author response:
Regarding further questions:

The CARAE framework effectively addresses all four challenges outlined in Section 2. The incorporation of the conceptor in the autoregressive mode ensures stable dynamics, protecting against exploding dynamics, interferences, and side dynamics. Our explicit training approach within the CARAE loss term thereby specifically tackles fixed-point dynamics, a critical aspect when aiming for interpolation. These improvements are needed to yield stable interpolated temporal patterns when dealing with sparse data as showcased by the two experiments discussed in the manuscript. Additionally to the failing vanilla RNN used in the sine wave interpolation, we added unsuccessful experiments using a RAE on both training sets to the appendix.