Paper: STanHop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Prediction (ICLR-2024)

Reviewer comment (Writing):
Explanations for Lemma 3.1, 3.2, and other lemmas in the main paper, can be written in a better way. Rather than stating to refer to proof, you should try and provide the simplest explanation of what each proof talks about.

Author response:
Thanks for asking the following clarification questions. In response, we have revised the draft to enhance clarity and address the points raised as follows.

>Finally, what is the memory footprint? 

In the **newly added Sec. E.5**, we report the footprint of memory and GPU usage. The results show that the difference in memory and GPU usage between the proposed GSH and the vanila **dense** modern Hopfield layers is **negligible**. Please see Fig. 10 and Fig. 11 for more details.

>How much time model takes per epoch? 

The time per epoch varies depending on the specific setting.

As a reference, when using a batch size of 32 and an input length of 168, STanHop-Net requires approximately 2 minutes per epoch for the ETTh1 dataset. This duration is roughly equivalent to that of using dense modern Hopfield layers, such as STanHop-Net (D). Together with our results from Thm. 3.1 and Sec. E.1 — the tighter fixed point convergence of the Generalized Sparse modern Hopfield (GSH) model leads to fewer epochs required for convergence  — **STanHop-Net using the GSH model generally requires less time to converge** (than those of using dense Hopfield layers.) 

>Model size? 

The total number of parameters of the STanHop-Net is ~0.78 million. We have added this in the **new Sec. E.6** of the latest revision.

>Do you observe faster convergence? 

The tighter fixed point convergence of the Generalized Sparse modern Hopfield (GSH) model leads to fewer epochs required for convergence. Thus, yes, our empirical results (Fig. 4 in Sec. E.1) verifies our theoretical results (Thm. 3.1) --- the GSH model converges faster than the dense modern Hopfield model.

>I would like to see the variance of the model, including baseline models.

In Multivariate Time Series (MTS) prediction research, it is common to exclude the variance due to compactness of papers.

This convention is observed in all our baseline models — including Crossformer, Informer, FEDformer, DLinear, and Autoformer — none of which report variance in their respective papers.  As most of our baseline values are directly quoted from these original papers, we adhere to the same convention for consistency.

However, **we did report the maximum variance across all settings of our model in our table caption, stating “with variance omitted as they are all <=0.44%”.**

>Explanations for Lemma 3.1, 3.2, and other lemmas in the main paper, can be written in a better way. Rather than stating to refer to proof, you should try and provide the simplest explanation of what each proof talks about.

Thanks for pointing this out. We apologize for any confusion caused. In earlier drafts, proof sketches were included in the main text, but they were mostly removed or relocated to the appendix due to page constraints. We have now added the removed ones back in Sec.C. Please see revised version for details.

To clarify:

- **Lemma 3.1**: The high-level proof sketch begins in Section C.1, followed by detailed elaboration.
The explanation/intuition/purpose are outlined directly below Lemma 3.1: 
>Lemma 3.1 demonstrates how the energy function (3.2) aligns with the overlap-function construction of Hopfield models, referenced in Hu et al. (2023) and Ramsauer et al. (2020). The corresponding retrieval dynamics satisfying the monotonicity property (T1) are then introduced. 

- **Lemma 3.2**: The proof sketch for Lemma 3.2 is presented at the start of Section C.1, distinctly marked by a gray block for emphasis and ease of reference.
>Our proof is built on (Hu et al., 2023, Lemma 2.1). We first derive T by utilizing Lemma 3.1 and Remark G.1, along with the convex-concave procedure (Yuille and Rangarajan, 2003; 2001). Then, we show the monotonicity of minimizing H with T by constructing an iterative upper bound of H which is convex in x t+1 and thus, can be lowered iteratively by the convex-concave procedure.
- **Lemma 3.3**: he proof sketch for Lemma 3.2 is presented as the first sentence of the proof.
>Since the energy function H monotonically decreases with respect to increasing t in Lemma 3.2, we can follow [(Hu et al., 2023), Lemma 2.2] to guarantee the convergence property of T by checking the necessary conditions of Zangwill’s global convergence.
- **Lemma 3.4**: the proof sketch is inside and highlighted by the gray blcok:
>Our proof, built on (Hu et al., 2023, Lemma 2.1), proceeds in 3 steps:...

We hope these changes and clarifications enhance the clarity of explanations for our theoretical results.

We hope that our responses adequately address the reviewer's concerns, and we look forward to any further feedback. Thank you for your time and consideration.

Reviewer comment (Experiments):
Improving the memory capability of Hopfield network is widely studied these days [7-9], thus comparison should be done with these relevant approaches, especially 8 and 9.

Author response:
We appreciate the reviewer for directing our attention to these relevant references. Please see below added discussions.

>Second, improving the memory capability of Hopefield network is widely studied these days [7-9], thus comparison should be done with these relevant approaches, especially 8 and 9.

**The major difference between our work and that presented in [8] lies in the scope of theoretical assumptions.** Our analysis avoids overly strong assumptions on the model and data, contrasting with the focus on combinatorial problems with strong assumptions in [8]. While both studies demonstrate exponential memory capacity relative to pattern size, the practical relevance of these findings differs significantly. **Our results hold greater practical applicability** in areas we prioritize, such as developing new methodologies and theoretical insights for attention/transformer models and large foundation models. However, this approach comes with a trade-off: our analysis may not provide as detailed characterizations as those found in [8], such as the derivation of Shannon’s bound for low-density error-correcting codes.

In contrast to [7], which offers a unified framework for associative memory models by dissecting retrieval dynamics into three components: similarity, separation (distinct from our paper's definition), and projection, our work takes a different approach. [7] introduces the Manhattan distance as an alternative similarity metric and highlights its empirical relevance. However, while they connect their framework to [Kortov and Hopfield, 2020], their theoretical exploration, particularly in terms of retrieval error, robustness, and memory capacity, remains limited. **Our work addresses this gap by providing a comprehensive theoretical understanding of a range of separation functions, employing the Euclidean distance as the similarity metric.**

In [9], the authors introduce the Partial Forgetting Function as a novel projection function within the frameworks of [8] and [Kortov and Hopfield, 2020]. Their approach, which includes the development of the Partially Forgetting Attention mechanism, effectively addresses the phenomenon of partial forgetting in the human brain. Similar to the above discussion, our research complements [9] by investigating theoretical properties. **Although our study does not explore changes in the projection map — a modification that might intuitively lead to efficient sparse modern Hopfield models, as suggested in [Beltagy, 2020] — we recognize this as a compelling and significant direction for future research.**

We hope these discussions have addressed your concerns. Thank you!

---
- [Kortov and Hopfield, 2020] Krotov, Dmitry, and John Hopfield. "Large associative memory problem in neurobiology and machine learning." arXiv preprint arXiv:2008.06996 (2020).
- [Beltagy, 2020] Beltagy, Iz, Matthew E. Peters, and Arman Cohan. "Longformer: The long-document transformer." arXiv preprint arXiv:2004.05150 (2020).

Reviewer comment (Experiments):
I would like to see a comparison against RNNs and memory-augmented RNNs, given that the proposed model is focused on time-series which is a stateful problem.

Author response:
>I would like to see a comparison against RNNs and memory-augmented RNNs, given that the proposed model is focused on time-series which is a stateful problem.

Our paper proposes a multivariate time series prediction model. To show the **prediction performance** of our model, we believe it is more reasonable and sufficient to compare our model’s prediction performance with the latest and best multivariate time series prediction models. Namely, **a newly proposed time series prediction model should always benchmark with SOTA baselines, regardless of whether the proposed model is stateful or stateless**. In light of this, we have conducted the comparison experiments and included the results in Table 1 in the original submission. The stateful/stateless models the reviewer mentioned refer to recurrent neural networks because they have the RNN-specific intern state. After a thorough literature review, we are not aware of any published stateful-RNN time series prediction model with top-tier prediction performance in major conferences or journals. Thus, we kindly ask the reviewer for specific suggestions on what additional literature/baseline should be considered.

On the other hand, the external memory module is only one component of our proposed method. Because there are other variables of the model affecting the performance, **directly comparing our model with other memory augmented models is not an apple-to-apple comparison** (as detailed below) and cannot provide a clear demonstration on the effectiveness of the external memory module. To more effectively showcase its impact, we believe an ablation study is both reasonable and sufficient. Thus, we have conducted this study and included the results in Table 2 of our original submission.

Here, we highlight the differences and advantages of our external memory design compared to previous similar approaches. To the best of our understanding, in previous works, their main focus is to learn an accessible/informative memory set from the training data. However, in our paper, we focused on allowing models to accept external stimuli during inference. Besides, the main novelty of our external memory design lies in utilizing the memory retrieval mechanism of Hopfield models. This design enables us to incorporate both feature-level and label-level auxiliary information, corresponding to the Plug-and-Play and Tune-and-Play modules, respectively. In contrast, previous works involving memory-augmented RNNs typically utilize an additional learnable weight matrix and incorporate auxiliary information only at the feature level. 

---

### **TL;DR** 

Directly comparing our model with other memory augmented models is not an apple-to-apple comparison.

**Different Focus:**
- Previous memory-augmented RNN methods emphasize learning to memorize specific data from the training set  [Kaiser, 2017;Santoro, 2016;Sukhbaatar, 2015;Graves, 2014].
- Our approach focuses on enabling models to incorporate external stimuli during inference for a performance boost via enhanced memory.

**Richer and Different Axillary Information from Memory:**
- Previous methods leverage auxiliary information within the feature space  [Kaiser, 2017;Santoro, 2016;Sukhbaatar, 2015;Graves, 2014; Weston, 2014].
- STanHop-Net's Tune-and-Play memory module retrieves pseudo-labels, incorporating both label-specific and feature space information to enhance predictions. 

**Different Training Requirements:**
- Previous works require extensive training of the memory module to access auxiliary information, making it challenging to modify for different tasks without retraining the entire system  [Kaiser, 2017;Santoro, 2016].
- STanHop-Net doesn't demand complete memory module retraining and aims to accept external stimuli, allowing for the inclusion of human-curated patterns.

**Module Flexibility:**
- Previous works are not compatible with plug-and-play memory integration, and the memory pattern is fixed after training.
- Our Plug-Memory module doesn't require training, facilitating easy integration of external memory. This indicates that our method tackles different challenges and possesses broader applicability compared to prior approaches.

---

Thank you for your time and valuable feedback. We hope that above clarifications address the reviewer's concerns.

---
- [Kaiser  2017]  Łukasz Kaiser, Ofir Nachum, Aurko Roy, Samy Bengio. “Learning to Remember Rare Events”
- [Santoro 2016] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, Timothy Lillicrap, “One-shot Learning with Memory-Augmented Neural Networks.”
- [Sukhbaatar 2015] Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, Rob Fergus, “End-To-End Memory Networks”
- [Graves 2014] Alex Graves, Greg Wayne, Ivo Danihelka, “Neural Turing Machines”
-  [Joulin 2015] Armand Joulin, Tomas Mikolov, “Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets”
- [Weston 2014] Jason Weston, Sumit Chopra, Antoine Bordes, “Memory Networks”

Reviewer comment (Evaluation):
What is the memory footprint? How much time model takes per epoch? Model size? Do you observe faster convergence? How stable is the model? All these questions should be addressed.

Author response:
Thanks for asking the following clarification questions. In response, we have revised the draft to enhance clarity and address the points raised as follows.

>Finally, what is the memory footprint? 

In the **newly added Sec. E.5**, we report the footprint of memory and GPU usage. The results show that the difference in memory and GPU usage between the proposed GSH and the vanila **dense** modern Hopfield layers is **negligible**. Please see Fig. 10 and Fig. 11 for more details.

>How much time model takes per epoch? 

The time per epoch varies depending on the specific setting.

As a reference, when using a batch size of 32 and an input length of 168, STanHop-Net requires approximately 2 minutes per epoch for the ETTh1 dataset. This duration is roughly equivalent to that of using dense modern Hopfield layers, such as STanHop-Net (D). Together with our results from Thm. 3.1 and Sec. E.1 — the tighter fixed point convergence of the Generalized Sparse modern Hopfield (GSH) model leads to fewer epochs required for convergence  — **STanHop-Net using the GSH model generally requires less time to converge** (than those of using dense Hopfield layers.) 

>Model size? 

The total number of parameters of the STanHop-Net is ~0.78 million. We have added this in the **new Sec. E.6** of the latest revision.

>Do you observe faster convergence? 

The tighter fixed point convergence of the Generalized Sparse modern Hopfield (GSH) model leads to fewer epochs required for convergence. Thus, yes, our empirical results (Fig. 4 in Sec. E.1) verifies our theoretical results (Thm. 3.1) --- the GSH model converges faster than the dense modern Hopfield model.

>I would like to see the variance of the model, including baseline models.

In Multivariate Time Series (MTS) prediction research, it is common to exclude the variance due to compactness of papers.

This convention is observed in all our baseline models — including Crossformer, Informer, FEDformer, DLinear, and Autoformer — none of which report variance in their respective papers.  As most of our baseline values are directly quoted from these original papers, we adhere to the same convention for consistency.

However, **we did report the maximum variance across all settings of our model in our table caption, stating “with variance omitted as they are all <=0.44%”.**

>Explanations for Lemma 3.1, 3.2, and other lemmas in the main paper, can be written in a better way. Rather than stating to refer to proof, you should try and provide the simplest explanation of what each proof talks about.

Thanks for pointing this out. We apologize for any confusion caused. In earlier drafts, proof sketches were included in the main text, but they were mostly removed or relocated to the appendix due to page constraints. We have now added the removed ones back in Sec.C. Please see revised version for details.

To clarify:

- **Lemma 3.1**: The high-level proof sketch begins in Section C.1, followed by detailed elaboration.
The explanation/intuition/purpose are outlined directly below Lemma 3.1: 
>Lemma 3.1 demonstrates how the energy function (3.2) aligns with the overlap-function construction of Hopfield models, referenced in Hu et al. (2023) and Ramsauer et al. (2020). The corresponding retrieval dynamics satisfying the monotonicity property (T1) are then introduced. 

- **Lemma 3.2**: The proof sketch for Lemma 3.2 is presented at the start of Section C.1, distinctly marked by a gray block for emphasis and ease of reference.
>Our proof is built on (Hu et al., 2023, Lemma 2.1). We first derive T by utilizing Lemma 3.1 and Remark G.1, along with the convex-concave procedure (Yuille and Rangarajan, 2003; 2001). Then, we show the monotonicity of minimizing H with T by constructing an iterative upper bound of H which is convex in x t+1 and thus, can be lowered iteratively by the convex-concave procedure.
- **Lemma 3.3**: he proof sketch for Lemma 3.2 is presented as the first sentence of the proof.
>Since the energy function H monotonically decreases with respect to increasing t in Lemma 3.2, we can follow [(Hu et al., 2023), Lemma 2.2] to guarantee the convergence property of T by checking the necessary conditions of Zangwill’s global convergence.
- **Lemma 3.4**: the proof sketch is inside and highlighted by the gray blcok:
>Our proof, built on (Hu et al., 2023, Lemma 2.1), proceeds in 3 steps:...

We hope these changes and clarifications enhance the clarity of explanations for our theoretical results.

We hope that our responses adequately address the reviewer's concerns, and we look forward to any further feedback. Thank you for your time and consideration.

Reviewer comment (Evaluation):
I would like to see the variance of the model, including baseline models.

Author response:
Thanks for asking the following clarification questions. In response, we have revised the draft to enhance clarity and address the points raised as follows.

>Finally, what is the memory footprint? 

In the **newly added Sec. E.5**, we report the footprint of memory and GPU usage. The results show that the difference in memory and GPU usage between the proposed GSH and the vanila **dense** modern Hopfield layers is **negligible**. Please see Fig. 10 and Fig. 11 for more details.

>How much time model takes per epoch? 

The time per epoch varies depending on the specific setting.

As a reference, when using a batch size of 32 and an input length of 168, STanHop-Net requires approximately 2 minutes per epoch for the ETTh1 dataset. This duration is roughly equivalent to that of using dense modern Hopfield layers, such as STanHop-Net (D). Together with our results from Thm. 3.1 and Sec. E.1 — the tighter fixed point convergence of the Generalized Sparse modern Hopfield (GSH) model leads to fewer epochs required for convergence  — **STanHop-Net using the GSH model generally requires less time to converge** (than those of using dense Hopfield layers.) 

>Model size? 

The total number of parameters of the STanHop-Net is ~0.78 million. We have added this in the **new Sec. E.6** of the latest revision.

>Do you observe faster convergence? 

The tighter fixed point convergence of the Generalized Sparse modern Hopfield (GSH) model leads to fewer epochs required for convergence. Thus, yes, our empirical results (Fig. 4 in Sec. E.1) verifies our theoretical results (Thm. 3.1) --- the GSH model converges faster than the dense modern Hopfield model.

>I would like to see the variance of the model, including baseline models.

In Multivariate Time Series (MTS) prediction research, it is common to exclude the variance due to compactness of papers.

This convention is observed in all our baseline models — including Crossformer, Informer, FEDformer, DLinear, and Autoformer — none of which report variance in their respective papers.  As most of our baseline values are directly quoted from these original papers, we adhere to the same convention for consistency.

However, **we did report the maximum variance across all settings of our model in our table caption, stating “with variance omitted as they are all <=0.44%”.**

>Explanations for Lemma 3.1, 3.2, and other lemmas in the main paper, can be written in a better way. Rather than stating to refer to proof, you should try and provide the simplest explanation of what each proof talks about.

Thanks for pointing this out. We apologize for any confusion caused. In earlier drafts, proof sketches were included in the main text, but they were mostly removed or relocated to the appendix due to page constraints. We have now added the removed ones back in Sec.C. Please see revised version for details.

To clarify:

- **Lemma 3.1**: The high-level proof sketch begins in Section C.1, followed by detailed elaboration.
The explanation/intuition/purpose are outlined directly below Lemma 3.1: 
>Lemma 3.1 demonstrates how the energy function (3.2) aligns with the overlap-function construction of Hopfield models, referenced in Hu et al. (2023) and Ramsauer et al. (2020). The corresponding retrieval dynamics satisfying the monotonicity property (T1) are then introduced. 

- **Lemma 3.2**: The proof sketch for Lemma 3.2 is presented at the start of Section C.1, distinctly marked by a gray block for emphasis and ease of reference.
>Our proof is built on (Hu et al., 2023, Lemma 2.1). We first derive T by utilizing Lemma 3.1 and Remark G.1, along with the convex-concave procedure (Yuille and Rangarajan, 2003; 2001). Then, we show the monotonicity of minimizing H with T by constructing an iterative upper bound of H which is convex in x t+1 and thus, can be lowered iteratively by the convex-concave procedure.
- **Lemma 3.3**: he proof sketch for Lemma 3.2 is presented as the first sentence of the proof.
>Since the energy function H monotonically decreases with respect to increasing t in Lemma 3.2, we can follow [(Hu et al., 2023), Lemma 2.2] to guarantee the convergence property of T by checking the necessary conditions of Zangwill’s global convergence.
- **Lemma 3.4**: the proof sketch is inside and highlighted by the gray blcok:
>Our proof, built on (Hu et al., 2023, Lemma 2.1), proceeds in 3 steps:...

We hope these changes and clarifications enhance the clarity of explanations for our theoretical results.

We hope that our responses adequately address the reviewer's concerns, and we look forward to any further feedback. Thank you for your time and consideration.

Reviewer comment (Novelty):
The paper's contributions are limited in scope.

Author response:
>The paper's contributions are limited in scope.

We appreciate the opportunity to clarify the scope of our contributions. 
In the initial draft, we may not have described the breadth and depth of our work as effectively as we intended. 

**On the technical side**, our contributions are threefold:

1. **Generalized Sparse Modern Hopfield Model (GSHM):** We present the GSHM as a generalization of [Hu, 2023] and include [Hu, 2023] and [Ramsauer, 2020] as special cases with $\alpha=2$ and $\alpha=1$, respectively. Additionally, we provide a theoretical analysis characterizing the fixed-point convergence property, memory retrieval error bound, noise robustness, and memory capacity. We leverage the GSHM's ability to capture varying degrees of sparsity and noise robustness to handle the intrinsic noise and sparsity of multivariate time series [Masini, 2023; Fawaz, 2019; Wang, 2013; Ozaki, 2012].
2. **STanHop-Net for Multivariate Time Series (MTS):** Recognizing that MTS inherently possesses a multi-resolution structure [Shabani, 2023; Cui, 2016], we introduce STanHop-Net. This network utilizes hierarchically stacked GSH layers to manage the multi-level sparsity and noise of an MTS, thereby enhancing performance. Our method effectively captures the intrinsic multi-resolution structure of both temporal and cross-variate correlations of time series, with resolution-specific sparsity at each level (as demonstrated through the component ablation study in Sec. E.3 of the submitted draft).
3. **External Memory Plugin Schemes:** We introduce two schemes for memory-enhanced prediction: a Plug-and-Play module and a Tune-and-Play module for train-less and task-aware memory enhancements, respectively. Unlike existing works [Kaiser, 2017], our approach avoids over-specialization on rare events, thus broadening its applicability and enhancing generalization capabilities across various tasks where the frequency and recency of data are less critical. Furthermore, our method demonstrates superior real-time adaptability.

**On the strategic side**, in the era of Large Foundation Models (such as ChatGPT), a theoretical understanding of transformer/attention-based models without overly strong assumptions is increasingly important. MHM offers a low-assumption framework for analyzing attention mechanisms through the lens of associative memory models. This unique perspective opens up new opportunities to study large generative foundation models within a rigorous, brain science-informed scientific framework. We believe our work contributes timely insights by proposing and analyzing the generalized sparse modern Hopfield model.

In summary, our contributions extend beyond the conventional scope, offering both theoretical advancements and practical applications in the realm of neural networks and time series analysis. We hope this clarification better highlights the novelties and impact of our work.

---
- [Masini, 2023] Masini, Ricardo P., Marcelo C. Medeiros, and Eduardo F. Mendes. "Machine learning advances for time series forecasting." Journal of economic surveys 37, no. 1 (2023): 76-111.
- [Shabani, 2023] Shabani, Amin, Amir Abdi, Lili Meng, and Tristan Sylvain. "Scaleformer: iterative multi-scale refining transformers for time series forecasting." ICLR (2023)
- [Fawaz, 2019] Ismail Fawaz, Hassan, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. "Deep learning for time series classification: a review." Data mining and knowledge discovery 33, no. 4 (2019): 917-963.
- [Kaiser, 2017] Kaiser, Łukasz, Ofir Nachum, Aurko Roy, and Samy Bengio. "Learning to remember rare events." arXiv preprint arXiv:1703.03129 (2017).
- [Cui, 2016] Cui, Zhicheng, Wenlin Chen, and Yixin Chen. "Multi-scale convolutional neural networks for time series classification." arXiv preprint arXiv:1603.06995 (2016).
- [Wang, 2013] Wang, Zhaoran, Fang Han, and Han Liu. "Sparse principal component analysis for high dimensional multivariate time series." In Artificial Intelligence and Statistics, pp. 48-56. PMLR, 2013.
- [Ozaki, 2012] Ozaki, Tohru. Time series modeling of neuroscience data. CRC press, 2012.

Reviewer comment (Experiments):
The model is too complex, it's may difficult to optimize.

Author response:
>The model is too complex, it may be difficult to optimize.

We appreciate the reviewer's concern regarding the complexity and optimizability of our model. 

To address this, we evaluate the model's optimization difficulty qualitatively by considering two factors: the size of the training dataset and the model's prediction performance.

Our results, as detailed in the submitted paper, demonstrate that our model achieves top-tier prediction performance, even with relatively small training datasets. This indicates that despite its complexity, the model is not challenging to optimize. For instance, on the Weather (WTH) dataset, which is only 1.5MB in size, our model attains state-of-the-art (SOTA) and near-SOTA performance across various prediction horizons.

Here are the sizes of the training datasets used and the corresponding performance achievements:

- **Electricity (ECL):** 35.7MB
- **ETTh1:** 1.8MB
- **ETTm1:** 6.9MB
- **Influenza-Like Illness (ILI):** 56KB
- **Traffic:** 92MB
- **Weather (WTH):** 1.5MB

These results clearly demonstrate that our model efficiently handles training on datasets of varying sizes while maintaining high prediction accuracy. This evidence strongly suggests that our model, despite its complexity, is not difficult to optimize.

We hope this clarify your concerns. Thank you!

Reviewer comment (Reproducibility):
Can the author provide the time complexity and number of the parameters of the model?

Author response:
>Can the author provide the time complexity and number of the parameters of the model?

Thanks for pointing this out. We report the numbers here and update the draft accordingly. Please see the **new Sec. E.6** of the latest revision.

**Number of Parameters:** 0.78 million

**Time Complexity:**
Let $T$ be the number of input length on the time dimension, $D_{\text{hidden}}$ be the hidden dimension, $P$ be patch size, $D_{\text{out}}$ be the prediction horizon, $\text{len}_Q$ be the size of query pattern in $\mathtt{GSHPooling}$.

- **Patch Embedding:** $O(D_{\text{hidden}} \times T) = O(T)$
- **Temporal and Cross Series GSH:** $O(D_{\text{hidden}}^2 \times T^2 \times P^{-2}) = O(T^2 \times P^{-2}) = O(T^2)$
- **Coarse Graining:** $O(D_{\text{hidden}}^2 \times T) = O(T)$
- **GSHPooling:** $O(D_{\text{hidden}} \times \text{len}_Q \times T^2 \times P^{-2}) = O(\text{len}_Q \times T^2)$
- **PlugMemory:** $O(T^2)$
- **TuneMemory:** $O((T + D_{\text{out}})^2)$
- **One STanHop Encoder Layer with PlugMemory:** $O(T + T^2 + T + (\text{len}_Q \times T^2) + T^2) = O(T + T^2 + \text{len}_Q)$
- **One STanHop Encoder Layer with TuneMemory:** 
  $O(T + T^2 + T + (\text{len}_Q \times T^2) + (T + D_{\text{out}})^2)$  $=O(T + T^2 + \text{len}_Q + (T + D_{\text{out}})^2)$
- **One STanHop Decoder Layer with PlugMemory:** 
  $O(T + T^2 + T + (\text{len}_Q \times T^2) + T^2) = O(T + T^2 + \text{len}_Q)$
- **One STanHop Decoder Layer with TuneMemory:** 
  $O(T + T^2 + T + (\text{len}_Q \times T^2) + (T + D_{\text{out}})^2) = O(T + T^2 + \text{len}_Q + (T + D_{\text{out}})^2)$

Reviewer comment (Presentation):
Can the author provide the limitation section?

Author response:
>Can the author provide the limitation section?

Thank you for pointing this out. We have included a discussion on the limitations of this work, the **new Sec. B1** in the latest revision, which we quote as follows:

The proposed generalized sparse modern Hopfield model shares similar inefficiencies due to the $\mathcal{O}(d^2)$ complexity. In addition, the effectiveness of our memory enhancement methods is contingent on the relevance of the external memory set to the specific inference task. Achieving a high degree of relevance in the external memory set often necessitates considerable human effort and domain expertise, just like our selection process detailed in Sec. F.2. This requirement could potentially limit the model's applicability in scenarios where such resources are scarce or unavailable.

Reviewer comment (Reproducibility):
You treat $\alpha$ as a learnable parameter: how is it parametrized and what do the learned values converge to?

Author response:
>You treat $\alpha$ as a learnable parameter: how is it parametrized and what do the learned values converge to?

In each layer of the Generalized Sparse Hopfield Model (GSH), we treat $\alpha$ as a learnable scalar parameter, with its range set between 1 and 2. This parameter is updated concurrently with the $W_K, W_Q, W_V$ matrices through backpropagation. As for initialization, we start with a default value of 1.5. As for convergence, we observe that the learned values of $\alpha$ typically scatter within the 1-2 range. This variation aligns with our expectations, as the optimal value of $\alpha$ is expected to be data-dependent

In addition, we provide additional experimental visualizations demonstrating the relationship between 
* (1) training loss and alpha learning curve,
* (2) noise-level and learned alpha,

in Sec. E.1 and Sec. E.4.

Reviewer comment (Experiments):
The case $\alpha \to \infty$ corresponds to using an argmax instead of softmax, i.e., retrieving the most compatible pattern in one step. How would such an "argmax"-based lookup memory perform like in the experiments? (Some of the gradients will be zero, but the same happens some of the times with high alpha too, even if not always.)

Author response:
>The case corresponds to using an argmax instead of softmax, i.e., retrieving the most compatible pattern in one step. How would such a "argmax"-based lookup memory perform like in the experiments? (Some of the gradients will be zero, but the same happens some of the times with high alpha too, even if not always.)

Thanks for the question. It is an instructive piece to include into our paper.

In the **newly added Sec. E.4 (Fig. 8 & Fig. 9)**, 
we examine the impact of increasing the value of $\alpha$ on memory capacity and noise robustness. 

It is known that as $\alpha$ approaches infinity, the $\alpha$-entmax operation transitions to a hardmax operation [Peters 2019; Correia, 2019]. Furthermore, it is also known that memory pattern retrieval using hardmax is expected to exhibit perfect retrieval ability, potentially offering a larger memory capacity than the softmax modern Hopfield model in pure retrieval tasks [Millidge, 2022].

**Our empirical investigations confirm that higher values of $\alpha$ frequently lead to higher memory capacity.**

We report results only up to $\alpha=5$, as we observed that values of $\alpha$ greater than 5 consistently lead to numerical errors, especially when using float32 precision.
It is crucial to note, that while the hardmax operation (realized when $\alpha \rightarrow \infty$) may maximize memory capacity, its lack of differentiability renders it unsuitable for gradient descent-based optimization.

We hope that the revisions and clarifications provided in this response address the reviewer's concerns and make the value of our work clear. We look forward to further feedback and discussion.

--- 
- [Peters, 2019] Ben Peters, Vlad Niculae, and Andr´e FT Martins. Sparse sequence-to-sequence models. arXiv preprint arXiv:1905.05702, 2019.
- [Correia, 2019] Gonc¸alo M Correia, Vlad Niculae, and Andr´e FT Martins. Adaptively sparse transformers. arXiv preprint arXiv:1909.00015, 2019.
- [Millidge, 2022] Beren Millidge, Tommaso Salvatori, Yuhang Song, Thomas Lukasiewicz, and Rafal Bogacz. Universal hopfield networks: A general framework for single-shot associative memory models. In International Conference on Machine Learning, pages 15561–15583. PMLR, 2022.

Reviewer comment (Theory):
In the definition of the Tsallis entropy (3.1), the bottom branch for alpha=1 seems wrong, as it gives the negative of what would be the top branch for alpha=2. I expected by continuity to define alpha=1 as the Shannon entropy -sum p log p. Am I missing something?

Author response:
> in the definition of the Tsallis entropy (3.1).... I expected by continuity to define alpha=1 as the Shannon entropy -sum p log p. Am I missing something?

You are absolutely correct. That is a typo. Thanks for pointing this out. We have fixed it.

Reviewer comment (Theory):
In equation (3.2), the definition of $\Psi_{\alpha}^\star$ seems surprising: $\alpha$-entmax is a vector-value function, thus so should be its integral, but the energy H(x) should be scalar-value. I expect (as stated also elsewhere in the paper) that $\Psi_\alpha^\star$ should be the Fenchel convex conjugate of $\Psi_\alpha$, i.e. $\Psi_\alpha^\star(z) = \sup_{z^\star \in \operatorname{dom}{\Psi_\alpha}} \langle z^\star, z \rangle - \Psi_\alpha(z^\star)$. Could you please clarify?

Author response:
> In equation (3.2), the definition seems surprising: alpha-enmax is a vector-value function, thus so should be its integral, but the energy H(x) should be scalar-value. I expect (as stated also elsewhere in the paper) that  should be the **Fenchel convex conjugate** of , i.e. . Could you please clarify?

Yes, your observation is astute. Sorry for the lack of clarity. We stated "$\Psi^\star$ is the convex conjugate of $\Psi_\alpha$" in the 1st line of page 4 (and provided explicit form in Definition C.1). However, we believe it can be more precise to avoid the confusion reviewer raised. We have modified the draft accordingly. The revision will be posted shortly.

Reviewer comment (Reproducibility):
It was not clear to me how the memories Y are constructed; in section 4.3 it seems like the memories must be the same length as the input sequence R. Is this a strong requirement or could it be avoided? A nice property of attention models is that they should support variable length data.

Author response:
>It was not clear to me how the memories Y are constructed; in section 4.3 it seems like the memories must be the same length as the input sequence R. Is this a strong requirement or could it be avoided? A nice property of attention models is that they should support variable length data.

Sorry for the confusion. The sequence length of $\mathbf{Y}$ and $\mathbf{R}$ can be arbitrary. 

The idea of Sec. 3.2 is to make the retrieval dynamics of modern Hopfield model into a deep learning layer.
This new layer (GSH) should store and retrieve (hidden) representations throughout a DL pipeline/architecture.

To do so, we first introduce two hidden representations as input of the GSH layer:
* $\mathbf{Y} \in \mathbb{R}^{M \times a}$: $M$ **raw** stored memory patterns $\mathbf{y}\in \mathbb{R}^a$ 
* $\mathbf{R} \in \mathbb{R}^{T \times b}$: $T$ **raw** query patterns $\mathbf{r}\in \mathbb{R}^b$ 

As we know MHM operates in the $d$-dimensional *associative space*. We have memory and query patterns as follows:
* $\boldsymbol{\Xi} \in \mathbb{R}^{M \times d}$: $M$ stored memory patterns $\boldsymbol{\xi} \in \mathbb{R}^d$ 
* $\mathbf{X} \in \mathbb{R}^{T \times d}$: $T$ query patterns $\mathbf{r}\in \mathbb{R}^d$ 

Therefore, to transform $\boldsymbol{\Xi}\cdot \alpha$-$\text{EntMax}(\beta \boldsymbol{\Xi}^T\mathbf{X})$ into $\alpha$-$\text{EntMax}(\beta \mathbf{X}^T\mathbf{K})\mathbf{V}$ in eqn. (3.8), we need some linear transformations (i.e., the $\mathbf{W}$ matrices) to connect $(\mathbf{Y},\mathbf{R}) \leftrightarrow (\boldsymbol{\Xi},\mathbf{X})$.

From the above discussion, it's clear that the sequence length of $\mathbf{Y}$ and $\mathbf{R}$ can be arbitrary, as long as their first dimensions match those of $(\boldsymbol{\Xi}, \mathbf{X})$. **Namely, MHM layers support data of arbitrary length.**
Similar discussions can be found at [https://ml-jku.github.io/hopfield-layers/](https://ml-jku.github.io/hopfield-layers/) (around Eq. 23).

Reviewer comment (Theory):
The qualitative change at alpha=2 in Theorem 3.1 seems surprising and interesting. Could you discuss the difference a bit, especially the difference between the Max terms that show up, and give some intuition about why we see this change? Is one of the bounds always tighter than the other or does this depend on choices of (m, M, d, beta, etc?) I did not have the time to read the proof.

Author response:
>The qualitative change at alpha=2 in Theorem 3.1 seems surprising and interesting. Could you discuss the difference a bit, especially the difference between the Max terms that show up, and give some intuition about why we see this change? Is one of the bounds always tighter than the other or does this depend on choices of (m, M, d, beta, etc?) I did not have the time to read the proof.

Let $\mathcal{T}(\mathbf{x})$ be the retrieval dynamics acting on query $\mathbf{x}$, abd $\|| \mathcal{T}(\mathbf{x})- \boldsymbol{\xi}||$ be the retrieval error of the GSH model.

For $2\ge \alpha \ge 1$, the max-terms shows up when we connect $\Delta_\mu$, the separation between $\boldsymbol{\xi}_\mu$ and all other memory patterns in $\boldsymbol{\Xi}$, with $\|| \mathcal{T}(\mathbf{x})- \boldsymbol{\xi}||$. Explicitly, the max term when we define such a separation as the minimal inner product difference from one memory pattern  to all other memory patterns, measured by inner product similarity. Please see Def. 3.1 of [Hu, 2023] for more details.

On the other hand, for $\alpha \ge 2$, it is easy to show that $\|\mathcal{T}(\mathbf{x}) - \boldsymbol{\xi}\|$ is consistently smaller than that of the Sparse Modern Hopfield Model [Hu, 2023]. Specifically, greater sparsity leads to a tighter error bound (thus hardmax can achieve perfect retrieval [Millidge, 2022].) Given that $\alpha = 2$ provides the upper bound on retrieval error within this range of $\alpha$: $\|\mathcal{T}(\mathbf{x}) - \boldsymbol{\xi}\| \le \|\mathcal{T}_{\alpha=2}(\mathbf{x}) - \boldsymbol{\xi}\|$, we can utilize the upper bound at $\alpha = 2$ (sparsemax result from [Hu, 2023]) to establish an upper bound. Here, different from above, the max-term arises from component-wise upper bounding, where the *largest* component is used to upper bound the *average/expected value*.

Yes, **it is possible the two bounds might overlap with some set of (M,beta,d,m,n)**. This is a natural consequence as we do not impose any strong assumption on the data and model. 

We hope these points clarify the theoretical analysis of our model.