Paper: Continual Memory Neurons (ICLR-2024)

Reviewer comment (Writing):
The writing should be significantly improved. The current manuscript is not easy to follow.

Author response:
**Weakness1** We will proofread the paper and improve the writing, where necessary. If the reviewer would like to point out which sections of the paper were hard to follow we would appreciate the feedback so as to be able to better address the reviewer’s concern.

Reviewer comment (Theory):
The presented techniques are not presented with convincing theoretical justifications. For example, what's the motivation for using Eq. (6) to update the keys K?

Author response:
**W2** Equation 6 is fully inspired by online K-Means. In fact, the WTA policy (as in K-Means) that we implement to update the memories in the neuron is what guarantees parameter isolation (thus avoiding forgetting), while also fostering forward transfer capabilities. The main idea is to let the network choose the $\delta$ most relevant memories using the similarity function defined in Equation 4, which will then be used in the update step. The remaining memories are going to be left untouched so as to prevent forgetting: if the memories are not relevant for the current sample, we do not want to update them to preserve the weights that were learnt for a different set of samples. At the same time updating $\delta$ memories rather than a single one can help to share information across similar tasks/distributions, which leads to forward transfer.  
Such considerations are confirmed by the experimental validation (e.g. Table 1), where forgetting is mitigated while forward transfer is present.  
In this update strategy, $\delta$ is a hyperparameter that can be adjusted depending on whether more isolation is preferable or not. We set $\delta=2$ for moons and modes and $\delta=5$ for ns-imagenet, as stated in the supplementary.

Reviewer comment (Novelty):
Strong statements are made without convincing justification. For example, in Abstract, "...we are the first to follow a significantly novel route..."

Author response:
**Q1** In our paper we propose a novel take on the continual learning problem by following a different route from prior work in the sense that we address the problem at the neuron level. The proposed Continual Memory Neuron is specifically designed to handle shifts in distributions and ingest ill-distributed data that can evolve continuously. Differently from prior works, we do not require any knowledge of task or distribution boundaries since weights are isolated in a data-driven fashion. In addition, we do not rely on rehearsal strategies that are commonly leveraged. Such strategies make use of non parametric memory buffers, require additional training steps to make a model recover forgotten capabilities and require to store samples based on some heuristics that again may be based on external knowledge about the current task or ground truth category. We believe that such research direction has not been investigated by other works.  
**We checked the paper for potential overstatements and removed them**.

Reviewer comment (Experiments):
How to select the hyperparameter $\delta$?

Author response:
**Q3**  As pointed out in the W2 answer above, $\delta$ is a hyperparameter that tunes the desired amount of parameter isolation. When the data exhibits lots of heterogeneous distribution shifts it is better to keep $\delta$ as low as possible, since this will harness the capability of isolating memories and specializing them for different distributions. At the same time, increasing $\delta$ can foster forward transfer. Note that when $\delta$ is set to be equal to the number of memories, all weights are updated for each training sample. This setting is likely the desired one in the presence of i.i.d. data. In our experiments, dealing with heavily biased data distributions, we set $\delta$ to 2 for moons and modes and 5 for ns-imagenet.

Reviewer comment (Novelty):
What are the insights in Section 2.1? How do the revealed connections with existing machine learning techniques contribute to the proposed CMN?

Author response:
**Q4** The insights in Section 2.1, as acknowledged by reviewer Lniw, should help to better frame the proposed approach in the state of the art, showing that CMNs lay their roots into well established approaches that involve memorization (Kernel Methods) and attention (Transformers). We believe that pointing out such a robust background will aid the understanding of the proposed approach as well as foster future research on the topic.

Reviewer comment (Experiments):
In the experiments, is a single data sample presented to the model at each time step? It seems in Figure 4 that each time step processes a new distribution (with many data samples)?

Author response:
**Q5** Indeed in the experiments a single sample is processed at a time. Figure 4 shows a snapshot of what the model is learning after having processed the last sample in each distribution. Actually, nor the training scheme nor the model are aware of task/distribution boundaries. Each subfigure in Figure 4 is thus showing the decision boundaries at different times, yet samples are processed one at a time. **We clarified this in the Figure caption**.

Reviewer comment (Evaluation):
In Figure 5, it seems that CMN behaves differently on the 2-dimensional simulated datasets and the real-world imagenet dataset. Why?

Author response:
**Q6**  In Fig. 5 (c,d) the main difference is due to the different dimensionality of the data. As expected, in fact, the cosine similarity/dot product is more appropriate in larger dimensional cases, such as the one of ImageNet.

Reviewer comment (Presentation):
Nearly a page discusses connections to kernel methods, Transformers and RELUs—however these seem quite non-central to the proposed mechanism.

Author response:
**Weakness 1**  Since we are proposing a novel neuron model, one might wonder how it is computationally different from what happens in popular neural architectures that share some of the intuitions that are also present in CMNs, that is why we investigated the connection with them. For example, Kernel Methods, in their dual form, learn function that depend on all the seen training examples, thus they do memorize all the data, and we found it important to see how CMNs, being them memory-oriented neurons, are related to them (of course, CMNs do not store all the data). Moreover, since we are building on key-value functions, the connection with Transformers seems appropriate. We remark that other Reviewers (Lniw) appreciated this discussion. 
The reviewer emphasizes the similarity between our work and retrieval-augmented continual learning, referring to a list of recent works on LLMs where retrieval is interpreted as access to external sources of information (e.g., search engines). This is entirely different from what we do. We propose an enhanced version of a neuron that learns different set of weights to handle non-stationary data distributions during training. In any case, our neural architectures are entirely self-contained without any proper ‘retrieve’ operation. The only thing in our model which is vaguely connected with retrieval is the attention strategy to access the memories.
In a sense, every attention-oriented model can be intended as retrieving keys from a storage, thus we do see  the connection that the Reviewer is pushing, but we find it to be way too weak/generic. CMNs work at the lowest possible abstraction level, and they could be paired with any other existing CL approaches, including Retrieval-augmented CL, experience replay, … We ask the Reviewer to better point out which methods we should discuss in the ‘related works’ section, in order to provide an appropriate answer.

Reviewer comment (Experiments):
The memory here is stored as weight matrices, which are inevitably smaller than storing images—this in my opinion creates an unfair comparison. This is because CMNs can effectively store more samples than compared approaches, effectively achieving higher performance.

Author response:
**W. 2** The Reviewer is remarking that CMNs store *“more examples”* than other methods in the *“weight matrices”* that are smaller than storing images. We compared our approach and exemplar based competitors from the computational and memory point of view, and we proposed an in depth analysis in  Appendix C of the submitted paper geared toward a fair comparison. Storing data in a more efficient way is indeed an important feature of CMNs. We believe that having the ability to store more information with the same amount of memory is a crucial characteristic in the online continual learning setting. We do not understand the short comment on the GPU memory. Also mini-batches of rehearsal buffer are loaded into GPU VRAM when processing mini-batches, and our comparisons were built in a way in which such memory consumption when comparing CMNs and standard neurons  is the same.  The Reviewer refers to *“too low”* memory constraints, but we remark that we deal with an online learning scenario where frugal data storage is an extremely important requirement.

Reviewer comment (Experiments):
The memory constraint is set by equalizing megabytes (MB) of storage, but this limit is too low. The proposed deep CMN networks (and comparisons) require higher GPU VRAM than the claimed available space on HDDs.

Author response:
**W. 2** The Reviewer is remarking that CMNs store *“more examples”* than other methods in the *“weight matrices”* that are smaller than storing images. We compared our approach and exemplar based competitors from the computational and memory point of view, and we proposed an in depth analysis in  Appendix C of the submitted paper geared toward a fair comparison. Storing data in a more efficient way is indeed an important feature of CMNs. We believe that having the ability to store more information with the same amount of memory is a crucial characteristic in the online continual learning setting. We do not understand the short comment on the GPU memory. Also mini-batches of rehearsal buffer are loaded into GPU VRAM when processing mini-batches, and our comparisons were built in a way in which such memory consumption when comparing CMNs and standard neurons  is the same.  The Reviewer refers to *“too low”* memory constraints, but we remark that we deal with an online learning scenario where frugal data storage is an extremely important requirement.

Reviewer comment (Experiments):
For larger datasets, some degree of features beyond raw pixels is needed; however, CIFAR10 performance of 27% here is an astonishingly low bar to outperform.

Author response:
**W. 3**  As usual, numbers must be evaluated in the considered experimental setup, that is the one of a continual online learning scenario. We really do not understand the comments about the comparison with [1], which is about a metric-learning approach, assessed on a completely different setting and dataset. The results we provided include a large number of comparisons with SOTA approaches. We believe that experimental results should not be superficially dismissed just because ‘accuracy looks too low’ and we stress that reproducibility of experiments is possible since we shared the code. We recall we are in a continual online setting (1 example at a time, no epochs, no batches,…), that is way more extreme that what is explored in several other papers! The choice on the simple neural architecture is what yields the performance we reported in CIFAR10 - again, this is continual online learning! The learning setting is indeed challenging and it is a precious testbed for CMNs and for comparisons. Recall (see the paper) that we validated the model parameters in a realistic setting in which a small proportion of the stream is exploited, that makes the setting more appropriate for comparisons and also more tough.

*[1] Mensink etal, Distance-Based Image Classification: Generalizing to new classes at near-zero cost, 2013*

Reviewer comment (Experiments):
Concern that the proposed mechanism might be useful, but the current evaluations are under-powered to verify benefits of CMNs.

Author response:
**W. 3**  As usual, numbers must be evaluated in the considered experimental setup, that is the one of a continual online learning scenario. We really do not understand the comments about the comparison with [1], which is about a metric-learning approach, assessed on a completely different setting and dataset. The results we provided include a large number of comparisons with SOTA approaches. We believe that experimental results should not be superficially dismissed just because ‘accuracy looks too low’ and we stress that reproducibility of experiments is possible since we shared the code. We recall we are in a continual online setting (1 example at a time, no epochs, no batches,…), that is way more extreme that what is explored in several other papers! The choice on the simple neural architecture is what yields the performance we reported in CIFAR10 - again, this is continual online learning! The learning setting is indeed challenging and it is a precious testbed for CMNs and for comparisons. Recall (see the paper) that we validated the model parameters in a realistic setting in which a small proportion of the stream is exploited, that makes the setting more appropriate for comparisons and also more tough.

*[1] Mensink etal, Distance-Based Image Classification: Generalizing to new classes at near-zero cost, 2013*

Reviewer comment (Experiments):
In the moons/modes dataset, how do non-CMN methods perform with the RBF kernel? Am I correct that the ψ(x) other than identity is only used in this scenario? What is the contribution of using the key-value pairing in CMN and RBF kernel for the task? Would equalizing the RBF kernel aspect significantly affect the performance gain of CMN method?

Author response:
**Q. 1** We remark that $\psi$ is not a kernel, but a fixed transformation that maps $x$ toward a customizable space in which it might be easier to compute similarities. We  set $\psi$ to the identity function in all the experiments, for simplicity. Indeed, both CMNs and non-CMN methods process the same data $\psi(x)=x$.   Conversely, the RBF kernel is one of the possible implementations for the similarity function  $\mathrm{sim}$ that drives comparison with keys (see the first part of our response ).
We do not understand the comparisons that the Reviewer is indicating in the “questions” box. Changing the variance of the kernel will not affect the WTA procedures, but it would change the way the winning key is updated.

Reviewer comment (Evaluation):
Why are the results varying so much in MNIST-CI? It is very strange to see 21.0 for ER-Random, 23.2 for MoE and 14.3 for GDumb but 70.3 for ER-Reservoir-Imbalanced. Reservoir-Imbalanced sampling (Chrysakis & Moens, 2020) approximates random sampling as there is no imbalance to correct, so they should have identical performance. Overall, 14.3/21.0/23.2 on MNIST-CI seems suspiciously low performance despite the memory size being 100.

Author response:
**Q. 2**  Reservoir-based management of the buffer reduces the probability of adding a sample to it as long as time passes, and has been proven to be very effective in the Continual Online Learning scenario [2]. In bare random-management of the buffer, it is more likely to have replacements (that leads to a buffer containing a lot of recent samples), that is why performances are different. See [2] for further details on the difference between the two algorithms. 

*[2] Mai, Zheda, et al. "Online continual learning in image classification: An empirical survey." Neurocomputing 469 (2022): 28-51.*

Reviewer comment (Experiments):
It seems that in the experiments the authors have only tried with a single layer of CMN, but the naming of “neuron” and the methodology suggests that it can be applied more widely across layers. It would be better to showcase the general applicability into multi layer CMNs. A single CMN can be achieved with a more standard online clustering algorithm.

Author response:
As the Reviewer correctly suggests, we devised CMNs to be general neural components that could be plugged into any arbitrary architecture. We achieved impressive results exploiting single layered CMNs architecture, even surpassing the performance of models equipped with several layers of standard neurons, which require, in addition, to leverage rehearsal buffers in order to be competitive in the continual learning scenario – that is not our case. Thus, in principle, the proposed CMNs are general enough to be plugged into any neural architecture, and they can be paired with existing approaches for continual learning. However, their interactions in deep networks with several layers, or the way they behave with other learning approaches still require specific and detailed investigations. For example, we focused on networks in which the input of the CMNs was not affected by the learning dynamics, both when using backbones with several layers or when directly classifying data in their original representations. Each CMN virtually and softly partitions its input space, dynamically learning how to do it and how to behave in each partition. In multiple layers of CMNs, the progressive-compositional development of such partitions can lead to instabilities or difficulties in quickly learning in an online manner.
This will be the subject of our future work. We remark that many other works in the CL scenario propose to leverage pretrained/freezed backbones and to learn on top succinct memory systems that exploit the underlying knowledge by selecting the most appropriate one depending on the current sample [a]. We believe that our proposal is per-se an already relevant contribution in this direction, as evidenced by the Imagenet-NS experiment.  
Moreover, by stacking multiple layers of CMNs we did not observe any significant gain. Interestingly, when multiple layers are stacked, on average we observe a slight decrease in accuracy, both for our method and for the competitors. This finding confirms that larger models tend to perform better than deeper models for continual learning, due to exploding gradients leading to forgetting in the earliest layers, as discussed in [b, c]. We added these considerations in the revised paper (please see Appendix E).

*[a] Wang, Zifeng, et al. "Learning to prompt for continual learning."  IEEE/CVF CVPR 2022.*

*[b] Mirzadeh, S. I. et al. 2022. Wide neural networks forget less catastrophically. In ICML  2022*

*[b] Mirzadeh, S. I. et al. 2022. Architecture matters in continual learning.  arXiv:2202.00275.*

Reviewer comment (Experiments):
To continue with the previous point, a concern with applying CMN to multiple layers is that when the representations are not fully trained or in early layers, committing to the winner memory slot may hinder the learning progress, but more studies would be needed.

Author response:
We indeed agree with the Reviewer that more studies are needed in the case of multiple layers.  Please also refer to our response to W1. We do believe the novelty and the experimental quality of this proposal are enough to yield a significant impact in the context of a conference.

Reviewer comment (Novelty):
The memory unit has resemblance to Ren et al. (2021) so it would be great to discuss its relations in the paper. Both works use a slot-based memory module based on input similarity. Both recycle the least used entry and create a new entry if the match strength is below a threshold.

Author response:
Thanks for this interesting suggestion! We referenced the paper in the updated submission, due to some analogies with a few components that are also present in what we propose, even if the two proposals are indeed overall different.
Indeed, Ren et al. propose a prototype-based memory network with a control component that determines when to form a new class prototype. This is done via a probabilistic clustering module that is jointly learned with the representation encoder. Cluster assignment updates and parameters estimates are achieved via the EM-algorithm based on the input-prototypes similarity, in a streaming setting. The cluster formation is guided by a self-supervised contrastive loss that encourages different views of the same image to be assigned to the same prototype, in addition to other loss terms that are needed to guarantee the formation of new clusters. Least used clusters are removed based on thresholds.
Conversely, we propose a novel neuron model leveraging a decoupled key-value mechanism within the neuron itself. While keys are updated via an online clustering method on the input space, top-$\delta$ memories are learned via the backpropagated gradients computed on the current loss function without the need of ad-hoc loss terms. Every CMN has its own independent memory bank, and CMNs can be combined into layers.

Reviewer comment (Reproducibility):
Hyperparameters selection of CMN seems like a burden. It would be great to show a list of hyperparameters and their optimal range for each task, and discuss how sensitive these values are.

Author response:
We thank the Reviewer for this feedback. In the supplementary materials (appendices) we reported the hyperparameter grids and the best selected parameter for the main results. We also investigated the role of key design choices in the ablation study (Fig. 5). In our experience, their impact was prevalent, so we found it more meaningful to provide this kind of information rather than a fine-grained exploration of the hyperparameter values. Of course, there is always room to investigate deeply the sensitivity to each hyperparameter.