Paper: Adiabatic replay for continual learning (ICLR-2024)

Reviewer comment (Theory):
The paper lacks justification for the adiabatic assumption. The paper creates specific settings for the experiments and makes some discussions, but are there many real-world scenarios in which such adiabatic assumptions can be applied? Also, there seems to require a more detailed formulation of such an adiabatic assumption. Say if the new information is too little, then it gets back to the ordinary training and there is no reason to do continual learning. Some mathematical formulation of the adiabatic assumption should be defined.

Author response:
1a. In virtually all articles about CL, the same amount of samples is added for each task, so the adiabatic assumption is automatically fulfilled after a few tasks. E.g., a 10-fold split of CIFAR along class boundaries will add 3000 samples per task, so at task t>=2 the number of new samples is smaller than the number of previous samples. So we feel the adiabatic assumption is adhered to to good approximation.
Of course you could imagine tasks where this assumption is always violated, but we argue that learning in humans or robots get more and more adiabatic over time, since additions tend to have the same size and the pool of existing knowledge is ever-growing.

1b. We will add a more formal statement of the adiabatic assumption. We will also add experiments on 5-fold and 10-fold equal splits to show that a violation does not break AR.

Reviewer comment (Experiments):
In addition to the mathematical definition of the adiabatic assumption, it is also not clear why the selected settings in the experiments satisfy such an assumption. Why the tasks are chosen in such a way? From a more practical perspective, if we are dealing with real-world tasks, how do we check if the incoming task satisfy the adiabatic assumption and we can use the proposed method?

Author response:
2. The tasks are chosen such that the a.a. is always fulfilled, and are otherwise modeled after common continual learning benchmarks. As stated in 1b, a violation of the a.a. does not break AR, and we will show this by adding more experiments. From a practical perspective, it is very easy to detect whether a new task violates the a.a.: we just need to measure how many different existing GMM components are best-matching ones when exposed to the new task's data. If there are few of them, then the a.a. is valid. This will be discussed as well.

Reviewer comment (Experiments):
The paper only tests for certain restricted settings and uses a relatively simple model (GMM). Though the paper mentions that there could be more advanced version of the GMM model that could solve the capacity problem, but there does not seem to be much evidence to support such a claim.

Author response:
3. We will remove the section about deep GMMs as this is not followed up in the article. Yes, there are deep convolutional extensions to GMMs that can be used, but thanks to latent replay, the flat GMMs do a sufficiently good job as shown by the experimental results.

Reviewer comment (Theory):
It is claimed that the proposed method does not have the problem of scaling up as the number of tasks increases. However, for GMM, there is the number of clusters and I wonder should such a number be set according to the total number of tasks? If there are infinite number of classes coming in a stream, should the number of clusters also increase? Even though the change of size could be small, if we want to use a much more capable model as mentioned in the paper, will the model size go up as the tasks increase?

Author response:
4a. No, our GMM does not grow over time. Instead, we choose K as high as our memory allows it because, as stated in app. B, more is always better when it comes to choosing K. This was shown in Gepperth&Pfülb(2021). If the number of classes is higher than K, performance will suffer, to be sure. So yes, we need to know or guess the total number of classes in advance, but any other DNN model needs to do this as well.

4b. Even the most sophisticated DNN will fail for an infinite number of classes. Virtually *all* recent works on CL that we are aware of assume that the total number of classes is known beforehand, and all works use static DNNs/CNNs that do not grow in size. A priori knowledge of the number of classes is required for tuning generator and solver structure, as well as for choosing the size of the classification head. So our method is no different in that respect.

Reviewer comment (Experiments):
I am afraid critical comparisons are missing for a convincing proof-of-principle demonstration that selective replay can provide substantial benefits. The paper compares AR (using GMMs) with DGR and ER (both using VAEs), and because AR performs better than DGR, the authors conclude that selective replay is beneficial. But I do not think that this can be concluded from this comparison. There are several important differences between AR and DGR – not only the use of selective replay – that could explain the difference in performance. Perhaps the authors would argue that DGR is a “state-of-the-art” technique, and that showing improved performance with a method using selective replay would be enough demonstration, but I do not agree with that.

Author response:
1a. All DGR and ER experiments are conducted in the constant-time setting, i.e., as many generated/replayed samples as there are new samples.  Normally, one would train DGR in a balanced fashion, i.e., generate more samples for each new task. Since we do this differently, the results for DGR are "not good", as you remarked. However, this ensures that the comparison to AR is fair, and this is why we conclude that selective  replay is beneficial since DGR obviously cannot cope with constant-time replay.

Reviewer comment (Experiments):
Firstly, it is not clear how DGR is implemented and it seems this is not done in an optimal way (see issues 2 and 3). Secondly, the performance obtained by DGR is simply not good (e.g., for MNIST it is substantially lower than a linear classifier could obtain). Improvements on top of that are thus not necessarily meaningful. I would like to encourage the authors to instead include a comparison that directly assesses the benefit of using selective replay over normal replay (e.g., AR against the exact same version but using normal replay).

Author response:
1a. All DGR and ER experiments are conducted in the constant-time setting, i.e., as many generated/replayed samples as there are new samples.  Normally, one would train DGR in a balanced fashion, i.e., generate more samples for each new task. Since we do this differently, the results for DGR are "not good", as you remarked. However, this ensures that the comparison to AR is fair, and this is why we conclude that selective  replay is beneficial since DGR obviously cannot cope with constant-time replay.

1b. To show that our DGR implementation is sound, we will include results when  DGR is trained in the  standard fashion. We actually did this long ago, and results are indeed stronger and more in agreement with the literature.

1c. We will compare  AR to the exact same version but with normal replay. This will probably not change the results, and thus prove that selective replay performs similarly to normal replay, but at far better efficiency.

Reviewer comment (Novelty):
Discussion of past literature is insufficient, and some important aspects are ignored. Firstly, it is claimed at several places that current replay methods must replay an amount of samples proportional to the number of past tasks, but Van de Ven et al. (2020) empirically showed that it is possible to do better than this because “preventing forgetting is easier than learning”. Although to make this possible, it is important that the loss terms from the replayed data and the current data are balanced appropriately. (It is also unclear to me whether this is done in the current study when a limited amount of data is replayed with DGR and ER, see also the last question under issue 3 below.)

Author response:
Concerning (2): we will measure the baselines for ER and DGR with task-specific weights as proposed in Van de Ven et al. (2020), it really does make sense to use baselines that are widely known and accepted. We will provide a discussion of the pros and cons of such an approach and justify why we choose not to do this for AR.

Reviewer comment (Novelty):
Another paper that would be good to discuss is Klasson et al. (2023, TMLR; https://openreview.net/pdf?id=Q4aAITDgdP), as they show that benefits can be obtained by being smart about what to replay. I think it is also relevant to discuss McClelland et al. (2020, Phil Trans R Soc B; https://royalsocietypublishing.org/doi/full/10.1098/rstb.2019.0637), as they make a conceptual argument that it should not be necessary to replay everything.

Author response:
- discussed additional related work, mainly  Klasson et al. (2023, TMLR; https://openreview.net/pdf?id=Q4aAITDgdP) and McClelland et al. (2020, Phil Trans R Soc B; https://royalsocietypublishing.org/doi/full/10.1098/rstb.2019.0637) as conceptual foundations of our approach

Reviewer comment (Reproducibility):
How is the solver of AR trained? Is the solver also trained with “adiabatic replay”? Or is the solver trained with “regular replay”, and suffers from a linear increase in the amount of replayed data? How are the labels for the replayed data obtained?

Author response:
* The AR "solver", which is just a linear classifier, is trained with AR as well and thus does not compromise AR's sample efficiency.
* The labels are obtained by running the generated samples through the solver. We experimented with conditional replay but found that this enormously impaired the quality of generated samples.

Reviewer comment (Reproducibility):
For DGR, how are the labels obtained for training the solver?

Author response:
* The labels are obtained by running the generated samples through the solver. We experimented with conditional replay but found that this enormously impaired the quality of generated samples.

Reviewer comment (Reproducibility):
For ER and DGR, how are the loss terms from the replayed data and the current data weighed? Are they balanced as in Van de Ven et al. (2020), or are they simply added?

Author response:
* The loss terms are not weighted: all generated samples have the same weight in the loss. The reason: we found at several occasions that specific weights for each replay task can lead to complications. As more and more tasks are added, the weights for long-ago tasks must be chosen higher and higher. If coefficients are too high for a given task, gradient descent can fail completely because the effective learning rate is too high. In addition, we feel it would be unfair if task-specific weights were used for DGR and ER, since they are neither used nor required for AR, resulting in a biased comparison. We can of course include such results for reference.

Reviewer comment (Reproducibility):
Given the importance of ER in the current manuscript, the buffer size that is used should be mentioned in the main text.

Author response:
* The ER buffer  size is 50.

Reviewer comment (Writing):
The writing needs to be improved.

Author response:
1. we will add pseudocodes, no problem. Could you be more specific in what ways the writing should be improved?

Reviewer comment (Experiments):
I usually don't focus too much on the experiments, but I do believe that the experiments should be improved where more baselines and time cost experiments are necessary.

Author response:
2. What baselines are you referring to? Could you provide references?

Reviewer comment (Experiments):
The reasonable way of chosing hyperparameter $K$ is necessary. Please discuss more about $K$.

Author response:
3. As stated in appendix B, we use the best-practice procedures from Gepperth&Pfülb(2022) for choosing the hyperparameters of the GMM. In this reference, it is shown through experimentation that K follows a "the more the better" principle. This is also stated in appendix B, but we will include this info and a citation in the main text.

Reviewer comment (Presentation):
It seems that the main text of this paper is over the limit of 9 pages, where Section 5.1 is in page 10.

Author response:
5. It is clearly stated in the ICLR 2024 author's guidelines that the reproducibility statement does not count towards the page limit. Please be sure to take this into account in your evaluation.

Reviewer comment (Theory):
Concern about the proposed method AR when $K$ is greater than the number of seen classes: One component of GMM corresponds to one distribution of several classes. When $K>$ the number of seen classes, is it possible to choose an unknown component at the query step and what do the generated samples look like in this scenario? It seems that the ability of preventing forgetting is determined by $K$. When there are so many classes whose number $\gg K$, it is hard to prevent forgetting due to the overlapping. If this understanding is wrong, please correct it.

Author response:
(3a) At every task, all components are used, so it is not possible to choose an unknown/unconverged component. For each subsequent task, components will be partially re-allocated to describe new classes.

(3b) We do not preallocate components to certain classes (this is how we interpret your question,did we understand you correctly?). Neither do we make any other assumptions regarding classes other than the maximal number of classes that will occur. This is a common assumption that all other works make, even [4]. Essentially, this information is needed to determine the size of the single-head classifier (which we use as well, not a multi-head one).
Component re-allocation for each new task is a dynamic process where AR adapts those components which are most similar to the new tasks' data. And leaves the ones that are dissimilar unchanged. However, components are not simply overwritten but must now interpolate between past and new knowledge.

Reviewer comment (Experiments):
There are only two old replay methods (DGR and ER). It is better to compare the proposed method with more and newer replay baselines.

Author response:
1. again, we would be grateful for references here. There are many variants of generative replay, some are very specific to certain settings.

Reviewer comment (Experiments):
The potential low time cost of the proposed method is a significant advantage. It is better to demonstrate this by more time cost experiments.

Author response:
2. We will include a table with runtime measurements. The time cost will be basically proportional to the nr. of samples in Fig. 4.

Reviewer comment (Experiments):
The split tasks $T_2, T_3, \dots$ contain only one class. How does the proposed method perform if there are more classes in one task?

Author response:
3. There is no problem at all with more than one class per task. We will include splitMNIST and splitCIFAR (5-fold split with two classes per split) experiments to demonstrate this.

Reviewer comment (Experiments):
The baseline numbers are very weak, which raises concerns about the empirical rigor of the work. For example, deep generative replay with a VAE of similar complexity in MIR [1] gets 80% on split mnist, *trained online* with 2 classes per task, which is arguably a much more difficult setting than the one in the paper. On that note, the paper would benefit from including this baseline in the paper.

Author response:
2.Our issue with the constant-time settings used in other approaches is that samples are replayed indiscriminately, even though the number of samples remains constant. So, over time, the number of replayed samples for each past task will go to 0, at which point forgetting is ensured.

Reviewer comment (Novelty):
To the best of my understanding, the conceptual differences with MIR are small: the idea of fetching points close in the VAE's latent space is exactly what MIR does. In other words, AR is similar to Gen-MIR where 0 gradient ascent steps to maximize interference are done.

Author response:
3. The key difference to MIR is selective updating, which the VAEs in GEN-MIR cannot do. They have to be trained with a balanced dataset. The idea of selective replay is of course quite similar, although our method does not require gradient descent to do it.

Reviewer comment (Writing):
The paper could benefit from some tweaks in the main paper. For example, it is unclear at which level latent replay operates; adding a detailed figure would greatly aid understanding. Moreover, I think the authors' interpretation of Foundation Models are merely any pretrained model, rather than a "generalist" model trained on vast, diverse data.

Author response:
- adapted our use of the term "foundation models" in favor of pre-trained feature extractors

Reviewer comment (Reproducibility):
What is the buffer size used for ER?

Author response:
Concerning your questions:
1. Buffer size is 50

Reviewer comment (Experiments):
When is latent replay (vs standard replay) performed? It seems that only SVHN and CIFAR use latent replay?

Author response:
Concerning your questions:
2. Correct, latent replay is done for CIFAR and SVHN.