Paper: Adaptive Continual Learning: Rapid Adaptation and Knowledge Refinement (ICLR-2024)

Reviewer comment (Presentation):
Figure 1 is too abstract and symbolic to be understood.

Author response:
### Q: “Figure 1 is too abstract”  
A: Thanks for your feedback. We agree that a concrete algorithm illustration may better convey our intuition. Therefore, we have revised Figure 1 to provide a more detailed and concrete illustration of our algorithm, including its key components of one fast model, multiple slow models, and a router that keeps a mixing weight. Our algorithm is designed to let the router (1) assign a high weight to the relevant slow model when a prior task reoccurs, or to the fast model when a new task arrives; (2) update the corresponding model without information overwrite – namely one model already mastered one task continues to learn another task.

Reviewer comment (Writing):
Tempered Bayesian updates is a key component of the fast learner, but is not introduced.

Author response:
### Q: “Figure 2...without further explanation” “Tempered Bayesian updates … not introduced.”  
A: Thanks for pointing out. To enhance clarity, we improve the explanations of Figure 2 in Section 3.1: “As illustrated in Figure 2, when the task switches to a previously learned task, a direct update without recall attempts to increase a small mass around the optimal parameter. However, once mixed with the slow density, that is, recall, the mass around the optimal parameter increases, resulting in faster recognition of the learned task.”  

We include a brief introduction of tempered Bayesian updates in Section 3.1. “The introduction of the temperature parameter $\eta$ is designed to moderate the impact of varying loss scales, especially since our approach does not rely on a probabilistic setting or the use of negative log-likelihood loss.”

Reviewer comment (Presentation):
Important details, such as the derivation of Algorithm 2 and the definition of Adaptiveness are provided in Appendix, making it difficult to understand when reading the main text.

Author response:
### Q: “ proof of Lemma A.2. applies Hoeffding's lemma directly, but the derivation is not straightforward”  “ derivation of Algorithm 2 and the definition of Adaptiveness are provided in the Appendix, making it difficult to understand...”  

A: We are grateful for these suggestions. In the updated manuscript, we include the full derivation in the proof of Lemma A.2, and more details in other proofs in Appendix A. We also add key steps of the derivation of Algorithm 2 in Section 3.1, and definitions of three evaluation metrics: Average accuracy, Knowledge accuracy, and Adaptiveness in Metric, Section 4.1, to improve clarity.

Reviewer comment (Theory):
There is a discrepancy between the motivation and the algorithm. As mentioned, "The primary goal is to activate the relevant slow learner for improved performance on seen tasks, and to utilize the fast learner for identifying and quickly learning new tasks." However, there is no new task identification in the proposed algorithms.

Author response:
## Discrepancy between the motivation and the algorithm  
A: Thanks for the excellent question. We want to emphasize that Algorithm 1 implicitly recalls and identifies task identity, and scalable Algorithm 2 converts Algorithm 1 to a direct recall, identification, and refinement by Gaussian Mixture Model (GMM) approximation.  
To better explain the connection between motivation and algorithm, we have moved key steps in the derivations in Appendix to Section 3.3. Intuitively, the slow density in Algorithm 1 is approximately a mixture density – a weighted sum of several components, each component represents information of one task. Here, (1) rapid task identification refers to a quick increase of mixing weight of corresponding task components; (2) recall refers to the process of fast density mixed with slow density – with such a mixing, it is easier to pull up the weight of the prior task component; (3) refinement refers to update of slow density.

Reviewer comment (Reproducibility):
There is no identification of relevant slow learners in Algorithm 1.

Author response:
## Discrepancy between the motivation and the algorithm  
A: Thanks for the excellent question. We want to emphasize that Algorithm 1 implicitly recalls and identifies task identity, and scalable Algorithm 2 converts Algorithm 1 to a direct recall, identification, and refinement by Gaussian Mixture Model (GMM) approximation.  
To better explain the connection between motivation and algorithm, we have moved key steps in the derivations in Appendix to Section 3.3. Intuitively, the slow density in Algorithm 1 is approximately a mixture density – a weighted sum of several components, each component represents information of one task. Here, (1) rapid task identification refers to a quick increase of mixing weight of corresponding task components; (2) recall refers to the process of fast density mixed with slow density – with such a mixing, it is easier to pull up the weight of the prior task component; (3) refinement refers to update of slow density.

Reviewer comment (Theory):
In Algorithm 1, the recall and refinement do not seem to leverage the knowledge of the recurrence of tasks. They are both updated by simply combining $\tilde{f}$ and $g$ with predefined ratios.

Author response:
## Discrepancy between the motivation and the algorithm  
A: Thanks for the excellent question. We want to emphasize that Algorithm 1 implicitly recalls and identifies task identity, and scalable Algorithm 2 converts Algorithm 1 to a direct recall, identification, and refinement by Gaussian Mixture Model (GMM) approximation.  
To better explain the connection between motivation and algorithm, we have moved key steps in the derivations in Appendix to Section 3.3. Intuitively, the slow density in Algorithm 1 is approximately a mixture density – a weighted sum of several components, each component represents information of one task. Here, (1) rapid task identification refers to a quick increase of mixing weight of corresponding task components; (2) recall refers to the process of fast density mixed with slow density – with such a mixing, it is easier to pull up the weight of the prior task component; (3) refinement refers to update of slow density.

Reviewer comment (Novelty):
This paper does not discuss and compare the methods proposed for the very related setting where data of previous tasks/classes may appear again in CL, such as blurry task setting like "Koh et al. Online continual learning on class incremental blurry task configuration with anytime inference. In ICLR 2022", or the methods using fast and slow learner "Pham et al. DualNet: Continual Learning, Fast and Slow. In NeurIPS 2021".

Author response:
### Q: “compare … very related setting where data of previous tasks/classes may appear again in CL…”  
A: We appreciate the concerns. During the rebuttal stage, we add methods CLS-ER[3], CLIB [5], and DualNet [6]. In summary, these methods exibit less satisfactory performance in the results attached at the end. While this setting seems to be similar, we hope to emphasize that papers such as [1,2] focus on changing class distribution, where the key challenge is to handle possible class imbalance. However, our proposed scenario considers possibly recurring tasks, highlighting the need to rapidly recognize seen tasks and refine corresponding information.

Reviewer comment (Experiments):
In Table 2, the only task-free baseline performs much better than the proposed method. The proposed method should be compared with more task-free baselines.

Author response:
### Q: “What are Average Accuracy and Knowledge Accuracy…”  
“in Table 2, the only task-free baseline outperforms…”  

A: Thanks for the question. For better clarity, we have included the formulas for Average Accuracy and Knowledge Accuracy both in this rebuttal and the revised manuscript Average Accuracy measures the cumulative performance over time, and Knowledge Accuracy measures the information retained only at the last timestep, assuming task identity is known. Therefore, the method such as ExpVAE, which separately trains a classifier per task, has a high Knowledge Accuracy, while poor Average Accuracy since inference by VAEs is inaccurate.  
$
\text{AverageAccuracy} = \frac{1}{T}\sum_{t=1}^T acc_t, \quad
\text{Knowledge Accuracy} = \frac{1}{N} \sum_{i=1}^N acc_{T,i},$
where $acc_t$ is the accuracy at time $t$, and $acc_{T,i}$ is the test accuracy of task $i$ assuming task identity is known at the final time.  

We want to emphasize that as indicated in Compared methods, Section 4.1, the baseline ExpVAE requires task identity during training, which separately trains a pair of generator and classifier for each task. In the inference stage, the generator is used to choose the correct classifier. ExpVAE serves as an upper bound on how such a structure of pairs of generators and classifiers will behave, even if task identities are known during training. As seen from the average accuracy in Table 1, inference from generator is inaccurate. Table 2 evaluates the final average test accuracy of the best classifier per task. Therefore the Knowledge accuracy of ExpVAE behaves similarly to Oracle which employs a multi-head ResNet with shared feature extractor, while during inference generators fail to correctly recall the correct classifier.

Reviewer comment (Experiments):
The randomness introduced by the random shuffling of 200 segments can have a significant impact on experimental performance.

Author response:
### Q: “random shuffling of 200 segments can have a significant impact on experimental performance.”  
A: Thanks for the insightful point. In order to avoid performance being significantly affected by randomness, we repeated experiments for 10 runs and reported the standard error. Since the standard error is moderate, we believe the reported results reflect the performance.

Reviewer comment (Evaluation):
What are the definitions of Average Accuracy and Knowledge Accuracy? Can you provide their definition similar to the one of adaptiveness in Appendix D.4? In addition, why do the methods perform very differently in terms of knowledge accuracy and average accuracy?

Author response:
### Q: “What are Average Accuracy and Knowledge Accuracy…”  
“in Table 2, the only task-free baseline outperforms…”  

A: Thanks for the question. For better clarity, we have included the formulas for Average Accuracy and Knowledge Accuracy both in this rebuttal and the revised manuscript Average Accuracy measures the cumulative performance over time, and Knowledge Accuracy measures the information retained only at the last timestep, assuming task identity is known. Therefore, the method such as ExpVAE, which separately trains a classifier per task, has a high Knowledge Accuracy, while poor Average Accuracy since inference by VAEs is inaccurate.  
$
\text{AverageAccuracy} = \frac{1}{T}\sum_{t=1}^T acc_t, \quad
\text{Knowledge Accuracy} = \frac{1}{N} \sum_{i=1}^N acc_{T,i},$
where $acc_t$ is the accuracy at time $t$, and $acc_{T,i}$ is the test accuracy of task $i$ assuming task identity is known at the final time.  

We want to emphasize that as indicated in Compared methods, Section 4.1, the baseline ExpVAE requires task identity during training, which separately trains a pair of generator and classifier for each task. In the inference stage, the generator is used to choose the correct classifier. ExpVAE serves as an upper bound on how such a structure of pairs of generators and classifiers will behave, even if task identities are known during training. As seen from the average accuracy in Table 1, inference from generator is inaccurate. Table 2 evaluates the final average test accuracy of the best classifier per task. Therefore the Knowledge accuracy of ExpVAE behaves similarly to Oracle which employs a multi-head ResNet with shared feature extractor, while during inference generators fail to correctly recall the correct classifier.

Reviewer comment (Reproducibility):
Algorithm 2 needs to maintain 1+$m_{t-1}$ models. Why does it have almost the same number of trainable parameters as the other methods?

Author response:
### Q: “Algorithm 2 needs to maintain $1+m_t$ models…almost the same number of trainable parameters as the other methods”  

A: We appreciate your observation. As mentioned in Architecture, Section 4.1, our proposed method and Oracle employ a reduced ResNet18 with a shared feature extractor. Therefore the excessive $m_t$ output heads contribute a small number of parameters compared with the feature extractor.

Reviewer comment (Reproducibility):
We do not know the number of distributions. Therefore we don't know $m_{t-1}$. How is $m_{t-1}$ obtained in Algorithm 2?

Author response:
### Q: “How is $m_t$ obtained in Algorithm 2”  
A: Thanks for the excellent question. We want to clarify that slow learners are sequentially added once patience is above a threshold, and $m_t$ is the number of slow learners maintained at time $t$, which is not the underlying truth. To avoid confusion, we have revised the notation to $\hat{m}_t$ in the manuscript.

Reviewer comment (Evaluation):
In the experiment, how many slower learners are there at the end of training?

Author response:
### Q: “How many slower learners are there at the end of training”  
A: Thanks for the helpful suggestion. Along with the maximum trainable parameters of our proposed algorithm reported in Table 4 (Section 4.1), the final count of slow learners at the end of training is detailed in Table 4, attached to the rebuttal.

Reviewer comment (Experiments):
The method is verified on the classification task, as the problem setting is basically constructed according to label. The reviewer is wondering whether it is also applicable for regression problem. A related question is how to formulate the problem setting for regression task?

Author response:
Q: “whether it is also applicable for regression problem”

A: Thanks for the insightful point. Although our experiments focused on classification tasks, a primary focus in computer vision, the proposed algorithm is applicable to any supervised learning problems including regression. In the regression case, the label is output which is usually a scalar and loss is typically the mean square error. The exact theoretical guarantees for the supervised case in Section 3 directly hold for such a scenario.

Reviewer comment (Presentation):
In Figure 1, the meaning of the y-axis and the star symbol is unclear or this figure is just an illustration figure? It's also confusing why the third Refinement (blue lines) occurs before the second Recall (green arrows).

Author response:
A1: Thanks for the helpful suggestion. To address this, we have revised Figure 1 in Section 1 to clearly depict our proposed network structure, showcasing the integration of fast/slow models and the router.

Reviewer comment (Writing):
In Section 3.3, it’s unclear what the `scalability challenges' refer to. In Algorithm 2, the notation $\beta_{t-1,i}$ is also not explained.

Author response:
A: Thanks for pointing out.  The term 'scalability challenges' in our context refers to the computational difficulties of managing probability distributions over a high-dimensional parameter space in previous algorithms. We have updated the expression in Section 3.3 to “However, this approach, which requires density over a high dimensional space, faces scalability challenges in large-scale deep learning tasks.”

Thanks for the excellent critic. Algorithm 2 involves three components at time $t$: (1) fast model $\theta_t$ (2) slow models $\beta_{t-1,i}$ for $i=1,\dots,m_{t-1}$ (3) mixing weights $w_{t,i}$ to combine fast/slow models.
The $m_t$ is the estimated number of models in Algorithm 2. In Algorithm 2, when a fast learner consistently outperforms slow learners, it signals the emergence of a new task, prompting the addition of the fast learner as a new slow model.
To improve clarity, in the updated manuscript, we will (1) move the key GMM approximation steps in the Appendix to Section 3.3, and (2) change $m_t$ to $\hat{m}_t$ to avoid confusion.

Reviewer comment (Theory):
At the time (t+1), it's not clear why the slow learners have the correct $m_t$ models, especially considering that Section 1.1 asserts that the method doesn't require knowledge of the task count. Is $m_t$ also indicative of the number of components in the GMM model? The paper needs to provide a more detailed explanation of how the GMM model is updated and how the relevant slow learner is selected in Figure 3.

Author response:
Thanks for the excellent critic. Algorithm 2 involves three components at time $t$: (1) fast model $\theta_t$ (2) slow models $\beta_{t-1,i}$ for $i=1,\dots,m_{t-1}$ (3) mixing weights $w_{t,i}$ to combine fast/slow models.
The $m_t$ is the estimated number of models in Algorithm 2. In Algorithm 2, when a fast learner consistently outperforms slow learners, it signals the emergence of a new task, prompting the addition of the fast learner as a new slow model.
To improve clarity, in the updated manuscript, we will (1) move the key GMM approximation steps in the Appendix to Section 3.3, and (2) change $m_t$ to $\hat{m}_t$ to avoid confusion.

Reviewer comment (Experiments):
In the experimental section, the methods chosen for comparison are outdated. Considering that the proposed method incorporates the concept of the complementary system and emphasizes task-free CL, it should at least be compared with closely related methods such as Cls-ER[3] and recent task-free methods like [4]. Additionally, the paper should address the storage cost, as the method requires storing multiple slow learners.

Author response:
A: Thanks for your excellent suggestions. During the rebuttal stage, we add methods CLS-ER[3], CLIB [5], and DualNet [6]. Since the focus of [4] is to propose a way to simulate a scenario with changing class distribution, [4] does not propose a new task-free algorithm. We show that these 3 new methods have less satisfactory performance. The results are discussed and attached at the end. 

To better clarify our choice of baseline methods, we want to mention that our selected methods (EWC++, MIR, ER, GDumb, AGEM) align with recent task-free research conventions [4,5,6,7]. These baselines are especially suitable for task-free scenarios because they do not require explicit task identity. For example, [7] used MIR, ER with GMED [8] variant; Similarly, the mentioned [4] used ER, MIR, GDUMB, and GMED, supplemented with methods designed for class imbalance – CBRS (Class-balancing reservoir sampling), ACE. 

### Q: “address the storage cost”

A: Thanks for your feedback. As indicated in Table 4, Section 4.1, the additional storage cost incurred by our algorithm is minimal.  Since we adopt a multi-head ResNet18 with a shared feature extractor, adding more heads introduces a small additional cost compared with the large shared feature extractor

Reviewer comment (Reproducibility):
The actual network of the proposed model is not clear.

Author response:
Thanks for your suggestions. We have updated Figure 1 to clearly illustrate the actual network architecture, comprising a fast model, multiple slow models, and a router that keeps a mixing weight to combine these models. Our algorithm guarantees that the router can (1) assign a high weight to the relevant slow model when a prior task reoccurs, or to the fast model when a new task arrives; (2) update the corresponding model without information overwrite, avoiding the case where a model already mastered one task continue to learn another and forget mastered one.

Reviewer comment (Theory):
Why use the Regret Bound to explain the proposed approach instead of other theories?

Author response:
Thanks for mentioning the choice of regret instead of test accuracy in the end. In the online scenario, it's crucial to assess cumulative performance at each timestep, rather than just the final outcome. Therefore, regret, which measures the difference between the cumulative performance and the cumulative best performance when given the truth, is a standard metric in the literature of online learning.

Reviewer comment (Evaluation):
The proposed approach requires three steps in each time, which leads to huge computational costs.

Author response:
Thanks for the excellent question. The router update, involving merely a one-dimensional vector, is computationally light. The major computations are the backpropagation of the fast model and several slow models. Similar to other methods involving multiple models [1,2], in the experiment, we keep the fast and slow models light – a multi-head ResNet18 with a shared feature extractor, so that the trainable parameters align with the method involving a single network. Given that each model is designed to handle a specific task, employing light models is both efficient and effective.  

[1] Lee, Soochan, et al. "A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning." International Conference on Learning Representations. 2019.  

[2] Ye, Fei, and Adrian G. Bors. "Task-free continual learning via online discrepancy distance learning." Advances in Neural Information Processing Systems 35 (2022): 23675-23688.