Paper: Meta Continual Learning Revisited: Implicitly Enhancing Online Hessian Approximation via Variance Reduction (ICLR-2024)

Reviewer comment (Theory):
In Proposition 3, the author assumes that the batch size for inner step adaptation is sufficiently large. How do we quantify the term "sufficiently large" in reality? Is there any principle we can obtain from the proposed theorem to guide us in choosing the batch size?

Author response:
Thank you sincerely for your thoughtful and positive feedback on our work. We are particularly grateful for your recognition of the various aspects of our research. Below, we have provided a detailed explanation for your remaining concern as follows. Please do not hesitate to let us know if you have any further questions.

 **Q1**: How to quantify the "sufficiently large" inner batch size? Is there any principle we can obtain from the proposed theorem.
 > - We made this assumption of a sufficiently large batch size for inner step adaptation to (1) **guarantee the accurately estimated inner gradients** given the number of inner steps $K=1$ in Proposition 3, and thus (2) facilitate the proof in Appendix A.3.2.
 > - In practical applications where $K>1$, as discussed in Appendix A.2, the updating rule incorporates the average of inner gradients over all $K$ steps. This averaging mechanism **alternatively ensures the accuracy of inner gradient estimation, thereby alleviating the necessity for an excessively large batch size**.
 > - We validate the above analyses with two groups of experiments.
 >   - We compare the performance of (1) VR-MCL with $K=8$ and an inner batch size of 4 and (2) VR-MCL with $K=4$ and an inner batch size of 8. The results are presented in the table below. The comparable performance of the two cases supports the notion that either a large batch size or a large number of inner steps suffices.  
>
>| Inner batch size | Case (1)  |        Case (2)           |  
>|:----------------:|:---:|:--------------------:|  
>|   Acc      |  56.16 $\pm$ 2.18   | 56.48 $\pm$ 1.79 |   
>|   AAA      |  66.42 $\pm$ 2.06   | 66.97 $\pm$ 1.58 |   
>
>    - - On Seq-CIFAR10, we conduct an ablation study of VR-MCL by **fixing the inner steps to $K=4$** and  evaluating two other inner batch sizes of 6 and 10. As anticipated, a larger inner batch size indeed contributes to improved performance, while the improvement brought by increasing the inner batch size tends to plateau. This is attributed to the nature of the online setting, where all samples are seen only once. As a result, a larger inner batch size leads to a reduced number of outer loop steps, potentially compromising the performance gain. 
>
>| Inner batch size |  6  |          8           | 10  |
>|:----------------:|:---:|:--------------------:|:---:|
>|       Acc        | 53.80 $\pm$ 2.36   | 56.48 $\pm$ 1.79 |  56.81 $\pm$ 1.07   |
>|       AAA        | 65.73 $\pm$ 3.06   | 66.97 $\pm$ 1.58 |  67.26 $\pm$ 0.61  |

Reviewer comment (Theory):
I wonder whether the assumptions during the mathematical derivation always hold in the practical scenarios. For example: In Proposition 2, the authors assumed that $\theta_{(K)}$ is located in the $\epsilon$-neighbourhood of the optimal model parameter. Is it too strong?

Author response:
**Q1**: Whether the assumptions during the mathematical derivation hold in the practical scenarios?
 > - **The assumption of $\theta_{(K)}$ in Proposition 2.**  
 >   - The optimal model $\theta^*=\arg\min_\theta\mathcal{L}^{[1:j]}(\theta_{(K)})$ within the framework of Meta-CL methods [1], as defined in Proposition 2, is the one that minimizes the loss over all seen tasks in the outer loop, i.e., $\mathcal{T}^1,\mathcal{T}^2,...,\mathcal{T}^j$. Thus, $\theta^*$ is task-dependent, signifying that **the introduction of a new $j$-th task corresponds to a new $\theta^{*}$**.
 >   - $\theta_{(K)}=U_K(\theta;\mathcal{T}^j)$ as defined in Eqn. (3) represents the parameter at the the $K$-th step optimized toward the current task $\mathcal{T}^j$ in the inner loop.
 >   - The assumption that $\theta_K$ resides in the $\epsilon$-neighborhood of $\theta^*$ is grounded on our **empirical observation**. Our experiments show that for each $j$-th task, the loss value of  $\arg\min_\theta\mathcal{L}^{[1:j]}(\theta_{(K)})$ becomes small within an average of 5 outer loop steps, indicating a close proximity between $\theta_{(K)}$ and $\theta^*$. Further details regarding this empirical finding are provided in Appendix F.2. 
 >   - We attribute this proximity as the recursive nature inherent in the above optimization process in the context of continual learning. Starting with $\theta^{j-1}$, which minimizes $\mathcal{L}^{[1:j-1]}(\theta_{(K)}^{j-1})$ in the $(j-1)$-th task, $\theta^K$ optimized for task $j$ swiftly approaches the local minimum of $\mathcal{L}^{[1:j]}(\theta_{(K)})$. This fast convergence can be elucidated by decomposing $\mathcal{L}^{[1:j]}(\theta_{(K)})$ into $\mathcal{L}^{[1:j-1]}(\theta_{(K)}) + \mathcal{L}(\theta_{(K)})$, where $\mathcal{L}^{[1:j-1]}(\theta_{(K)}^{j-1})$ has been optimized before.  
 > [1] Look-ahead meta learning for continual learning. NeurIPS 2020.

Reviewer comment (Theory):
In Proposition 3, the authors assumed that the batch size of the inner step adaptation is sufficiently large. I wonder how large is enough to make the following analyses hold. And how did the authors set it during the practical training?

Author response:
**Q1**: Whether the assumptions during the mathematical derivation hold in the practical scenarios?
 > - **The sufficiently large batch size in Proposition 3.**  
 >   - This assumption of a sufficiently large inner batch size **ensures that the inner gradients of the current task $\mathcal{T}_j$ are highly accurate and stable**, given that Proposition 3 considers the number of inner steps $K$ as 1. Accurate inner gradients facilitate subsequent analyses (i.e.,$\nabla_\theta \mathcal{L}^{j}(\theta_2)\approx \nabla_\theta \overline{\mathcal{L}}^{j}(\theta_{1}) \approx \nabla_\theta \mathcal{L}^{j}(\theta_{1})$), as demonstrated in the proof in Appendix A.3.2.  
 >   - In practice, we adopt multiple inner steps, specifically $K=4$, aligning with the extended version of Proposition 3 (details provided in Appendix A.2). In this version, the prerequisite of Proposition 3, namely accurate inner gradients, is readily fulfilled by averaging over multiple steps. 
 >   - We have further **explored the practical impact of the inner batch size** by evaluating alternative settings, specifically considering inner batch sizes of 6 and 10. From the following table, we conclude that
 >     - a larger inner batch size indeed yields improved performance, aligning with the aforementioned analysis;
 >     - the improvement brought by increasing the inner batch size tends to plateau. This is attributed to the nature of the online setting, where all samples are seen only once. As a result, a larger inner batch size leads to a reduced number of outer loop steps, potentially compromising the performance gain.
>
> | Inner batch size |  6  |          8           | 10   |
> |:----------------:|:---:|:--------------------:| --- |
> |       Acc        | 53.80 $\pm$ 2.36   | 56.48 $\pm$ 1.79 |  56.81 $\pm$ 1.07   |
>|       AAA        | 65.73 $\pm$ 3.06   | 66.97 $\pm$ 1.58 |  67.26 $\pm$ 0.61  |

Reviewer comment (Evaluation):
The motivation for the evaluations under the imbalance CL setting was not clear to me. I did not get the relationship between the superior performance under this setting and the main objective of this paper. Or does the author just intend to show that the proposed method could still perform well under this challenging setting? Besides, it was disappointing to see that the authors did not provide further analyses about why the proposed method could address this challenging setting.

Author response:
**Q2**: The motivation for the evaluations under the imbalanced CL setting.
> - As acknowledged by the reviewer in the Summary, our main objectives include addressing "the erroneous information during the Hessian estimation due to the sampling process from the random memory buffer". This is achieved through the proposed VR-MCL, which effectively "controls the high variance of the hypergradient under online continual learning".
> - In the imbalanced CL setting, a memory buffer $\mathcal{M}$ constructed using the common practice of reservior sampling strategy stores **imbalanced samples across different tasks** [2]. Compared to online continual learning, this imbalance in $\mathcal{M}$ further results in a **less accurate Hessian estimation with higher variance**.
> -  Thus, we conduct the evaluations under this challenging imbalanced CL setting to specifically validate the effectiveness of VR-MCL, examining whether we achieve our main objective of controlling the high variance. Our results in Table 4 affirmatively confirm this efficacy. 
> - We appreciate the reviewer for pointing this out, and have incorporated this analysis into the main text (Sec. 5 Q3) for better understanding.

Reviewer comment (Theory):
In Proposition 2, the authors mentioned the assumption of $\beta$. However, it was not contained in the final main conclusion.

Author response:
**Q3**: The notation of $\beta$ is not appear in the final main conclusion of the Proposition 2.
 > To facilitate a transparent comparison between the iterative update rule presented in Proposition 2 and others showcased in Table 1, we adopt $\alpha=\beta^2$ in the main conclusion of Proposition 2. The related proof is provided in Appendix A.2. We appreciate the reviewer for bring this to our attention and have included this clarification in Proposition 2 to enhance readability.

Reviewer comment (Evaluation):
How about the time and memory complexity of the proposed method compared to the baseline approaches, especially the Meta-CL methods, like LA-MAML? Could the authors provide quantitative comparisons? I believe such a comparison will help the readers to better understand the superiority of the proposed VR-MCL.

Author response:
**Q5**: Time and memory complexity analysis of VR-MCL and other Meta-CL methods.
 > - We have provided the **time complexity analysis** of VR-MCL and other Meta-CL methods, like MER and La-MAML, in **Appendix D.1. (Table 9)**. For ease of illustration, the results are also summarized in the table below. 
 >   - The analysis clearly demonstrates that VR-MCL significantly reduces training time compared to MER (approximately 93.7% reduction in training time) and requires slightly longer training time compared to La-MAML. 
 >   - To assess the value of the additional training time incurred by VR-MCL, we extended the training epoch of La-MAML from 1 to 2. Despite having an even longer training time than VR-MCL, La-MAML with 2 epochs only achieved a performance of 38.89%, which is extremely lower than the performance of VR-MCL (i.e., 56.48%). This outcome serves as compelling evidence for the superiority of VR-MCL in balancing between training efficiency and performance.
 > 
> |     Method     | La-MAML | MER |  VR-MCL    |  
> |:--------------:|:--------------:|:--------------:|:---:| 
> |   Training Epochs  |  1   |   1   |  1   |  
> | Training Time (s) |  750.61   |   20697.7   |  1297.10    | 
> | | | | | 
> |   Training Epochs  |  2   |   1   |  1   |  
> | Training Time (s) |  1511.54   |   20697.7   |  1297.10    | 
> | AAA |  53.21%   |   50.99%   |  66.97%   | 
> | Acc |  38.89%   |   36.92%   |  56.48%   | 
>
> - Regarding **memory complexity**, we compared the memory usage of three methods, including MER, La-MAML, and VR-MCL, using the PcCNN network with $|\mathcal{M}|$=600 when training on Split-CIFAR10. The results are shown in the table below. 
>    - Notably, VR-MCL only requires 8.8% more memory than La-MAML, yet it achieves a performance improvement of 16.07% and 35.3% over Acc and AAA metrics, respectively. This demonstrates the superior "effective" efficiency of VR-MCL.
> 
> |   Method   | La-MAML |   MER   | VR-MCL  |
> |:----------:|:-------:|:-------:|:-------:|
> | Memory(MB) |    1462    |    1306    |    1594    |
> |    Acc     |   32.78 $\pm$ 1.53      |    44.69 $\pm$ 0.43     |   **48.85** $\pm$ 0.66     |
> |    AAA     | 47.22 $\pm$ 0.87  | 63.20 $\pm$ 0.43 | **63.86** $\pm$ 0.52 |

Reviewer comment (Experiments):
The experiments could be enhanced by varying the number of tasks on each dataset, thereby showcasing the adaptability of VR-MCL under different task configurations.

Author response:
**Q3**: Additional experiments under scenarios involving varying task lengths across datasets.
> - First, we would like to humbly emphasize that **our experiments have already covered scenarios with diverse task lengths**. Specifically, the Seq-CIFAR10 dataset contains 5 tasks, Seq-CIFAR100 includes 10 tasks, and Seq-TinyImageNet has 20 tasks. The results demonstrate that VR-MCL consistently surpasses other baselines across the three datasets, notwithstanding the variations in task lengths. 
> - To further address the reviewer's concern regarding catastrophic forgetting under longer task sequences, we conduct additional experiments, as suggested, wherein we **configure each dataset with varying task lengths** to validate the effectiveness of VR-MCL. 
>   - Concretely, we reconfigure Seq-CIFAR100, originally composed of 10 tasks with 10 classes each, into 20 tasks, each encompassing 5 classes. Similarly, we adjust Seq-TinyImageNet, initially consisting of 20 tasks with 10 classes each, into 40 tasks, each comprising 5 classes. For clarity, we hereby denote the two newly split datasets as Seq-CIFAR100$^l$ and Seq-TinyImageNet$^l$.
>   - We compare the performance of the proposed VR-MCL against ER, GEM, A-GEM, ER-OBC, DER++, La-MAML, and ClsER, with the results listed below (also **included into Appendix F (Table 18)**). Despite a general trend of performance degradation across all methods in the extended task length setting, the proposed VR-MCL **consistently outperforms** the other baselines over both *AAA* and *Acc* metrics.
>   - The standard practice in the existing continual learning literature [7,8] that involves ImageNet-1k is to divide it into 10 tasks -- a number even fewer than that of Seq-TinyImageNet. We have re-configured ImageNet-1k into 50 tasks with 200 classes each, and will update the corresponding results shortly.
>
>|     Method     | Seq-CF100$^l$ | Seq-CF100$^l$ |  Seq-Tiny$^l$    | Seq-Tiny$^l$ |
>|:--------------:|:--------------:|:--------------:|:---:|:----------------:|
>|                |  AAA   |   Acc   |  AAA   | Acc        |
>| SGD |  9.23$\pm$ 0.26   |   3.34$\pm$ 0.13   |  4.95$\pm$ 0.16       | 1.19$\pm$ 0.27  |
>| A-GEM |  10.84$\pm$ 0.23   |   3.56$\pm$ 0.17   |  6.10$\pm$ 0.14    | 1.67$\pm$ 0.07  |
>| GEM |  16.04$\pm$ 2.28   |   7.01$\pm$ 1.95   |  7.92$\pm$ 0.15      | 2.93$\pm$ 0.38  |
>| ER |  20.46$\pm$ 0.48   |   12.71$\pm$ 0.28   |  13.75$\pm$ 0.12     | 6.82$\pm$ 0.11  |
>| DER++ |  16.32$\pm$ 0.49  |    8.24$\pm$ 0.41    |9.39$\pm$ 0.07     |  3.76$\pm$ 0.81 |
>| CLSER |  22.03$\pm$ 0.96  |    15.39$\pm$ 2.36   |14.93$\pm$ 0.36    |  7.74$\pm$ 0.91 |
>| ER-OBC |  21.04$\pm$ 0.65  |    15.87$\pm$ 1.26   |14.92$\pm$ 0.20   |  8.45$\pm$ 2.29 |
>| La-MAML |  17.42$\pm$ 0.79  |    10.27$\pm$ 0.46  |10.83$\pm$ 0.59   |  5.29$\pm$ 0.28 |
>| VR-MCL |  **24.29$\pm$ 1.07**  |   **17.44$\pm$ 0.97**   |**18.28$\pm$ 0.14**   |  **10.54$\pm$ 0.30**|
>
>[7] How efficient are today's continual learning algorithms? CVPR workshop 2023.
>[8] A simple but strong baseline for online continual learning: repeated augmented rehearsal. NeurIPS 2022.

Reviewer comment (Reproducibility):
Regarding the algorithm, the paper mentions that the memory buffer is updated to ensure a balanced storage of tasks. Could you provide more details on how this task-balancing process is implemented within the algorithm?

Author response:
**Q1**: More details on the update mechanism of the memory buffer $\mathcal{M}$.
> - In rehearsal-based CL methods, the memory buffer $\mathcal{M}$ serves as a repository for storing samples from previous tasks. During sequential training, samples are sampled from $\mathcal{M}$ and combined with those from the new task in a joint training process to mitigate forgetting. The proposed VR-MCL **aligns with established rehearsal-based CL methods** [1,2,3] by employing the **reservoir sampling strategy** to update the samples stored in $\mathcal{M}$.
> - This strategy updates $\mathcal{M}$ to ensure that the stored examples are uniformly sampled from the tasks during online training. Under this updating scenario, 
>   - (1) when the online training tasks have an equal number of samples, the memory buffer $\mathcal{M}$ will contain balanced samples from different tasks.  
>   - (2) if the number of samples varies across tasks, the memory buffer $\mathcal{M}$ will store imbalanced samples across different tasks [4]. 
>   - Since the majority of experiments in this paper adhere to the standard setting where all online tasks have an equal number of samples (setting (1) above), we stated that the memory buffer is updated to maintain balanced samples from different tasks. 
>  - Our contributions do not entail a novel update mechanism for the memory buffer. Instead, the proposed variance reduction exactly alleviates inaccurately estimated $\mathbf{H}^j_M$, a potential outcome of imbalanced samples within $\mathcal{M}$ in setting (2).
>  - Thanks for pointing this out, and in response, we have revised the corresponding expression in Algorithm 1 (Appendix E) and provided additional explanations to eliminate any potential ambiguity.
> 
> [1] Experience replay for continual learning. NeurIPS 2019. 
>
> [2] Dark experience for general continual learning: a strong, simple baseline. NeurIPS 2020. 
>
> [3] Learning fast, learning slow: a general continual learning method based on complementary learning system. ICLR 2021.
>
> [4] Online continual learning from imbalanced data. ICML 2020.

Reviewer comment (Experiments):
It would be valuable to include additional experiments as mentioned earlier, especially those assessing the method's performance under scenarios involving varying task lengths across each datasets.

Author response:
**Q3**: Additional experiments under scenarios involving varying task lengths across datasets.
> - First, we would like to humbly emphasize that **our experiments have already covered scenarios with diverse task lengths**. Specifically, the Seq-CIFAR10 dataset contains 5 tasks, Seq-CIFAR100 includes 10 tasks, and Seq-TinyImageNet has 20 tasks. The results demonstrate that VR-MCL consistently surpasses other baselines across the three datasets, notwithstanding the variations in task lengths. 
> - To further address the reviewer's concern regarding catastrophic forgetting under longer task sequences, we conduct additional experiments, as suggested, wherein we **configure each dataset with varying task lengths** to validate the effectiveness of VR-MCL. 
>   - Concretely, we reconfigure Seq-CIFAR100, originally composed of 10 tasks with 10 classes each, into 20 tasks, each encompassing 5 classes. Similarly, we adjust Seq-TinyImageNet, initially consisting of 20 tasks with 10 classes each, into 40 tasks, each comprising 5 classes. For clarity, we hereby denote the two newly split datasets as Seq-CIFAR100$^l$ and Seq-TinyImageNet$^l$.
>   - We compare the performance of the proposed VR-MCL against ER, GEM, A-GEM, ER-OBC, DER++, La-MAML, and ClsER, with the results listed below (also **included into Appendix F (Table 18)**). Despite a general trend of performance degradation across all methods in the extended task length setting, the proposed VR-MCL **consistently outperforms** the other baselines over both *AAA* and *Acc* metrics.
>   - The standard practice in the existing continual learning literature [7,8] that involves ImageNet-1k is to divide it into 10 tasks -- a number even fewer than that of Seq-TinyImageNet. We have re-configured ImageNet-1k into 50 tasks with 200 classes each, and will update the corresponding results shortly.
>
>|     Method     | Seq-CF100$^l$ | Seq-CF100$^l$ |  Seq-Tiny$^l$    | Seq-Tiny$^l$ |
>|:--------------:|:--------------:|:--------------:|:---:|:----------------:|
>|                |  AAA   |   Acc   |  AAA   | Acc        |
>| SGD |  9.23$\pm$ 0.26   |   3.34$\pm$ 0.13   |  4.95$\pm$ 0.16       | 1.19$\pm$ 0.27  |
>| A-GEM |  10.84$\pm$ 0.23   |   3.56$\pm$ 0.17   |  6.10$\pm$ 0.14    | 1.67$\pm$ 0.07  |
>| GEM |  16.04$\pm$ 2.28   |   7.01$\pm$ 1.95   |  7.92$\pm$ 0.15      | 2.93$\pm$ 0.38  |
>| ER |  20.46$\pm$ 0.48   |   12.71$\pm$ 0.28   |  13.75$\pm$ 0.12     | 6.82$\pm$ 0.11  |
>| DER++ |  16.32$\pm$ 0.49  |    8.24$\pm$ 0.41    |9.39$\pm$ 0.07     |  3.76$\pm$ 0.81 |
>| CLSER |  22.03$\pm$ 0.96  |    15.39$\pm$ 2.36   |14.93$\pm$ 0.36    |  7.74$\pm$ 0.91 |
>| ER-OBC |  21.04$\pm$ 0.65  |    15.87$\pm$ 1.26   |14.92$\pm$ 0.20   |  8.45$\pm$ 2.29 |
>| La-MAML |  17.42$\pm$ 0.79  |    10.27$\pm$ 0.46  |10.83$\pm$ 0.59   |  5.29$\pm$ 0.28 |
>| VR-MCL |  **24.29$\pm$ 1.07**  |   **17.44$\pm$ 0.97**   |**18.28$\pm$ 0.14**   |  **10.54$\pm$ 0.30**|
>
>[7] How efficient are today's continual learning algorithms? CVPR workshop 2023.
>[8] A simple but strong baseline for online continual learning: repeated augmented rehearsal. NeurIPS 2022.