Paper: Overcoming the Stability Gap in Continual Learning (ICLR-2024)

Reviewer comment (Experiments):
As different weight initialization techniques tend to be prone to their specific limitations (e.g., a saturation of the activation function, suitability to sigmoid vs softmax activation, etc.), it would be interesting to see how the proposed initialization method stands against other existing ones (Kim's [2] for CL, He's [4] for non-CL) under these circumstances. Such a study could be beneficial to the continual learning community in understanding when (not) to use the proposed init method.

Author response:
Based on our hypothesis, data-driven weight initialization methods that reduce the loss would make the most sense. That said, we did compare against Kaiming (He) initialization, which the reviewer suggests, since it was used by vanilla rehearsal for the output layer throughout experiments. We added this to supplemental materials.

Reviewer comment (Experiments):
The main table of results lacks a proper comparison with the existing literature [3].

Author response:
We strongly disagree. We are aiming to do science and test hypotheses, and Table 3 reports the needed information to do this. For testing our hypotheses, adding many methods that do not use the same pre-trained backbone or keep other variables constant would be scientifically unjustifiable. Moreover, we are not aiming to compete with the SoTA, but to demonstrate the broad applicability of SGM for mitigating the stability gap. 

We did a fair comparison when we combined SGM with other existing CL algorithms e.g., DERpp, GDumb, REMIND, LwF which are widely used CL baselines. All compared methods used the same pre-trained model. We wanted to cover a variety of CL algorithms: 1) rehearsal based: GDumb, 2) rehearsal and knowledge distillation: DERpp, 3) non-rehearsal and knowledge distillation: LwF, 4) online CL: REMIND. Of course there are many more different CL methods but experimenting with all CL methods is intractable. We want to demonstrate that our method mitigates the stability gap and increases computational efficiency in most common CL methods.

Reviewer comment (Theory):
Since LoRA restricts the number of trainable parameters in hidden layers, I believe that it might be susceptible to situations where if the lower layer parameters have been frozen at the earlier tasks, then it might affect the network's capacity to learn new low-level feature compositions for later tasks. This is indeed one of the reasons why freezing lower-layer features is not preferable in continual learning [2]. Can the authors comment on this?

Author response:
Prior work [1-3] found that earlier layer representations are universal and transfer well across tasks. Following prior work, we keep earlier layers frozen. However SGM does not rely on freezing earlier layers. SGM can be applied the same way even if we do not freeze earlier layers and want to update those layers. Regarding the paper that the reviewer cited that points to freezing lower-layer features may not be preferable, we note that the result of the paper they mention may not be broadly applicable outside of the small scale experiments conducted in that paper on CIFAR and MNIST, which is why their conclusions differ from those of others [1-3]. When searching for the paper they mentioned, we note that the ICLR reviews for the paper noted numerous concerns with the paper that do not look to have been addressed in the version accepted to ICIAP.

Reviewer comment (Evaluation):
What concerns me is the efficacy of the proposed SGM method in terms of the computational overhead involved. While the work compares the computational efficiency in terms of the number of iterations required to overcome the stability gap (Figures 1 and 3), there seems to be no mention of the more common measures of computational overhead (e.g. complexity, training time). For a practitioner, a few additional iterations of training might be preferable to a complex training algorithm with a lesser number of iterations yet a significantly longer training time. Can the authors comment on this?

Author response:
We agree that including timing experiments would be helpful beyond network updates. We will provide these numbers in a revised version of the paper in terms of FLOPS and hours.  

R3: Timing analysis is somewhat nuanced. SGM requires very little overhead vs. vanilla; however, the analysis is complicated by the fact that vanilla rehearsal never fully recovers performance on ImageNet regardless of the number of updates and vanilla rehearsal has much lower results on new classes than SGM as well. We are trying to think of the best way to do this analysis.

However, we can say that updating the model in our main results with SGM requires only 40 minutes whereas conventional fine-tuning on the joint ImageNet-1K + Places dataset requires 12.1 hours on the same hardware.

We note that a critical feature of Figure 3 is that SGM's curve has a downward slope, except at the beginning, which is much steeper than vanilla rehearsal, indicating that SGM is becoming a more efficient learner over time.

Reviewer comment (Presentation):
In Figure 3, it would have been better to also show the performance graphs in comparison to the rehearsal-based methods of Table 3 (DERpp and GDumb).

Author response:
We shall do this in a revised version of the paper.

Reviewer comment (Experiments):
In section 3.1 (second paragraph), the rationale behind setting f=0.3 is not clearly elucidated. A more detailed explanation in the experimental section would be beneficial.

Author response:
Lower $f$ yields higher computational efficiency. In a real-world setting, a continual learner has to adapt to a large-scale data stream (ideally never-ending), thereby requiring more computation than size of the data stream may not be feasible. For example, for many applications such as on-device learning, embedded devices, and AR/VR, a continual learner has to learn new information quickly without increasing computational overhead. Our principal focus was to align continual learning with resource-constrained applications. That’s why we chose $f$=0.3. We have revised the paper to better explain this choice.

Reviewer comment (Experiments):
While the main experiments utilize ConvNeXtV2-Femto, which has undergone unsupervised pertaining on ImageNet1k followed by supervised fine-tuning, it would be advantageous to also present results for training ResNet18 from scratch. This is important given the common use of ResNet18 in the CL community.

Author response:
While ResNet-18 has been historically used, we see no reason to not use more recent architectures that would actually be employed in real-world applications. Moreover, ConvNeXtV2-Femto performs better than ResNet-18 (78.25% vs 69.76% on ImageNet-1K) using 2x fewer parameters (5.2M vs. 11.6M parameters) [1]. Many recent works are using vision transformers and more advanced architectures (e.g., [7]), and we believe that continual learning must use the latest advances for the community to care about it rather than focusing on the ResNet architecture which is now almost a decade old. Note that LoRA is not appropriate for ResNet architectures, but it can be used with ViT models, ConvNeXt, ConvNeXtV2, etc.

Here are some additional reasons:
1. For the major real-world applications of continual learning, training from scratch is not employed. Hence, we focused our experiments on a setting that matches real-world use-cases, where companies and many academic projects start with ImageNet-1K pre-trained models. 
2. Many recent methods in continual learning that have had significant impact use pre-trained models, e.g., DualPrompt uses a ViT pre-trained on ImageNet-1K. Indeed with the shift to ViT models, many continual learning papers are adopting this paradigm. We note that this is often hidden in their supplemental material rather than presented in the main text. We aim to be upfront about what we are doing. 
3. Many continual learning methods evaluated only on ImageNet-1K have their first offline batch as the first 100-500 classes of ImageNet-1K [2-6], and we think that would suffice. For example, in ImageNet-1K experiments, BiC [5] and PODNet [6] used 500 ImageNet classes for initializing the network before the CL phase began on the remaining 500 ImageNet classes. During the initialization phase, they trained a ResNet-18 on 500 ImageNet Classes for 90 epochs.

Reviewer comment (Experiments):
Table 1 could benefit from including results from normal fine-tuning as a lower-bound reference. Additionally, explicitly reporting accuracy on ImageNet-1k is recommended due to the class imbalance between ImageNet-1k and Places365-LT. In light of this, could the authors discuss whether stability takes precedence over plasticity due to this imbalance? Furthermore, given the context of continual learning with a pre-trained model on a large dataset like ImageNet-1k, it would be beneficial to discuss why other relevant methods [1-3] were not investigated.

Author response:
Done.

# Does Stability Take Precedence Over Plasticity?
Stability and plasticity are both measured in all experiments and reducing both stability and plasticity gaps indicates that stability and plasticity is well balanced. SGM achieves this goal in all experiments where it increases both stability and plasticity (Fig. 5). 

# Compare Against Other Methods that Use ImageNet-1K Pretrained Models
We agree that it would be interesting to study some of these methods such as DualPrompt. We intend to do this, but it is not clear if we would have sufficient time during the rebuttal period to get it running.

Reviewer comment (Experiments):
While the exploration of the stability gap is intriguing, my primary concern lies in the generalizability of the conclusions drawn from the experimental results, given the use of imbalanced datasets and an unsupervised pre-trained model with a large dataset. Even with the inclusion of experiments using a supervised pre-trained model in section 6.3, the findings may still lack generalizability, as the model might inherently learn more general features with a sufficiently large dataset. For example, this type of pre-trained model may possess good plasticity (transferability) without extensive weight updates. I would recommend the authors conduct experiments with a random initialized model and balanced tasks for a more comprehensive evaluation.

Author response:
We included training from scratch experiments in supplemental section F. As shown in Table 10, our method improved performance over vanilla. Besides imbalance dataset (Places365-LT), we have included experiments with balanced datasets (Places365-Standard) as well. The results with a blanched dataset are presented in Table 3, 9, 10, and 11.

# Do the Results Generalize?
While our training from scratch experiments demonstrate SGM is effective in that setting, it showed less benefit. We hypothesize this is probably due to CIFAR being inadequate for learning good representations before plasticity is reduced. We are planning to conduct additional experiments with recent continual learning methods using pretrained ViT models, although we are not sure if we will have sufficient time to do this during the rebuttal period.

Reviewer comment (Experiments):
The extremely relaxed setting for continual learning limits the contribution of the proposed method. For this setting, the continual learner has a strong pre-trained model and can access all prior source data for data replay (e.g., ImageNet-1K) which is unusual in continual learning.

Author response:
# Concerns Regarding Starting with a Pretrained Model
Given that some of the methods the reviewer mentions start with large pre-trained ViT models  (Learning to Prompt and DualPrompt), this concern is especially surprising. Here are three reasons why we do not think this is a valid concern:
1. For the major real-world applications of continual learning, training from scratch is not employed. Hence, we focused our experiments on a setting that matches real-world use-cases, where companies and many academic projects start with ImageNet-1K pre-trained models. 
2. Many recent methods in continual learning that have had significant impact use pre-trained models, e.g., DualPrompt uses a ViT pre-trained on ImageNet-1K. Indeed with the shift to ViT models, many continual learning papers are adopting this paradigm. We note that this is often hidden in their supplemental material rather than presented in the main text. We aim to be upfront about what we are doing. 
3. Many continual learning methods evaluated only on ImageNet-1K have their first offline batch as the first 100-500 classes of ImageNet-1K [2-7], and we think that would suffice. For example, in ImageNet-1K experiments, BiC [6] and PODNet [7] used 500 ImageNet classes for initializing the network before the CL phase began on the remaining 500 ImageNet classes. During the initialization phase, they trained a ResNet-18 on 500 ImageNet Classes for 90 epochs. 

We see little scientific value in using an ImageNet-1K pre-trained model on CIFAR-100, and Places is a much harder and more realistic dataset. Could the reviewer please justify why split CIFAR-100 would be better as done in DualPrompt over class incremental learning with Places?

Reviewer comment (Novelty):
Such a setting narrows the proposed hypothesis for continual learning with the pre-trained model. Meanwhile, the proposed techniques are specially designed for that. With the pre-trained model, some of the proposed techniques like weight initialization and LoRA are natural selections that are hard to be considered as contributions.

Author response:
# Why Use ImageNet-1K followed by Places-365?
As described in the introduction, our goal is to study CL in large scale problems which are more realistic and align with real-world applications. While others have not done Places-365 after learning ImageNet-1K, to be honest, we think it makes no sense from either a scientific or practical purpose to take an ImageNet-1K pre-trained model and then learn something like split CIFAR-100. It is like taking an incredibly powerful network to then learn a toy dataset. We do not see the scientific value. Could the reviewer please elaborate on why they think this would be better? Note that Places-365 is widely used as hard out-of-distribution dataset in papers regarding out-of-distribution detection with ImageNet-1K datasets due to the domain shift between Places-365 and ImageNet-1K [1], whereas the classes in CIFAR significantly overlap with ImageNet-1K (see Table F.1 in [2]). Our problem setting is harder than the ones suggested by the reviewer.

Reviewer comment (Evaluation):
Such extremely relaxed settings still cannot reach the accuracy of offline settings as shown in Table 1.

Author response:
# Concerns About The State-of-the-Art
We are aiming to do science rather than chase benchmarks. SGM mitigates the stability issues and aligns with our hypotheses. In Table 1, SGM almost achieves the same accuracy (70.30%) as offline upper bound (70.69%), there exists only 0.39% difference. In Table 10, SGM improves performance over compared methods by absolute 3.43% (50K samples in buffer) and 3.45% (5K samples in buffer) in final accuracy.

Reviewer comment (Experiments):
Lack of comparisons with most related works on continual learning with pre-trained models. In the paper, the authors mainly compare the proposed method to DERpp and GDumb which are designed for training from scratch. Thus, comparisons with the most recent works on continual learning with pre-trained models are needed.

Author response:
We did a fair comparison when we combined SGM with other existing CL algorithms e.g., DERpp, GDumb, REMIND, LwF which are widely used CL baselines. All compared methods used the same pre-trained model. We wanted to cover a variety of CL algorithms: 1) rehearsal based: GDumb, 2) rehearsal and knowledge distillation: DERpp, 3) non-rehearsal and knowledge distillation: LwF, 4) online CL: REMIND. Of course there are many more different CL methods but experimenting with all past CL methods is intractable. We want to demonstrate that our method mitigates the stability gap and increases computational efficiency in most common CL methods.

Reviewer comment (Experiments):
Lack of ablation study to show the performance of partially combined techniques in SGM. In Table 1, the authors list the ablation study of each component of SGM. Since the proposed SGM includes four different techniques, it's unclear what is the performance of partially combined techniques in SGM.

Author response:
# Try Every Possible Combination of the Mitigation Methods
There are 16 possible configurations, and we did 6 of these. It is worth noting that it requires significant computation and time to assess the performance of a model while being trained on a large scale ImageNet dataset combined with another dataset. Is there a specific hypothesis the reviewer has that would justify this?

Reviewer comment (Experiments):
The proposed method is evaluated in constrained settings. The network is ConvNeXtV1-Tiny, which is seldomly used in previous continual learning literature. The applicability of the proposed method to other network architectures, such as ResNet or simple CNNs, remains unclear. Additionally, the buffer size is much larger than the one used in previous literature.

Author response:
# Concerns Regarding ConvNext
While ResNet-18 has been historically used, we see no reason to not use more recent architectures that would actually be employed in real-world applications. Moreover, ConvNeXtV2-Femto performs better than ResNet-18 (78.25% vs 69.76% on ImageNet-1K) using 2x fewer parameters (5.2M vs. 11.6M parameters) [1]. Many recent works are using vision transformers and more advanced architectures (e.g., [8]), and we believe that continual learning must use the latest advances for the community to care about it rather than focusing on the ResNet architecture which is now almost a decade old. Note that LoRA is not appropriate for ResNet architectures, but it can be used with ViT models, ConvNeXt, ConvNeXtV2, etc.


# Concerns Regarding the Amount of Data Stored
The claim that we are using a larger buffer size compared to many past works for large-scale continual learning is false (see reason 2). We believe we adequately addressed this in the paper for three reasons:
1. We demonstrate that SGM is effective with LwF which is rehearsal free and does not use additional memory.
2. We conducted extensive experiments in Table 3, where we bounded the buffer size by 24K samples, which is only 0.8% of the total number of images in the two datasets (3M from the 1.2M in ImageNet-1K and 1.8M in Places). Prior work on class incremental learning with ImageNet has used 20K images (1.7% of the 1.2M in ImageNet-1K) [2-7]. So as a fraction of the total dataset, our constrained experiments are _more_ restrictive than most past works.
3. The unconstrained experiments we conducted in Table 1 enable us to focus on our scientific question, and for industrial applications the major concern is compute whereas storage is very cheap.

Reviewer comment (Experiments):
The effectiveness in reducing the stability gap depends heavily on the utilization of a pretraining model from ImageNet-1K. As shown in Tab. 10, the improvement is very limited when training from scratch, due to the fact that LoRA and freezing of old class units cannot be used in this case.

Author response:
# Concerns Regarding Starting with a Pretrained Model
We disagree that this is a concern for three reasons:
1. For the major real-world applications of continual learning, training from scratch is not employed. Hence, we focused our experiments on a setting that matches real-world use-cases, where companies and many academic projects start with ImageNet-1K pre-trained models. 
2. Many recent methods in continual learning that have had significant impact use pre-trained models, e.g., DualPrompt uses a ViT pre-trained on ImageNet-1K. Indeed with the shift to ViT models, many continual learning papers are adopting this paradigm. We note that this is often hidden in their supplemental material rather than presented in the main text. We aim to be upfront about what we are doing. 
3. Many continual learning methods evaluated only on ImageNet-1K have their first offline batch as the first 100-500 classes of ImageNet-1K [2-7], and we think that would suffice. For example, in ImageNet-1K experiments, BiC [6] and PODNet [7] used 500 ImageNet classes for initializing the network before the CL phase began on the remaining 500 ImageNet classes. During the initialization phase, they trained a ResNet-18 on 500 ImageNet Classes for 90 epochs. 

# Concerns Regarding Our Training From Scratch Experiments
In Table 10, our training from scratch experiments with CIFAR show that SGM improves performance over compared methods by absolute 3.43% (50K samples in buffer) and 3.45% (5K samples in buffer) in final accuracy. Regardless, we think this benefit would be much more dramatic if the network was trained with a larger dataset, as there isn’t much to learn in terms of strong representations for such a toy problem.

Reviewer comment (Evaluation):
It is interesting that though the main goal of the proposed method is not to improve performance, it performs slightly better than the vanilla method. Can you explain this?

Author response:
# Why Does SGM Perform Better than Vanilla?
We hypothesize that the mechanisms in SGM facilitate better forward and backward transfer, resulting in better performance than the vanilla method. That said, our primary objective is to study the stability gap in order to reduce the amount of compute needed to update models trained with rehearsal, which is the method that holds the most promise for industrial applications, where compute is limited, storage is cheap, and predictive performance cannot be sacrificed. The goal of SGM is to reduce stability gap, plasticity gap and continual knowledge gap and enhance performance while using fewer network updates as shown in Fig.1-5.

Reviewer comment (Presentation):
What does the ‘Best Mitigation Method’ shown in Figure 1 refer to? Is it the proposed RSGM?

Author response:
# What is the "Best Mitigation Method" in Figure 1?
The best mitigation method is our proposed method, SGM. We have revised the caption and figure to be more specific.