Paper: Bridging the gap between offline and online continual learning (ICLR-2024)

Reviewer comment (Theory):
While some theoretical analysis is provided, I don't think that this is sufficient to prove the claims of the paper. In particular, the analysis of Eq (6) seem to show that "a minimum amount of short term memory" should be used. In that case, why are you using short term memory at all if the analysis concludes that you should not use it? (It's also a question, please inform me if I missed something here). In the case that I am correct, I think that these analysis are not necessary to the paper.

Author response:
**W1. Why we argue to use the minimum short-term memory instead of zero short memory**
>the analysis of Eq (6) seem to show that "a minimum amount of short term memory" should be used. In that case, why are you using short term memory at all if the analysis concludes that you should not use it?

Great question. The reason is that $M_{short}=0$ leads to a significantly different procedure with non-zero short memory. More specifically,  for any non-zero short-term memory, no matter how small it is, there is a consistent training procedure that spans online and offline CL: each incoming batch will be trained by the model for $E$ iterations before it is discarded. Due to the continual learning nature, this procedure holds regardless of the exact value of $M_{short}$. In contrast, if there is no short-term memory, the incoming data is directly saved to long-term memory based on some selection mechanism _without being trained_. This means some incoming data are never seen by the model. We believe this represents a fundamentally distinct training approach from the current online and offline CL practices.

Our theoretical analysis only concerns the generalization bound and does not cover training procedures. Since the performance is affected by both factors, we apply our theoretical results only to provide a possible explanation for algorithm performance with similar training procedures.  Hence, we interpret our theoretical results only in the context of long-short term frameworks, where short-term memory lies in $[DataStreamBatchSize, TaskSize]$.

Reviewer comment (Experiments):
It would be nice to have results for Split-Cifar100 and Mini-Imagenet with more splits (20), since then the total allowed memory would also be smaller.

Author response:
**W3. results with more splits and different total memory budget**

The results of Split-Cifar100 with 50, 20 and 10 are shown in Fig 3 c. We use task size as the labels for x axis, i.e. 1.0 k, 2.5k, 5k. These experiments uses 7k total memory.

Besides using 7k memory budget, Fig 3b experiments with other total memory budgets (5.5k, 6k, 7k, 10k )

Reviewer comment (Reproducibility):
Do you plan to release the code?

Author response:
**Q1**. yes, we will release the code

Reviewer comment (Theory):
c.f weaknesses (2), did I understand something wrong in the analysis Eq (6)? If the analysis concludes that you should use minimum short term memory why do you use any at all? If it's just that it's working better experimentally but that the analysis concludes something different then I think it's better to remove this analysis altogether.

Author response:
**W1. Why we argue to use the minimum short-term memory instead of zero short memory**
>the analysis of Eq (6) seem to show that "a minimum amount of short term memory" should be used. In that case, why are you using short term memory at all if the analysis concludes that you should not use it?

Great question. The reason is that $M_{short}=0$ leads to a significantly different procedure with non-zero short memory. More specifically,  for any non-zero short-term memory, no matter how small it is, there is a consistent training procedure that spans online and offline CL: each incoming batch will be trained by the model for $E$ iterations before it is discarded. Due to the continual learning nature, this procedure holds regardless of the exact value of $M_{short}$. In contrast, if there is no short-term memory, the incoming data is directly saved to long-term memory based on some selection mechanism _without being trained_. This means some incoming data are never seen by the model. We believe this represents a fundamentally distinct training approach from the current online and offline CL practices.

Our theoretical analysis only concerns the generalization bound and does not cover training procedures. Since the performance is affected by both factors, we apply our theoretical results only to provide a possible explanation for algorithm performance with similar training procedures.  Hence, we interpret our theoretical results only in the context of long-short term frameworks, where short-term memory lies in $[DataStreamBatchSize, TaskSize]$. 

**Q2**. Hope the response above has addressed this question. Please inform us if you have any further inquiries.

Reviewer comment (Presentation):
It seems that Figure 1 and Figure 1 are the same figure. You need to fix this problem.

Author response:
Figure 1 and Figure 5 are not the same figure. Figure 1 is the results on Split-CIFAR100 and Figure 5 are the results on Split-Mini-ImageNet

Reviewer comment (Experiments):
The experiment for supporting the empirical observation is not very valid. You should consider more complex datasets (e.g., ImageNet) or class incremental learning in NLP tasks (e.g., intent classification). Only conducting experiments on the CIFAR dataset is not enough.

Author response:
We would like to clarify that apart from Split-CIFAR100, we also conducted experiments in other widely used continual learning benchmarks, split-mini-imagenet and CORE50 (see Table 1).  The appendix also includes the results of ImageNet in Table 4.  Our finding is consistent in all these experiments with different image types and dataset size.

Reviewer comment (Evaluation):
I don't know what benefit can we learn from this observation? Can we use it to reduce the forgetting problem?

Author response:
This work poses several questions into the current practice of the continual learning field:
| Current pratice                                                                                                                                                                              | Our contribution                                                                                                                                                                 | Implications                                                                                                                                                              |
|------------|--------------|-------------|
| Focus on the storage cost of exemplar samples                                                                                                                                               | Exemplar and new samples both exist within the memory-constrained operational space; Take into account both in our experiment | Online CL is probably undervalued in the community and it may provide better solutions than offline CL  in a practical application with limited memory.|
| Use single iteration in online CL (I=1)  and large epochs in offline CL (E=70-200); Complex offline CL algorithms are believed to be not suitable for online CL due to the single epoch constraint |  Experiment with the iteration number as large as the epoch number; Demonstrate complex algorithm design (e.g DER++) can be trained well in single epoch setting (online) by simply increasing iteration number |  Open the door of more algorithms candidates (the second line in the table below) for solving online CL problems                                                                                             |



|            | Algorithms                                                                            |   
|------------|---------------------------------------------------------------------------------------|
| online CL  | ER, ER-ACE, MIR, ASER, GSS, SCR, RAR, COPE, OCM,CBA                                             |  
| offline CL |  iCaRL, BIC, IL2M, LUCIR, EEIL, RWalk, DER, MEMO, FOSTER, PODNET, FOSTER, WA, RMM, COIL, CO2L, DER++|                                                                                        

We revised the introduction and conclusion of the paper to clarify the implications of our work.

Reviewer comment (Experiments):
Can it principally reduce the forgetting of vision or NLP models in continual learning?

Author response:
Yes. The stability-plasticity analysis in Figure 1(b) highlights the effect of the storage allocation policy on forgetting: using a small storage allocation parameter leads to better stability (less forgetting). One of our primary messages is that the storage allocation policy is a key factor concerning CL performance but is overlooked in the current practice of the field. Our experiment results on four CL algorithms suggest that tuning the storage allocation parameter leads to a bigger performance boost than specific algorithm design (Table 1).

Reviewer comment (Novelty):
I do not see the benefit of this observation.

Author response:
This work poses several questions into the current practice of the continual learning field:
| Current pratice                                                                                                                                                                              | Our contribution                                                                                                                                                                 | Implications                                                                                                                                                              |
|------------|--------------|-------------|
| Focus on the storage cost of exemplar samples                                                                                                                                               | Exemplar and new samples both exist within the memory-constrained operational space; Take into account both in our experiment | Online CL is probably undervalued in the community and it may provide better solutions than offline CL  in a practical application with limited memory.|
| Use single iteration in online CL (I=1)  and large epochs in offline CL (E=70-200); Complex offline CL algorithms are believed to be not suitable for online CL due to the single epoch constraint |  Experiment with the iteration number as large as the epoch number; Demonstrate complex algorithm design (e.g DER++) can be trained well in single epoch setting (online) by simply increasing iteration number |  Open the door of more algorithms candidates (the second line in the table below) for solving online CL problems                                                                                             |



|            | Algorithms                                                                            |   
|------------|---------------------------------------------------------------------------------------|
| online CL  | ER, ER-ACE, MIR, ASER, GSS, SCR, RAR, COPE, OCM,CBA                                             |  
| offline CL |  iCaRL, BIC, IL2M, LUCIR, EEIL, RWalk, DER, MEMO, FOSTER, PODNET, FOSTER, WA, RMM, COIL, CO2L, DER++|                                                                                        

We revised the introduction and conclusion of the paper to clarify the implications of our work.

Reviewer comment (Experiments):
**W1.** *$C_{task}$ and $M_{online}$ have different distributional constraints* [**Critical**]  
- The fundamental discrepancy arises from the proposed memory correction. Offline continual learning employing a 2000-sized memory buffer ($M_{offline}$) is notably disadvantaged relative to online continual learning with a 7000-sized memory ($M_{online}$).  
- This stems from the fact that while the data distribution within a task ($C_{task}$) remains static, the memory ($M_{online}$) has the potential to represent samples from all past data distributions.  
- Concretely, the primary factor favoring online CL over offline CL in Section 3.3 comes from this imbalance in representation: 5000 samples from the latest distribution $C_t$ (in $M_{offline}$) are markedly less useful than 5000 samples across $D_1, ..., D_t$ (in $M_{online}$), hence it performs poorer.  
- Reviewer states they would be very surprised if online continual learning ever outperforms the offline continual learning setting.  
- The key question is: *What if, with the only difference being short-term memory?* What is the fair correction to memory then? The reviewer agrees with the paper on equalizing computation but suggests equalizing the long-term memory alongside computation. Interpolation should be between short-term memory from size $C_{batch}$ to $C_{task}$.  
- In Section 3.3, the offline model can use a far larger short-term memory (5000) compared to an online model (64). Both have the same fixed sized long-term memory but have equal computation.  
- Informal claim: Any training algorithm (with compute budget C) where 64 samples are sequentially introduced can be mimicked when you have access to those 64 and subsequent samples (totaling 5000 samples). The reverse is not feasible due to the inaccessibility of future samples.  
- Conclusion: Offline continual learning should be strictly better than online continual learning in principle, because future samples help improve performance. This conclusion is diametrically opposed to the aim of the work. The reviewer hopes the above informal statements convey the core intuition behind why offline continual learning is going to be better than online alternatives.

Author response:
We greatly appreciate the reviewer providing their main concern W1 in a partial review to engage in further discussion. For expediency, we focus on replying to this concern here and will upload our replies to other concerns and the revised paper later. 

## The effect of aligning computation only

In our submission, we only presented the results of aligning both memory consumption and the number of iterations. Prompted by W1, we have now included an ablation study with results obtained by aligning the number of iterations or memory consumption only, summarized in the table below. Compared with no alignment of the number of iterations, regardless of whether memory consumption is aligned or not, the performance of online CL is significantly increased by moving from 1 to 50 iterations. More interestingly, when aligning the number of iterations only, the performance of online CL  becomes very close to the performance of offline CL, despite the fact it uses a much smaller memory budget: online-align-compute: 2064 samples (2k+64) and offline: 7000 samples (2k+5k)! 



|                           | #Iterations   |Memory    |CIFAR100    | MINI-IMAGENET |  
|---------------------------|-------------|---------------|---------------|---------------|
| offline   | 50 | 7K|40.1 $\pm$ 1.6 |    31.9 $\pm$ 2.8           |        |
| online-no-align | 1 | 2.064K  | 11.6 $\pm$ 1.1 | 11.4 $\pm$ 1.8              |        |
| online-align-compute |50|2.064K | 38.6 $\pm$ 1.4 |    31.0 $\pm$ 2.9          |        |
| online-align-memory   |   1 | 7k | 13.2   $\pm$ 1.2        |   12.3 $\pm$ 3.2            |        |
| online-align-both  |50|7K        | 52.7 $\pm$  0.8 | 43.9 $\pm$3.4  

Thanks for pointing out our error in calculating the memory cost of online CL. We updated it to exemplar memory + batch size.

Reviewer comment (Theory):
**W2** *Bound not tight, and likely vacuous* [**Critical**]  
- Theorem 1, in its current form, merely establishes an inequality without showing that the bound is tight. However, the subsequent arguments rely heavily on the bound being tight.  
- For the proof of Theorem 1 to be complete, it must additionally be shown that the same hypothesis $h^*$ achieves equality in both Eq. 9 and Eq. 8 in Appendix A.1. It is unclear whether the best hypothesis on the distribution of memory samples will necessarily be the best for the overall data distribution.  
- The reviewer has not looked at Mansour et al. (2009) but notes this uncertainty.  
- On whether tightness, if shown, would be convincing: It would make the proof of the theorem complete. However, for deep learning systems, generalization bounds more sophisticated than Theorem 1 are often shown to be vacuous.  
- Because of this, the reviewer believes the theoretical motivations for the empirical results are quite poor and that Section 5 adds little value to the work.  
- Nevertheless, the reviewer is happy to consider predictions from Corollary 2.2 on their own terms and empirical results in Figure 3 from it.

Author response:
**3.generalization bound**

We agree that the generalization bound may not be tight. However, it should be note that this is not because of the missing proof of this paper, but because theorem 8 in Mansour et al. (2009)  (Eq 8 in our paper) is not tight. Based on our knowledge, how to derive tight bounds for transfer learning and how to measure the quality of the bound is still a challenge due to the distribution difference in the source and target domain. Achieving equality in theorem 8 already requires h*(Q1) =h*(Q2) , which may not be achieved in practice. However, this upper bound is meaningful when h*(Q1) and h*(Q2) are close, and this is usually the assumption of transfer learning. In our work, Q1 and Q2 are the true distribution of the data stream and the _expected distribution_ of the memory data. In the rehearsal-based methods, the memory data includes both new and past samples, we assume h*(Q1) and h*(Q2) are also reasonably close. As verified in Fig 3, this generalization bound seems to provide meaningful insights into the influence of data stream size, task size and memory budget size and the prediction that online outperforms semi-online cases in Fig 2b.

Reviewer comment (Experiments):
Although combining online and offline CL is interesting, using the same memory and compute for online CL as used by offline CL makes both computationally expensive. Online CL uses many more iterations at each timestep to match offline CL’s compute overhead. Thus online CL becomes ill-suited for many real-world applications e.g., on-device learning where fast adaptation / speed is critical. Online CL’s performance gains are mostly attributable to higher compute (more SGD updates) and memory usage.

Author response:
To clarify, even with much smaller compute overhead, online CL with 10 iterations (48.4% $\pm$ 0.5) already leads to better performance than offline CL with 100 epochs (40.1% $\pm$ 0.8) as shown in Fig 1a. We analyze the influence of iteration number on stability and plasticity. With iteration <10, increasing iteration leads to better stability and plasticity at the same time and the performance boost is very significant. With iteration >10, increasing interaction starts to hurt stability (more forgetting) and the benefit on the overall CL accuracy is limited. One implication of our finding is, instead of always using a single iteration, one can adjust the iteration number based on the speed-accuracy requirements of the specific application. 
In addition, it should be noted that even with multiple iterations online CL still supports fast adaptation in the sense it updates the model as long as it receives an incoming batch. In contrast, offline CL waited until the collection of a task dataset to update the model. In an interactive application, online CL can make the system more responsive to user feedback.

Reviewer comment (Experiments):
This paper defines online CL based on batch learning i.e., learner receives a batch of new data and makes multiple iterations where each iteration combines a new batch with a sampled batch of old data from the buffer. This becomes very similar to offline CL with larger batch size. A more realistic online CL can be defined such that an online learner learns new data sample-by-sample (instead of batch-by-batch) using a single training iteration that combines a new sample with a batch of old data. This is much faster than the former definition and suitable for real world applications where speed matters. Another difference is the number of times each new sample is seen by the learner. Unlike the latter definition, the former one allows the model to see each new sample multiple times which is not truly a single pass (core ingredient of online CL).

Author response:
Our problem setting is that the incoming batch in the data streams comes with a batch size of B. This setting is commonly used in continual learning research works [1][2]. Under this problem setting, we compare two ways of continual model update:

1) update the model as long as each incoming batch data comes: model update frequency = batch size B
2) wait and collect a large batch of data before updating the model: model update frequency = task size T

We find that 1) leads to better performance given aligned memory and iterations. The table below shows a summary of CL algorithms following these two ways. Thus, we believe achieving a deeper understanding of the efficiency of these two is important and is the focus of this work. We agree that the sample-by-sample data stream is an interesting case, but is out of the scope of this work.

|            | Algorithms                                                                            |   
|------------|-----------|
| 1) online CL  | ER, ER-ACE, MIR, ASER, GSS, SCR, RAR, COPE, OCM,CBA                                             |  
| 2) offline CL |  iCaRL, BIC, IL2M, LUCIR, EEIL, RWalk, DER, MEMO, FOSTER, PODNET, FOSTER, WA, RMM, COIL, CO2L, DER++|  


[1] Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification:An empirical survey. Neurocomputing, 469:28–51,2022.

[2] Soutif-Cormerais, Albin, et al. "A Comprehensive Empirical Evaluation on Online Continual Learning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Reviewer comment (Evaluation):
I am not convinced that single pass causes underfitting and online CL requires multiple iterations at each timestep. Underfitting originates from specific design choices for example training from scratch where online CL model is randomly initialized and struggles due to lack of an optimum initialization (pre-training on a subset of data). There are methods that do not exhibit underfitting. An example of such a method is REMIND [1] that performs a base initialization using a subset (10% of ImageNet). REMIND performs online continual learning in sample-by-sample manner at each timestep with one SGD step / iteration on a mini-batch consisting of one new data and several old data. REMIND achieves competitive accuracy to the offline variant.

Author response:
[3] discusses the use of multiple iterations to deal with the underfitting-overfitting dilemma of online CL (when trained from scratch). Whether or not single pass causes underfitting is not the topic of this work. We simply show that with the same memory, compute and  CL algorithm, updating the model in an online manner leads to better performance than offline.

[3] Zhang, Yaqian, et al. "A simple but strong baseline for online continual learning: Repeated augmented rehearsal." Advances in Neural Information Processing Systems 35 (2022): 14771-14783.

Reviewer comment (Novelty):
Although this paper presents valuable insights and observations, the proposed framework focuses mainly on how to balance memory and compute between online and offline CL. It does not propose any specific mechanisms to mitigate catastrophic forgetting, enhancing stability and plasticity and underfitting issues (when present due to design choices). It lacks sufficient scientific contributions.

Author response:
We did not propose new algorithms. Instead, we study a fundamental problem concerning different CL algorithms, the usefulness of past samples vs. new samples. We highlight an overlooked but important factor concerning CL performance: the storage allocation policy$\alpha=M_{short}/M$. Online and offline CL are just variants of the same learning algorithm that only differ in the setting of the storage allocation parameter (Online CL: $\alpha=B/M$ and offline CL $\alpha=TaskSize/M$).
Tuning the storage allocation parameter $\alpha$ leads to a bigger performance boost (better stability-plasticty tradeoff) than the specific algorithm design (Table 1). The effect of this parameter on forgetting (i.e.-stability) is presented in Fig 1b and Fig 5b.