Paper: Chunking: Forgetting Matters in Continual Learning even without Changing Tasks (ICLR-2024)

Reviewer comment (Experiments):
The central motivation surrounding the exploration of limited data availability for continual learning (CL) might benefit from more clarification. Although the paper aims to explore the impact of limited data on CL, the assumptions introduced for 'chunking' in the continual learning framework seem to diverge significantly from practical real-world scenarios.

Author response:
We agree with you that the chunking setting is not a suitable model for many real-world scenarios and state as much already in the paper ([Page-2, Section-2, Para-3]). However, this misses the point of the paper which is that any more realistic CL setting with task/distribution shift can be decomposed into two sub-problems, one of which is the chunking problem. We show that the chunking problem is a significant part of the most common CL settings and has been underexplored as no current CL methods performs better than SGD on it. Therefore, a way to improve on any real-world CL problem would be to improve performance on the chunking problem, which is the key point of our work and is currently underexplored. We hoped that this point was clear in the paper but given your comment we see that we need to be clearer on this point and will edit the paper accordingly. 

Also, in your review you mention the use of class-balanced chunks, this is to ensure that our experimental findings are due to the chunking problem and not class imbalance. However, we agree with you that we could describe better the random-sampling experimental setting looked at in appendix B and have edited the paper accordingly.

Reviewer comment (Experiments):
Assuming data in each chunk are identically distributed negates the occurrence of distribution shifts. In scenarios lacking such shifts and with sufficiently large chunk sizes, performance degradation may be minimal. This could make comparing different methodologies highly sensitive to the chosen chunk size, complicating reproducibility for subsequent research.

Author response:
In task-free CL, as you mention, there is distribution shift but unlike other settings the distribution changes slowly through time. This is unlike the chunking setting where we look only at the problems arising from seeing the data in chunks and so by design contains no distribution shift. Therefore, the settings are quite different. However, like other CL settings, as we see the data in chunks, chunking is a subproblem of task-free CL and so improving performance in the chunking setting is necessary for task-free CL performance to approach that of offline performance.

Reviewer comment (Experiments):
The constraint that each chunk must contain an approximately equal number of instances for each class is often unrealistic. In real-world scenarios, new categories often emerge over time, and it’s impractical to anticipate equal representation of all classes. Although class imbalance is evaluated in Appendix B, the experimental settings for the imbalance settings are not well articulated.

Author response:
Also, in your review you mention the use of class-balanced chunks, this is to ensure that our experimental findings are due to the chunking problem and not class imbalance. However, we agree with you that we could describe better the random-sampling experimental setting looked at in appendix B and have edited the paper accordingly.

Reviewer comment (Novelty):
The paper compares its approach with online continual learning but fails to adequately differentiate it from task-free continual learning approaches [1-4], which do not assume explicit task boundaries. In many situations, identifying shifts in tasks is more critical than defining boundaries for model updating.

Author response:
In task-free CL, as you mention, there is distribution shift but unlike other settings the distribution changes slowly through time. This is unlike the chunking setting where we look only at the problems arising from seeing the data in chunks and so by design contains no distribution shift. Therefore, the settings are quite different. However, like other CL settings, as we see the data in chunks, chunking is a subproblem of task-free CL and so improving performance in the chunking setting is necessary for task-free CL performance to approach that of offline performance.

Reviewer comment (Novelty):
The method of averaging weights upon the completion of each chunk or task closely resembles existing strategies used in both offline training and online federated learning, raising questions about its innovative aspect. Additionally, this technique presupposes the availability of parameters from all chunks, a condition that might not always be feasible in practical scenarios.

Author response:
In your review you point out that weight averaging has been used in offline learning and federated learning before and hence question its general novelty. We agree with this point and already state so in the paper ([Page-3, Section-2, Para-4]). However, we hope that you agree with us that it has not been looked at as a method to improve performance in the chunking setting, which is the reason we look at it. That is, we look a per-chunk weight averaging to show that you can improve performance and reduce forgetting in the chunking setting and that this performance improvement transfers to full CL settings where there is also task/distribution shift. Therefore, we hope you now can see that in the context of chunking by looking at per-chunk weight averaging we have provided a useful starting point for improving performance in the chunking setting.

Reviewer comment (Experiments):
Referencing DER++ as the state-of-the-art method does not align with recent advancements. Comparisons with newer models like GMED (Jin et al., 2021) and CLS-ER (Arani et al., 2022) would be more relevant, considering their similar experimental contexts.

Author response:
We do look at a data free method in our experiments on the chunking setting, EWC and have run experiments with SI where it had the same behaviour as EWC in the chunking setting. Additionally, we thank you for pointing out the two CL methods GMED and CLS-ER, we agree with you that adding CLS-ER to our chunking experiments would be beneficial and will do so but this might take some time. However, in the case of GMED it is only designed for the online CL so can not be used for any chunk size greater than that of a mini-batch and so can not be readily added to our chunking experiments. Also, we would like to say that both papers focus on the challenge of dealing with task/distribution shift and do not mention or give thought to the other sub-problem of CL, the chunking problem. This is the main point we are trying to put across in our paper, that the chunking problem has been overlooked as a sub-problem in CL and hence these works do not challenge the main point of this paper, instead they strengthen it.

Reviewer comment (Novelty):
The paper mentions the impossibility of resampling data or chunks but lacks a discussion and incorporation of data-free continual learning methods [8-10] in the existing literature.

Author response:
We do look at a data free method in our experiments on the chunking setting, EWC and have run experiments with SI where it had the same behaviour as EWC in the chunking setting. Additionally, we thank you for pointing out the two CL methods GMED and CLS-ER, we agree with you that adding CLS-ER to our chunking experiments would be beneficial and will do so but this might take some time. However, in the case of GMED it is only designed for the online CL so can not be used for any chunk size greater than that of a mini-batch and so can not be readily added to our chunking experiments. Also, we would like to say that both papers focus on the challenge of dealing with task/distribution shift and do not mention or give thought to the other sub-problem of CL, the chunking problem. This is the main point we are trying to put across in our paper, that the chunking problem has been overlooked as a sub-problem in CL and hence these works do not challenge the main point of this paper, instead they strengthen it.

Reviewer comment (Novelty):
The chunking setting is questionable. In Figure 1, there does not appear to be a difference between the so-called chunking setting and domain-incremental learning, where the change of task is only indicated by the change of data. In the language of chunking, each chunk is a task in domain-incremental learning.

Author response:
From your review it seems your main criticism is that you think the chunking setting is the same as domain incremental learning (DIL). However, the chunking setting and DIL are quite different settings. This is because by definition DIL has a task/distribution shift element, as p(x) is different for each task; while the point of the chunking setting is to only look at the chunking sub-problem and not the task-shift element of CL and so by construction it has no task/distribution shift element. Also, experiments from previous work show that in DIL methods like DER++ and ER perform much better than SGD (e.g., Buzzega et. al 2020). This is very different from the chunking setting where one of our main findings is that no current CL method we tested performed better than SGD. Therefore, we hope that you can see that the chunking setting is clearly different from DIL. Additionally, we would like to emphasise here why the chunking setting is important to CL, including for DIL. The chunking setting is a sub-problem of CL, including DIL, which is not currently solved or looked at and it is necessary to improve performance on it for CL methods to approach the performance of offline learning.

Reviewer comment (Theory):
The paper argues that there is no task shift in chunking (while domain-incremental learning assumes a task shift like rotation or permutation of the original data). However, this argument is not convincing, as task shift is hard to measure. For example, it is unclear which has a larger task shift: (1) from digit 1 to digit 4 (chunking); or (2) from digit 1 to a rotated digit 1 (domain-incremental learning).

Author response:
From your review it seems your main criticism is that you think the chunking setting is the same as domain incremental learning (DIL). However, the chunking setting and DIL are quite different settings. This is because by definition DIL has a task/distribution shift element, as p(x) is different for each task; while the point of the chunking setting is to only look at the chunking sub-problem and not the task-shift element of CL and so by construction it has no task/distribution shift element. Also, experiments from previous work show that in DIL methods like DER++ and ER perform much better than SGD (e.g., Buzzega et. al 2020). This is very different from the chunking setting where one of our main findings is that no current CL method we tested performed better than SGD. Therefore, we hope that you can see that the chunking setting is clearly different from DIL. Additionally, we would like to emphasise here why the chunking setting is important to CL, including for DIL. The chunking setting is a sub-problem of CL, including DIL, which is not currently solved or looked at and it is necessary to improve performance on it for CL methods to approach the performance of offline learning.

Reviewer comment (Novelty):
The notation of chunking appears to be a word game and the reviewer could not see its significance on its own or the difference from prior works.

Author response:
From your review it seems your main criticism is that you think the chunking setting is the same as domain incremental learning (DIL). However, the chunking setting and DIL are quite different settings. This is because by definition DIL has a task/distribution shift element, as p(x) is different for each task; while the point of the chunking setting is to only look at the chunking sub-problem and not the task-shift element of CL and so by construction it has no task/distribution shift element. Also, experiments from previous work show that in DIL methods like DER++ and ER perform much better than SGD (e.g., Buzzega et. al 2020). This is very different from the chunking setting where one of our main findings is that no current CL method we tested performed better than SGD. Therefore, we hope that you can see that the chunking setting is clearly different from DIL. Additionally, we would like to emphasise here why the chunking setting is important to CL, including for DIL. The chunking setting is a sub-problem of CL, including DIL, which is not currently solved or looked at and it is necessary to improve performance on it for CL methods to approach the performance of offline learning.

Reviewer comment (Novelty):
Based on the understanding that chunking is an instance of domain-incremental learning, the experiments are not of interest. The experimental results are natural consequences of domain shift and the specific training setups.

Author response:
From your review it seems your main criticism is that you think the chunking setting is the same as domain incremental learning (DIL). However, the chunking setting and DIL are quite different settings. This is because by definition DIL has a task/distribution shift element, as p(x) is different for each task; while the point of the chunking setting is to only look at the chunking sub-problem and not the task-shift element of CL and so by construction it has no task/distribution shift element. Also, experiments from previous work show that in DIL methods like DER++ and ER perform much better than SGD (e.g., Buzzega et. al 2020). This is very different from the chunking setting where one of our main findings is that no current CL method we tested performed better than SGD. Therefore, we hope that you can see that the chunking setting is clearly different from DIL. Additionally, we would like to emphasise here why the chunking setting is important to CL, including for DIL. The chunking setting is a sub-problem of CL, including DIL, which is not currently solved or looked at and it is necessary to improve performance on it for CL methods to approach the performance of offline learning.

Reviewer comment (Experiments):
In Figure 2, the chunk size is reduced from $>10^4$ to $10^2$, while the paper trains for 50 epochs in all cases. This means the model is given $10^2$ samples for each task and trained for $50$ epochs, which would easily lead to overfitting. It would only be surprising if the model does not forget previous knowledge. Moreover, for the chunk size of $10^2$, the size of the test set is unclear and likely much smaller, making it easily biased and not representative of performance over the entire distribution.

Author response:
We validated what number of epochs to use for each chunk size and choose the number of epochs which gave the best performance (50 for CIFAR-10 and CIFAR-100 and 100 for Tiny ImageNet). This is described in Appendix C, however we notice that we did not mention this in the main paper and so thank you for pointing it out as we have now added mention of it in the main paper.

Additionally, in your comment you guess that the size of the test set scales with the chunk size. This is incorrect as the size of the test set is fixed for any chunk size and is the standard test set used for the datasets used in the paper. We hoped to of explained this in Section 3 but given your comment we have edited the paper to make it clearer.

Reviewer comment (Evaluation):
The performance on Cifar100 and Tiny ImageNet in Table 2 does not look good. For example, DER++ has an accuracy of 5.48, and weight averaging improves this to 8.53, which is still smaller than 10%. This raises the question of whether weight averaging can really be claimed as a victory. From this table, the reviewer can only reason that none of the methods works in the proposed setting, including weight averaging.

Author response:
You mention in your review that for online class-incremental learning (CIL) for TinyImageNet the performance of DER++ with and without per-chunk-weight averaging is not very high, being under 10\%. We agree with you; online CIL is a challenging setting which has not been solved to a good degree yet. However, this misses the point of Table 2, which is to empirically show that a method which is designed to improve performance in the chunking setting can also improve performance in full CL settings, where there is task-shift.

Reviewer comment (Evaluation):
One of the paper's main conclusions "forgetting is the main reason for performance drop in chunking setting" is not logically supported by evidence. The authors observed forgetting on the training set in Figure 5 and a performance drop on the test in Figure 2, and claim that forgetting (on the training set) leads to a performance drop (on the test set), which does not appear to have a causal relationship. 

It is shown in Figure 5 that there is significant forgetting of the overfitting features on previous training chunks, but it is not clear whether the learned generalizable features are forgotten (previous chunk accuracy never dips below test accuracy). Learning overfitting features is not causal for generalization performance, as one can often train good-performing models with little overfitting by adjusting hyperparameters and/or using regularization. Therefore, forgetting of overfitting features is also not causal for losing generalization performance.

There is a simple explanation of the observed forgetting on past chunks: overfitting features are, by definition, unique to a specific combination of training examples, so overfitting features learned in one chunk are not experienced again in subsequent chunks, leading to their forgetting. Generalizing features, however, are repeatedly experienced in multiple chunks and should manifest much lower forgetting (at least in theory). Ultimately, we are concerned about the forgetting of generalizing features rather than overfitting features, as the former determines the final generalization performance.

Author response:
Yes, we agree with you we could be clearer on how the forgetting of a chunks training data impacts performance. However, there is a relationship as it is expected that training performance should be roughly greater than or equal to test performance, as the learner has trained to fit the training data well (this is precisely true, in expectation, for empirical risk minimisation methods and is true for our experiments). Therefore, given that the performance on the training data of a chunk quickly drops to a low level after learning on it for standard CL methods (i.e., they forget), in the near future the test performance can be no better than this training performance. Hence, the drop in training performance of a chunk after learning on it, that is the amount of forgetting, affects the current possible best case test performance. We also show that per-chunk weight averaging gets much better performance in the chunking setting than the CL methods we tested and it also forgets a lot less, giving experimental evidence to the claim also. We hope now that we have satisfactorily answered your question on this matter and thank you for it as we think it will strengthen our submission.

Reviewer comment (Experiments):
How would the Adam optimizer perform compared to plain SGD? If the moment estimations of Adam optimizer persist across chunks, maybe it could have a similar smoothing effect similar to weight averaging.

Author response:
In our exploration of the best learning setup we did look at using momentum, weight decay and other optimisers. However, we found the best performance across all methods was to use the setup that we use in the main paper, which is also the same as previous work (Buzzega et. al 2020). Additionally, we note that in our experience of CL using Adam usually performs worse than SGD.

Reviewer comment (Experiments):
Epoch numbers and learning rates are set to the same for different chunk sizes: I personally don't think that this is a very fair setting as the author believes. When training many epochs on a small chunk, the model will see the same examples more frequently and with shorter intervals, which could lead to more severe overfitting and negatively impact generalization. I would suggest finding optimal hyperparameters separately for different chunk sizes (maybe divide all chunk sizes into several ranges and find one set of hyperparameters for each range).

Author response:
We validated what number of epochs to use for each chunk size and choose the number of epochs which gave the best performance (50 for CIFAR-10 and CIFAR-100 and 100 for Tiny ImageNet), as described in Appendix C. Additionally, we validated our choice of learning rate and again saw that setting it two the same value for each chunk size gave the best performance, as described in Appendix A. However, we notice that we did not mention this in the main paper and so thank you for pointing it out as we have now added mention of it in the main paper.

Reviewer comment (Theory):
In my opinion (offering to the authors for pure discussion), the chunking problem may not be so different from a distribution shift, at least for small chunk sizes. With smaller chunks, the difference in the chunks' examples statistics will increase, even if examples are all sampled from the same distribution. For example, when the chunk size is small, some chunks will inevitably happen to have more images with indoor backgrounds, and some happen to have more outdoor backgrounds. This could be impossible to tell from true distribution change, and the model could still struggle with the changing examples statistics.

Author response:
We agree with you that small changes in distribution can be imperceptible from the differences between samples drawn from the same distribution (e.g., the difficulty of two sample testing). Additionally, we agree with you that the difficulty of the chunking setting comes from the empirical differences between chunks. However, this is not the same kind of shift that CL methods currently aim to tackle as shown by the fact the ones we test do not perform any better than SGD in the chunking setting. Therefore, there is a gap in the literature for researchers to solve this kind of shift and we show that by doing so it is likely to improve performance in the full CL setting, as chunking is a sub-problem of any CL setting. Lastly, I hope the reviewer agrees with us that the difference between i.i.d. samples and non-stationary processes can be quite distinct in theoretical results.

Reviewer comment (Novelty):
[Page-1, Section-1, Para-3] When a model sequentially learns over a sequence of datasets (in this case chunks) without having the measures of retaining the past knowledge it tends to forget the past learning as evident in the CL literature in both cases of homogeneous and heterogeneous data (chunk) distributions. Therefore, it is expected that the model performance will drop in such cases, hence the second claim is a valid expectation in such a setting and does not constitute a significant outcome of the analysis.

Author response:
We agree that the observation that learning over a stream of chunks results in a performance drop from offline learning is not surprising. However, we do think the finding that this performance drop is large enough to be a significant part of the performance drop of CL from offline learning is an important finding. This is because it suggests that a necessary step towards solving CL in general is improving the performance in the chunking setting. Additionally, given our finding that current CL methods do not improve performance over SGD in the chunking setting, we have shown that the chunking problem has been overlooked and is necessary to solve CL in general.

Reviewer comment (Experiments):
In practical scenarios where task boundaries are not pre-known or specified, per-chunk weight averaging could easily worsen the model performance as averaging is done without consideration of the current chunk's domain/distribution as compared to the past chunks.

Author response:
Also, you comment that in CL settings where task-boundaries are not known per-chunk weight averaging could hurt performance. This point might certainly be true. However, the reason we look at per-chunk weight averaging is to show that we can improve performance in the chunking setting (where there are no task boundaries) and that this improved performance in the chunking setting can transfer to improved performance in the most common CL settings. Therefore, we see your point as valid but not in the scope of this work as in this work we focus on exploring the benefits of looking at the chunking setting and not how to adapt methods to CL settings without task boundaries (but with distribution shift).

Reviewer comment (Novelty):
Per-chunk weight averaging can be seen as the simplest form of knowledge aggregation technique in a continual learning setting. Hence, it reduces forgetting in the ideal (class-balanced) scenario which is the usual expectation from such techniques and does not constitute high significance as more sophisticated methods have already been developed in CL literature like weight regularization, data replay and incorporating additional parameters.

Author response:
In your review you ask why we look at per-chunk weight averaging in the chunking setting and not a more complex knowledge aggregation method already proposed in the CL literature. This is because one of our key findings is that current knowledge aggregation CL methods (EWC, DER++, ...) do not perform any better than plain SGD learning in the chunking setting. Therefore, to improve performance in the chunking setting we had to start from scratch and from an analysis of the linear case we saw that per-chunk weight averaging could improve performance, which we empirically demonstrate.

Reviewer comment (Experiments):
The chunking setting described in the papers is ideal and data for such settings is also generated under simplified and impractical assumptions which will not scale to the online settings with changing data distributions in real-world scenarios.

Author response:
In your review you state that the chunking setting is not a suitable model for many real-world scenarios. We agree with this statement and state as much already in the paper ([Page-2, Section-2, Para-3]). However, this misses the point of the paper which is that any more realistic CL setting with task/distribution shift (including ones without task boundaries) can be decomposed into two sub-problems, one of which is the chunking problem. We show that the chunking problem is a significant part of the most common CL settings and has been underexplored as no current CL methods performs better than SGD on it. Therefore, a way to improve on any real-world CL problem would be to improve performance on the chunking problem, which is a key point of our work and is currently underexplored.

Reviewer comment (Writing):
Inconsistent use of terminologies. Please clarify the following terminologies (maybe in a tabular format) so that the reader can refer to them whenever required:  
- "offline learning"  
- "plain SGD learning"  
- "full CL setting"  
- "standard CL"  
- "online CL"

Author response:
Thank you for pointing this out. While in the paper we have whenever we introduce a term from the CL literature introduced what it means, from your comment we see that this could be clearer and so have edited the paper accordingly.

Reviewer comment (Writing):
[Page-1, Section-1, Para-1] "thwart" -> "thwarted", "focuses CL" -> "focuses of CL"

Author response:
Thank you for pointing out these typos, we have fixed them in the paper.