Paper: Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning (ICLR-2024)

Reviewer comment (Evaluation):
It appears that S-EWC does not have too much of a gap with UPGD judging from figure 7: it entirely avoids catastrophic forgetting, and the only setting where it loses plasticity where UPGD does not is on MNIST.

Author response:
While S-EWC is effective in catastrophic forgetting evaluation, it suffers from loss of plasticity, which was the aspect we evaluated in the input-permuted MNIST task. Only UPGD addressed both loss of plasticity and catastrophic forgetting, which is shown by its performance on both evaluations.

Reviewer comment (Evaluation):
S-MAS outperforms UPGD on miniImagenet at the end of training, and does not have a large gap overall.

Author response:
While we do not expect a new method to outperform in every single task, UPGD still achieves higher average accuracy in the first half of the experiment. It is desirable in streaming learning where the online accuracy at each time step is more important than the accuracy only at the end of training.

Reviewer comment (Experiments):
The ablation of figure 8 checks the contribution of each component of UPGD sequentially as they are added to regular SGD. Ideally, the ablation would study how each component affects UPGD when they are *individually* removed (e.g. UPGD without WP).

Author response:
We conducted the requested ablation. Please see this figure [[link]](https://drive.google.com/file/d/15aeYYq3QjeEgAbeFVeqzSNT5Ms2QvSYr/view?usp=sharing). The requested ablation bolsters the contribution of utility gating even more. Using utility gating on top of sgd makes sgd maintain its performance instead of dropping on input-permuted MNIST and makes sgd improve its performance continually on label-permuted problems. The role of weights decay and weight perturbation is not significant in label-permuted problems but including both with utilty gating improves performance and plasticity on input-permuted MNIST.

Reviewer comment (Presentation):
Figure 7 is referred to before Figure 6; ideally, their order would be swapped.

Author response:
Thank you for pointing out the order of mention for Figures 6 and 7; we will make the necessary amendments. The error margins represent standard error. Our results are averaged over 20 trials. We use non-overlapping intervals to average over. Therefore, our plots can have variabilities even though we have a large number of trials. We don’t average by using an exponential moving average, which will smoothen out the curves since we are interested in what each point in the plot means. For example, each point in Figure 3 represents the average performance over a single task. Smoothing over time wouldn’t give the same meaning.

Reviewer comment (Evaluation):
The results are averaged over 20 trials, but the meaning of the error margins in some of the figures is not made clear (e.g. figure 2).

Author response:
Thank you for pointing out the order of mention for Figures 6 and 7; we will make the necessary amendments. The error margins represent standard error. Our results are averaged over 20 trials. We use non-overlapping intervals to average over. Therefore, our plots can have variabilities even though we have a large number of trials. We don’t average by using an exponential moving average, which will smoothen out the curves since we are interested in what each point in the plot means. For example, each point in Figure 3 represents the average performance over a single task. Smoothing over time wouldn’t give the same meaning.

Reviewer comment (Experiments):
It is suggested to increase the number of trials to smooth out the curves if possible.

Author response:
Thank you for pointing out the order of mention for Figures 6 and 7; we will make the necessary amendments. The error margins represent standard error. Our results are averaged over 20 trials. We use non-overlapping intervals to average over. Therefore, our plots can have variabilities even though we have a large number of trials. We don’t average by using an exponential moving average, which will smoothen out the curves since we are interested in what each point in the plot means. For example, each point in Figure 3 represents the average performance over a single task. Smoothing over time wouldn’t give the same meaning.

Reviewer comment (Theory):
Is it possible to show theoretical performance guarantees for UPGD? For instance, can the approximation error of equation 2 be bounded? Alternatively, if the true utilities are used in equation 3, is it possible to derive some guarantees against forgetting or loss of plasticity?

Author response:
We think this would be interesting direction that is left for future work.

Reviewer comment (Evaluation):
How much more significantly does UPGD improve upon baselines S-EWC and S-MAS?

Author response:
Based on our results, we can see that UPGD outperforms S-EWC in all experiments and outperforms S-MAS in 3 out of 4 experiments.

Reviewer comment (Experiments):
How does UPGD-W perform with WP and WD removed individually?

Author response:
Please see this figure [[link]](https://drive.google.com/file/d/15aeYYq3QjeEgAbeFVeqzSNT5Ms2QvSYr/view?usp=sharing), which includes the requested ablation.

Reviewer comment (Novelty):
First and foremost, I do not think this paper makes a significant contribution. The methodology is incremental in that it combines two well-known ideas (perturbed gradient descent + keeping active neurons unchanged).

Author response:
We respectfully disagree with the reviewer. First, our method does not work by keeping active neurons unchanged. Our method uses a utility-gating mechanism to protect useful weights from change and perturb less useful ones. This idea is compatible with weight perturbation and weight decay, which makes the method stronger. Second, our novel utility-gating mechanism presents an instance of a new class of methods that has not been introduced before. We kindly ask the reviewer to refer us to works to which UPGD is considered incremental.

Reviewer comment (Experiments):
Second, it is tested on very toy setups. The experiments are not convincing enough to show the applicability of the method to interesting real-world setups. For instance, I am not sure the networks achieve similar plasticity if tested on, e.g., webcam data instead of MNIST, where the feature space is a lot richer and hence plasticity is much more difficult.

Author response:
Our tasks are widely used in continual learning for evaluating forgetting and plasticity (see [1,2,3,4]), where input-permuted tasks were used for evaluating plasticity and output-permuted tasks were used for evaluating forgetting. We certainly acknowledge the reviewer’s valuable suggestion for real-world setups, such as webcam data. However, such benchmarks are unavailable, especially for streaming learning, due to its complexity, and requires a separate dedicated paper. As you recognize that our experiments verify the paper’s central claim and reviewer KmBd highlights their scale, solidity, and comprehensiveness, we believe they sufficiently demonstrate the core contribution of our work.

Reviewer comment (Theory):
Third, theoretical properties/implications of the method should be carefully examined.

Author response:
We provided theoretical work in the Appendix. Please refer to Appendix A for convergence analysis, Appendix B for how utilities can be propagated in the network, and Appendix D for the connection between weight and feature utilities.

Regarding the comment on Taylor expansion, our first and second-order Taylor approximations require ignoring higher-order terms, which is common and valid for our case when weights are proper fractions, making them sufficiently small. Our empirical evaluation shows the effectiveness of this approximation in continual learning. For future work, one can consider a different utility definition based on the change in loss from sample to sample, which depends on the difference between the values of the weight from two consecutive time steps, making the approximation more accurate (see [5] and Appendix M).

Reviewer comment (Theory):
For instance, the Taylor expansion would only hold if $W_{l,i,j}$ are infinitesimally small. We do not know in general if this holds or not. I suggest the paper should include a (preferably rigorous) discussion on this.

Author response:
It is true that the Taylor approximation produces small errors when some conditions are satisfied. In our case, weights only need to be a small fraction, not infinitesimally small, to drop the higher-order terms, which we also mentioned above. Typically, the vast majority of the weights in a network are small fractions. For example, the last plot of Figure 5 in the paper shows SGDW has an L1 weight norm of ~6000 for a network with 282160 weights, giving an average weight magnitude of ~0.02. Moreover, we use weight decay, which also helps keep the weights small. Thanks for pointing this out. We will clarify it further in our paper.

Reviewer comment (Theory):
Likewise, gradient descent is no longer steepest descent but some approximation to it. Investigating why it works is important. As shown by the results, no collapse occurs but again, I wonder how this translates into more challenging settings where utilities of most parameters are high.

Author response:
In Appendix A, we provided convergence analysis for our method, where we showed that our method converges to a stationary point, similar to other methods (e.g., SGD). Our empirical evaluation in Figure 9 shows that UPGD also outperforms SGD in stationary problems.

Reviewer comment (Writing):
_Although a few methods address both issues simultaneously, such methods expect known task boundaries, maintain a replay buffer, or require pretraining, which does not fit streaming learning._ <--- reference needed for this claim.

Author response:
Thank you for letting us know. We will add these references in the paper revision (see [6,7]).

Reviewer comment (Writing):
What does "a Hessian diagonal approximation in linear complexity" mean? Linear in the number of parameters?

Author response:
This is correct: linear in the number of parameters.

Reviewer comment (Experiments):
Does "each learner is trained for 1M samples, one sample each time step" mean gradient descent using one sample only? Is this realistic?

Author response:
That is correct. The update is made based on one sample at a time. This is often used as the more challenging constraint of streaming learning [e.g., 8]

Reviewer comment (Theory):
Though empirical evidence provided in the paper suggest it does (in that it works), I am not sure that the proposed definition of weight utility make sense. The power of neural networks (and the problem of the interpretation of its computation) is its distributed computation. Utility of an individual weight is almost always nothing - in fact, quite often any particular weight, sometimes even large number of weights, can be taken out of the network, with little impact on performance. So, it's more about combinations of weights working together...and the proposed utility doesn't measure that. I understand that evaluating utility of combinations of weights is intractable, but I worry that this simplification, of judging utility of each weight in isolation, is encouraging less distributed representation, which might come with a penalty in performance.

Author response:
We agree with the reviewer that a better utility definition would include the interaction between weights (as we pointed out in Appendix M). However, including such interactions makes the method intractable, as the reviewer mentioned. Most methods addressing forgetting (e.g., EWC) use individual weight measures, and yet they are effective. While we are interested in learning about the kind of representations learned by each method, we believe this merits a dedicated paper to address fully. From our experiment, however, we found that UPGD improves performance over SGD even in the stationary case (see Figure 9 in Appendix F), likely due to finding better representations.

Reviewer comment (Evaluation):
Fundamentally, on the forgetting front, the proposed method is just another weight consolidation method, and it's a bit hard to believe it beats Elastic Weight Consolidation. I am not 100% sure that the proposed method doesn't favour plasticity over forgetting nor that the forgetting evaluation isn't biased towards methods that favour plasticity (see questions below).

Author response:
While there are strong connections and similarities between utility-gating and regularization-based methods, our approach differs from Elastic Weight Consolidation (EWC) in two major ways: 1) In EWC, a regularization term is essential, while our method has no such term., and 2) our proposed utility measure is different than EWC's squared gradient metric. The results from Figure 2 show the empirical superiority of our utility measure over the squared gradients, which partially explains the performance difference. More future theoretical work is needed to link between these two classes of methods. 

In our catastrophic forgetting (CF) evaluation, methods that only address CF can achieve high performance (e.g., S-MAS) since the problem is designed to favor methods that keep useful representations and improve on them continually. We do not think our CF evaluation is biased towards plasticity. Plasticity-inducing methods (e.g., Shrink and Perturb) achieve low performance in CF evaluation.

Reviewer comment (Theory):
Though I understand (and like) in principle what the utility-based update is supposed to do, I can't quite understand why it actually works. The proposed measure of the utility of parameters is a measure with respect to the loss on the new input/output pair. If this pair comes from a new task, how does measuring utility of the model parameters with respect to the loss of this new task have bearing on the utility of the parameters for the old tasks? Just because utility of a given weight is, say, low for the current sample, it doesn't mean it's low for previous samples. It seems to me that the proposed method would score high on plasticity (it finds available weights for new task)...but I don't see how it protects against forgetting, in principle.

Author response:
The utility measure we use is based on an exponential moving average of the sample utilities, as we explain in the last sentence of the second paragraph of section 4.2. Thus, it does not suffer from the issue the reviewer mentioned about sample instantaneous utilities.

Reviewer comment (Experiments):
I don't understand how 4.3 measures catastrophic forgetting. Permuting labels of CIFAR-10 with the new tasks suggests to me that it's all about plasticity again. Shouldn't it be an experiment, where labels are kept intact, but new tasks are added...and previous tasks examples are not used? Am I missing something about how experiments reported in 4.3 are done?

Author response:
Section 4.3 indeed evaluates catastrophic forgetting, as previously learned representations transfer and remain useful from one task to another. Hence, methods that forget previously learned representations should suffer in performance. Our results also confirm that methods that protect useful representations and improve them with more data achieve higher performance (e.g., S-MAS). A method that does not protect useful representations (e.g., SGD) forgets them by overwriting useful information and, therefore, does not improve its performance continually.

Reviewer comment (Evaluation):
Why are the accuracy results of training on CIFAR-10 and EMNIST so poor in Figure 6? State of the art CIFAR-10 is close (or above) 90%. Something close to 80% would be probably still acceptable...but 60% is quite poor. I am not exactly sure what EMNIST variant entails, but is 70% accuracy a good accuracy for this dataset? It is often easy to show improvements of something at the low end of the models' performance, but that doesn't always translate to same effect at the high (or close to) end of the models' performance...and in the end, the latter is what we really care about. So, does the proposed method prevent forgetting at the high end, when model is performing at or reasonably close to state of the art?

Author response:
In our streaming non-stationary problems, achieving state-of-the-art results as in stationary settings is unexpected due to their incomparability. Please see Figure 2b (rightmost of the three) in [1], where accuracy decreases with increased permutation frequency: from 99% (permuting every 1M samples) to about 88% (every 10K). Our experiment, with permutations every 5K samples, achieves a reasonable best accuracy of around 78%, aligning with this trend. 

In streaming learning, the evaluation is done on-the-fly at every time step, so maximizing the online performance (accuracy) at every step by reducing forgetting is desired. It’s a different problem setting from the one the reviewer described, where the final performance is what is important.

Reviewer comment (Experiments):
This is not a massive issue, but does the per batch normalisation of utility make the performance of the method variable with different mini-batch size settings?

Author response:
In our paper, we used a batch size of 1 (streaming case). Our method can also work with mini-batches where batch normalization might help.

Reviewer comment (Theory):
My main point of issue is the reasoning behind the perturbations in the update rule. The authors claim that by adding the perturbation we are making the unimportant weights more plastic; however, I am not really convinced by this explanation. I believe it requires elaboration both in the rebuttal and in the paper.

Author response:
Noise injection has been shown in the literature of plasticity as a way of introducing plasticity (see [1]). In Figure 3 of our work, S&P maintains its plasticity compared to SGD with weight decay, mainly because of noise perturbation (which is unscaled). While noise perturbation alone might not be effective (see PGD in the same figure), its role is pronounced when used with weight decay (S&P) or when used on a per-feature level [1]. We will add a sentence to make this clearer in our discussion of the method.

Reviewer comment (Theory):
Another related issue with the proposed perturbation is the fact that all of them are getting drawn from the same standard normal distribution. This design choice is strange to me since the parameters of a neural network usually differ in magnitude from layer to layer. By adding an unscaled random perturbation to all of the weights we are ignoring this scale difference, which I believe is sub-optimal. I know that in the unprotected version, they are getting weighted by different values, but this particular scaling is more correlated with changes in the loss value rather than the parameter magnitudes.

Author response:
First, we note that the standard deviation of the noise perturbation is a hyperparameter that can be tuned, which is effectively a scaling of the perturbation (see Table 2 and Table 3 for the values used). Second, we agree that noise perturbation can also use explicit layer-wise scaling. However, our utility scaling takes into account the importance of weight, meaning that weights in the earlier layers generally have higher utility and, therefore, perturbed less. Finally, we acknowledge that there are always ways to improve any method (we discussed some of them in Appendix M). Yet, our simple design was found to be very effective compared to other methods.

Reviewer comment (Experiments):
Highly relevant to the above issue, I believe it is also necessary to have an additional ablation study investigating the role of having and not having the perturbations in the update rule. I also want to disentangle the effect of weight decay. The only time that UG is added in the ablation is in the presence of WD. More specifically I am curious about the following scenarios in Figure 8:  
- SGD + UG + WP + WD (present in the paper)  
- SGD + UG + WP  
- SGD + UG + WD  
- SGD + UG

Author response:
We conducted the requested ablation. Please see this figure [[link]](https://drive.google.com/file/d/15aeYYq3QjeEgAbeFVeqzSNT5Ms2QvSYr/view?usp=sharing). The requested ablation bolsters the contribution of utility gating even more. Using utility gating on top of sgd makes sgd maintain its performance instead of dropping on input-permuted MNIST and makes sgd improve its performance continually on label-permuted problems.

Reviewer comment (Experiments):
In the experiments section I believe the authors need to include more diverse experiments. All of the streaming tasks are permutations of the same task, whether in the label or in the input space. It is not as obvious as the authors claim that after the permutation of the input space the previously learned representations are not relevant anymore (end of page 6). In the input-permuted scenario, only the first layer needs to have significant change. This is especially true for the label-permuted tasks, as the network does a good job of clustering the data up to the final FC layer. I encourage the authors to use the Cifar100 superclass dataset (or any similar sequence of tasks that does not simply rely on the permutation).

Author response:
We agree with the reviewer on the comment about the phrase “previously learned representations are not relevant anymore”. We meant to use the word "features" instead of "representations". We will correct it in our revision. 

The problems we use are extensively adopted by the continual learning community to evaluate forgetting or plasticity in a controlled manner. We tried split-based tasks, as the reviewer kindly suggested, with CIFAR 100, but we found that these problems become very easy to the extent that all methods achieve an online accuracy of above 98% because, in streaming learning with long sequences, tasks have to re-occur which makes the problem easy (see [2]). Therefore, we chose label-permuted tasks since task re-occurrence probability becomes very small. Finally, due to the complexity and significance of designing realistic benchmarks suitable for streaming learning, we believe it merits a dedicated paper to address fully.

Reviewer comment (Presentation):
I believe the visualization needs several improvements: the legends on the plot are very hard to read (Figures 2, 3, 4, 5). Some colors are similar to each other and the width of the lines in the legends is too thin (especially in Figure 4). In Figure 7, some numbers in dark blue cells are almost impossible to read.

Author response:
We thank the reviewer of this note. We will improve visualization in our revision.

Reviewer comment (Experiments):
**Q1:** Have the authors tried to use a scaled version of the perturbation that takes the magnitude of the parameters into account? (Other than the unprotected version). Also, I would appreciate it if you could elaborate on the effect of perturbations.

Author response:
*A1:* The weight perturbations in UPGD are still scaled by utility. They are just scaled by the same utility the gradients are scaled by. Future work can study or improve the relative scaling between weight perturbation and gradient information.

Reviewer comment (Evaluation):
**Q2:** Could you also explain about the average online accuracy? It is stated that "The average online accuracy is the percentage of correct predictions within each task." I cannot see the average part here. Is it calculating the accuracy on each task separately then averaging over the number of tasks?

Author response:
*A2:* In streaming learning, the learner receives a data stream and outputs a stream of predictions that are either correct (sample accuracy of 1) or incorrect (sample accuracy of 0). The average accuracy on a certain task is the average of these sample accuracies, which is also the percentage of correct predictions since the sample accuracies are either zero or one. We show the online average accuracy on each of the sequential tasks we have, so it is an average over each task separately.