Probing Representation Forgetting in Continual Learning
Abstract
Continual Learning methods typically focus on tackling the phenomenon of catastrophic forgetting in the context of neural networks.
Catastrophic forgetting is associated with an abrupt loss of knowledge previously learned by a model.
In supervised learning problems this forgetting is typically measured or observed by evaluating decrease in task performance.
However, a model’s representations can change without losing knowledge.
In this work we consider the concept of representation forgetting, which relies on using the difference in performance of an optimal linear classiﬁer before and after a new task is introduced.
Using this tool we revisit a number of standard continual learning benchmarks and observe that through this lens, model representations trained without any special control for forgetting often experience minimal representation forgetting.
Furthermore we ﬁnd that many approaches to continual learning that aim to resolve the catastrophic forgetting problem do not improve the representation forgetting upon the usefulness of the representation.
Introduction
Continual Learning (CL) is concerned with developing methods for learners to manage changing distributions.
The goal being to acquire new knowledge from new data distributions while avoiding forgetting of previous knowledge.
A common scenario is CL in the classiﬁcation setting where the class labels presented to the learner change over time.
In this scenario a phenomenon known as catastrophic forgetting has been commonly observed [13, 22].
This phenomenon is often described as a loss of knowledge about previously seen data and observed in the classiﬁcation setting as a decrease in accuracy.
Deep Learning has been traditionally motivated as an approach which can automatically learn representations [7], forgoing the need to design handcrafted features for data.
Indeed representation learning is at the core of deep learning methods in supervised and unsupervised settings [12].
In the case of many practical scenarios we may not be simply interested in the ﬁnal performance of the model but also the usefulness of its features for various downstream tasks [30].
Although a representation may change drastically at task boundaries [8], this does not necessarily entail a loss of useful information and may instead correspond to a simple transformation.
For example consider a standard multi-head CL setting, where each task shares a representation and only differs through task heads.
A permutation of features leads to total catastrophic forgetting as measured by standard approaches as the task heads no longer match with the representations, but this does not correspond to a loss of knowledge about the data.
As CL envisions having learners operate over long time horizons while continually maintaining old knowledge and integrating new information, it is sensible to consider the usefulness of their representations for previous tasks in addition to directly measuring the performance on previous tasks using the last layer classiﬁers.
In this paper we highlight that traditional approaches of evaluating forgetting are unable to properly disambiguate trivial changes in the features (e.g. permutation) from Conference on Systems (NeurIPS 2021). abrupt losses of useful representations.
We propose to use linear probes, commonly used to study unsupervised representations [11] and intermediate layer representations [25, 33], to evaluate CL algorithms and their usefulness.
In this work we revisit several classical CL settings and benchmarks and attempt to measure how much forgetting is observed by the representations using optimal Linear Probes (LP).
We observe that in many commonly studied cases of catastrophic forgetting the representations can be observed to avoid losing critical task information.
Related Work
Multiple approaches have been developed for CL, often the design of these methods is focused on mitigating the catastrophic forgetting phenomenon, with aspects such as maximizing forward and backward transfer between tasks taken as secondary [20].
One class of methods focuses on bypassing this problem by growing architectures over time as new tasks arrive [2, 17, 29, 31].
Under the ﬁxed architecture setting, one can identify two primary directions.
In the ﬁrst, methods rely on storing and re-using samples from the previous history while learning new ones, this includes approaches such as GEM [20] and ER [10].
The second class of methods encode the knowledge of the previous tasks in a prior that is used to regularize the training of the new task and includes approaches such as [1, 15, 19, 24, 34].
A classic method in this vain is Learning without Forgetting (LwF) [18], which mitigates forgetting by a regularization term that distills knowledge [14] from the earlier tasks.
Prior to learning a new task, the network representations are recorded, and are used during the training to regularize the objective by distilling knowledge from the earlier state of the network.
In Section 4 we will examine the effectiveness of this approach in mitigating representation forgetting.
Recent works on elucidating the nature of catastrophic forgetting have examined the inﬂuence of task sequence [23], network architecture [6], and change in representation similarity [27].
Our work is related in spirit to [27] as we pursue measuring how much forgetting has occurred on the learned representation and we additionally study this for depth.
However, in [4], the authors use linear
CKA [16] to measure the similarity between representations inﬂuenced by forgetting, while in our work we measure how much forgetting is observed by the representations using LP.
Several works [21] have focused on modifying the last layer of a classiﬁcation network to make more effective use of the representation for prior tasks.
This indirectly highlights that the last layer can be modiﬁed to yield better performance on prior tasks.
Particularly [21, 28] use a buffer of old examples at evaluation time to construct a class mean prototype.
This allows to more effectively use the representation of the network.
These works, however consider settings where CL methods are used to control training, while we also emphasize naive continuation of training under task shift can also yield strong representations.
Our work can also be seen as a way to explain and motivate the need for such approaches.
Linear Probes
Following work in supervised learning [11] and in the analysis of intermediate representations [33] we evaluate the usefulness of representations by an optimal linear classiﬁer using training data from the original task.
A linear classiﬁer is trained on top of the frozen activations of the base network given the training instances of a particular dataset.
The test set accuracy obtained by LP on the aforementioned dataset is used as a proxy to measure the quality of the representations.
The difference in performance of the LP before and after a new task is introduced acts as a surrogate measure to the amount of forgetting observed by the representations and is referred to as representation forgetting.
Experiments
We perform evaluations in several published CL scenarios, focusing on the task-incremental setting.
We ﬁrst consider the setup from [5, 27].
Subsequently we revisit the evaluations of [18] and ﬁnally we consider a longer task sequence in the online and ofﬂine setting with different model capacity [20].
In all cases LP are trained to convergence with Adam and a learning rate of 1e-3.
Two Task [-CIFAR
-]SplitCIFAR10 Sequence We consider a two task SplitCIFAR10 setting from [27].
We use the same models and training procedures and subsequently evaluate the forgetting experienced by the representations.
In Table 1, we study the shift in representations of each block of the network by measuring the performance of LP on Task 1 data before and after training the network on Task 2.
First observe that the model accuracy decreases from 85% to 63% suggesting a large degradation in performance and a large forgetting.
However, following the optimal classiﬁer evaluation protocol [Table] [Table] the accuracy degradation is observed to be only 6 . 8% , without any CL method applied to control forgetting.
This suggests the representations are still highly useful for Task 1 despite training on Task 2.
Second, similar to [27] we observe the forgetting is concentrated at the top layers.
Indeed early layers in the network experience almost no representation forgetting and in some cases improve their usefulness with regards to Task 1.
[27]’s analysis also showed forgetting occurring in early layers to a lower degree than in higher layers and suggested that forgetting is extreme in the upper layer representations.
Speciﬁcally, the authors measured linear CKA [16] performance between layers (Fig. 1 in [27]) showing this similarity metric dropped progressively from close to 1 to 0.2 for both ResNet and VGG models.
However, our evaluation suggest forgetting doesn’t exist in lower layers and the loss in information is less catastrophic at higher layers than suggested by [27].
ImageNet Transfer We now move to larger scale scenarios of models trained on large datasets and applied to a different task.
We take the setting of [18], which considers the ImageNet [30] transfer to various datasets, in particular CUB [32] and Scenes [26].
We use the same model (VGG-16) and training procedures described in [18].
The LwF method applies a regularizer to the training objective by distilling knowledge from the earlier state of the network, which constrains the optimization space of the parameters for the new task.
Table 2 shows the results in this setting which are our reproduction of [18] and additionally perform the LP evaluation using ImageNet data on Task 2 models.
Note that the LP training does not use any data augmentations.
Our evaluation reveals that although the traditional forgetting is much higher for ﬁne-tuning compared to LwF, the LP accuracy is nearly identical, especially for the CUB transfer task.
This suggests that LwF does not improve over naive ﬁne-tuning in terms of forgetting knowledge acquired on ImageNet.
Longer Task Sequences and Variable Model Capacity
So far we have studied two task sequence.
We now consider the SplitCIFAR10 benchmark popularly used in a variety of CL work [3, 9, 28] that contains a 5 task sequence.
We train models using a 5 task sequence and a multi-head setting.
We then evaluate a LP trained on all the data to compare the optimal classiﬁer performance across [Table] methods and across model capacity.
We consider both the online setting where the data samples are seen only once as well as the ofﬂine setting where the learner receives the entire set of task data and is allowed to train for 10 epochs.
In all cases we train with SGD and a learning rate of 0.01.
A number of recent works have illustrated that Experience Replay, particularly as the buffer size increases, is a strong baseline [10, 27] in this setting.
Thus we use Experience replay with both a small buffer, M=5 samples per class [Table] and a relatively large buffer, M=20 samples per class, to allow a representative comparison of ﬁne-tuning and popular CL methods.
To simplify the analysis we report the ﬁnal accuracy averaged on all tasks observed after the task sequence and the accuracy of a LP trained on all the training data.
Table 3 shows the results in both the online and ofﬂine setting for two different models.
One model is the modiﬁed Resent18 [20] with a width parameter of 20 used commonly in [3, 20] and the other is the same network but with all layers widened by a factor of 5.
First, we observe that as in the other cases the LP accuracy of ﬁne-tuning is higher than the observed accuracy, suggesting forgetting is less catastrophic than suggested by observed accuracy.
Secondly, we observe that the ﬁne-tuning evaluated using the observed accuracy is particularly deceptive in revealing how the model representations change both from online to ofﬂine case and especially with increasing capacity.
Using observed accuracy one would conclude that increasing width and capacity of the model without applying any CL speciﬁc method does not improve performance and can even decrease model performance overall due to forgetting.
This is consistent with the Appendix of [5], which evaluates only on observed accuracy.
However, if we observe the LP accuracy, it reveals a more clear picture of what occurs at the representation level, suggesting that larger models can indeed reduce forgetting even when trained from scratch without explicit control of forgetting.
Moreover we observe that at the representation level as model capacity increases, naive ﬁne-tuning becomes much closer in performance to costly (and under privacy constraints unusable)
CL methods such as ER which use more computation and memory.
Contrary to the observations of [5] our results illustrate that the model capacity does play a profound importance on forgetting even when the model is trained from scratch.
That is the observed overall accuracy at the end of the sequence does not greatly increase as the model widens, while the LP accuracy does greatly increase and yields representations that are much closer for ﬁne-tuning and methods which explicitly combat forgetting, some with large buffers.
Conclusion
We have highlighted the importance of evaluating representations and not just task accuracy in CL settings.
Our results suggest a) the feature forgetting under naive training in supervised settings is not as catastrophic as other metrics suggest b) we reconﬁrm that forgetting is concentrated at the top layers and show that forward transfer can happen in lower layers under naive (non-CL speciﬁc) training and c)
We demonstrate that without evaluation of features the effects of model size on forgetting and representation learning will be misinterpreted.  Acknowledgements
This work is supported by NSERC Discovery Grant "Towards Continual Learning in the Visual
APPENDIX Reproducing LwF Results
We followed the training procedure as closely as possible to the ones reported by [19].
However, our results are slightly different from the ones reported due to variations.
Table 4 highlights these differences.
[Table]
Finetune-[19]
[Table]
Finetune-Ours
[Table]