E FFECT OF MODEL AND PRETRAINING SCALE ON CATASTROPHIC FORGETTING IN NEURAL NETWORKS
A BSTRACT
Catastrophic forgetting presents a challenge in developing deep learning models capable of continual learning, i.e. learning tasks sequentially.
Recently, both computer vision and natural-language processing have witnessed great progress through the use of large-scale pretrained models.
In this work, we present an empirical study of catastrophic forgetting in this pretraining paradigm.
Our experiments indicate that large, pretrained ResNets and Transformers are signiﬁcantly more resistant to forgetting than randomly-initialized, trained-from-scratch models; this robustness systematically improves with scale of both model and pretraining dataset size.
We take initial steps towards characterizing what aspect of model representations allows them to perform continual learning so well, ﬁnding that in the pretrained models, distinct class representations grow more orthogonal with scale.
Our results suggest that, when possible, scale and a diverse pretraining dataset can be useful ingredients in mitigating catastrophic forgetting.
Continual learning is a current challenge in designing machine learning systems: when trained on multiple tasks in sequence , models tend to suffer performance losses on earlier tasks—this is known as catastrophic forgetting (McCloskey & Cohen, 1989)
Catastrophic forgetting occurs in many settings from supervised learning for classiﬁcation (Zenke et al., 2017) to reinforcement learning for games (Kirkpatrick et al., 2017) and downstream applications such as translation (Bapna & Firat, 2019) or medical diagnoses (Gupta et al., 2021).
While many methods have been proposed to help mitigate forgetting none of these have fully solved the problem; see Kemker et al.
(2017) for a review.
Increasing model and dataset size have been key to the success of deep learning (Kolesnikov et al., 2020; BIG-bench collaboration, 2021;
Radford et al., 2019; Zhai et al., 2021).
A recent trend has been the use of increasingly large models pretrained on diverse datasets, which are subsequently ﬁne-tuned or evaluated without ﬁne-tuning on downstream tasks of interest; see Bommasani et al.
(2021) and references therein.
A particularly striking realization of this has been large pretrained language models, which show consistent improvement across scales (Kaplan et al., 2020) and can perform well on diverse tasks with no ﬁnetuning (Brown et al., 2020).
From a continual learning perspective, this behavior is especially dramatic as, without ﬁnetuning, any training signal for these tasks appears mixed into a drastically larger pretraining dataset, and thus more than likely appeared many gradient steps before evaluation.
This observation in large part inspired our work.
Here, we empirically study continual learning in large pretrained models.
Our results suggest that pretrained models are signiﬁcantly more resistant to catastrophic forgetting than their randomlyinitialized counterparts, and that this resistance improves with the scale of both the model and pretraining data.
Most of our experiments concern vision—speciﬁcally image classiﬁcation; but we also conduct a preliminary investigation of language models.
Our key observations are as follows:
• Resistance to catastrophic forgetting improves with model and pretraining dataset scale (Figure 1).
• Large, pretrained image-classiﬁcation models are signiﬁcantly more resistant to forgetting than models trained from random initialization (Figure 6).
• Both supervised and unsupervised, SimCLR (Chen et al., 2020), pretraining improve forgetting (Supplement, Figure 12).
• We perform initial experiments aimed at exploring why scale and pretraining reduce catastrophic forgetting, showing that distinct class representations in pretrained models are more orthogonal than in trained-from-scratch models (Figure 9).
• We observe similar phenomena in language models (Supplement, Figure 11).
R ELATED WORK
Catastrophic forgetting Catastrophic forgetting was identiﬁed as a problem for neural networks as early as the late 1980s, as it occurred in the single-hidden-layer networks that were studied at the time (see Delange et al.
(2021) for a review).
Within the catastrophic forgetting literature, a number of potential solutions to the problem have been devised; some involve replaying previous-task examples during training on later tasks (Rebufﬁ et al., 2016; Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018; Shin et al., 2017; Riemer et al., 2018), while others like elastic weight consolidation (Kirkpatrick et al., 2017) limit the extent to which parameters can change once learned (Zenke et al., 2017; Lee et al., 2017; Farajtabar et al., 2019).
Perhaps most similar to our work are proposed mitigation methods which exploit the overparameterization of modern neural networks to perform continual learning of multiple tasks, e.g. by using pruning in Mallya & Lazebnik (2018) or task-speciﬁc subnetworks in Wortsman et al.
and recent works studying the impact of pretraining Anonymous (2022) and overparameterization via network width Mirzadeh et al.
(2021) on forgetting.
Large-scale models
The observation—initially in language models (Kaplan et al., 2020)—that model performance scales predictably with model and dataset size spurred an increased interest in training and deploying increasingly large pretrained models.
Large-scale models emerged in natural-language processing with models such as GPT (Radford et al., 2019) and BERT (Devlin et al., 2019), which are pretrained in an unsupervised manner and thus able to make use of the large corpora of natural text available on the internet.
More recently, image-classiﬁcation models have also beneﬁted from pretraining on large datasets (Kolesnikov et al., 2020), both supervised and unsupervised; a key example is the Vision Transformer architecture of Dosovitskiy et al.
(2020), which—despite not using the inductive bias provided by convolutions—attains downstream performance comparable to convolutional networks after pretraining on ImageNet-21k (Deng et al., 2009) or JFT-300M (Sun et al., 2017).
On many current NLP and computer vision benchmarks, large pretrained models achieve top performance.
All of the experiments in our work follow the pretrain-then-ﬁnetune method; recently, another paradigm, side-tuning, was proposed by Zhang et al.
(2020), which avoids forgetting altogether by leaving the pretrained model ﬁxed and employing auxiliary networks for downstream tasks.
E XPERIMENTAL SETUP
Tasks: Most of the experiments in this paper are conducted in the standard, task split setting (Kirkpatrick et al., 2017; Zenke et al., 2017).
We consider split CIFAR-10 task sequences – the ten class dataset is split and sequentially trained on two tasks of 5 classes each.
We also consider sequential 10 and 50 class splits of CIFAR-100.
Beyond split CIFAR, we study the following datasets:
Oxford-IIIT pet, Oxford Flowers 102, Street View House Numbers (SVHN), Caltech-UCSD Birds 200 (CUB-200), Cars196, Domainnet/Clipart.
We also consider the more recent input-distribution-shift CIFAR-100 setup (Ramasesh et al., 2021).
In this case each task has the same target types made up a ﬁxed subset of the 20 CIFAR-100 superclasses (e.g. aquatic mammals, ﬁsh, ...).
The sequence of tasks consists of modifying which subclasses are sampled from.
For example, Task A may have aquatic mammals drawn from beaver and dolphin, while Task B may have otter and whale.
This setup allows one to probe forgetting without changing the model head, in a setting where task identity is not known to the model.
For our language model experiments, we consider next token prediction generative tasks and use the
IMDb Reviews (Maas et al., 2011) and english Wikipedia (Foundation) datasets.
Vision models:
In most of our experiments we use the Vision Transformer (ViT) and ResNet architectures.
For the Vision Transformers, we use the ViT-S and ViT-B conﬁgurations from Dosovitskiy et al.
(2020) and a smaller conﬁguration we call ViT-xS – ranging from 5.7-86.7 million parameters.
For ResNets, we use ﬁve models: ResNet26, ResNet50, ResNet101, ResNet152, and ResNet200 – ranging from 14.0-62.6 million parameters; as in Kolesnikov et al.
(2020), these models use group normalization (Wu & He, 2018) and weight standarization (Qiao et al., 2019) in place of batch normalization.
Details can be found in Tables 1 and 2 in Appendix A.1.2.
Pretraining: Unless otherwise speciﬁed, we pretrain our image models in a supervised manner on the ImageNet21k dataset, which contains approximately 14 million images in about 26000 distinct categories using the Adam optimizer (Kingma & Ba, 2017).
Further details are in the Appendix.
Finetuning: When ﬁne-tuning on multiple tasks in sequence, we use the SGD optimizer with momentum β = 0 .
The ﬁrst task training employs a cosine decay schedule with a linear warmup; later tasks in the sequence are trained with ﬁxed learning rates.
For the split-CIFAR-10 task sequences, we use a multi-head setup in which each head is initialized to zeros.
R ESULTS 3. F ORGETTING IMPROVES WITH MODEL SCALE
Our main ﬁnding is that for both pretrained Vision Transformers and pretrained ResNets, catastrophic forgetting is mitigated to a large extent by scale: that is, larger models suffer less from forgetting.
This is shown in Figure 1 for the two-task split CIFAR-10 sequence: Task A { airplane, automobile, bird, cat, deer } →
Task B { dog, frog, horse, ship, truck } .
In order to make a fair comparison, we plot the test accuracies of the model on both tasks at different steps of Task B ﬁne-tuning for a variety of learning rates.
This allows us to see, for each model, the maximum performance achievable on the new task for a given performance on the initial task.
We call this the forgetting frontier.
Figure 1 shows that the forgetting frontier for both ViT and ResNet models improves signiﬁcantly with scale.
While a 5.7M-parameter ViT-xS model loses about 6.5% in Task A accuracy over the course of learning to perform Task B, an 86.7M-parameter ViT-B model loses less than half a percent.
To put this in context, in a previous studies of split-CIFAR-10 task sequences, a ResNet model lost close to 40% accuracy with no mitigation methods, 5% when using a replay buffer, and 20% when using elastic weight consolidation (see e.g. Figure 1 of Ramasesh et al.
It is also important to note that this effect of scale and pretraining on improving forgetting frontiers does not seem to be architecture-dependent.
Both pretrained vision transformers—based on self- [Table] attention—and ResNets—based on convolutions—exhibit improvements in their forgetting frontiers as they grow larger.
We see this improvement in forgetting performance beyond the simple two-task split-CIFARscenario.
Across a 10-task sequence of 10-class split-CIFAR-100 tasks (Figure 3 left), sequential 50-class splits of CIFAR-100 (Figure 3 right), the CIFAR-100 single-head input-distribution-shift task (Figure 4), and two-task sequences beyond the CIFAR dataset (Figure 2), large-scale pretrained models exhibit resistance to forgetting, and continual learning ability appears to improve systematically with scale.
For detailed results, scaling plots, and forgetting frontiers in the 10-task scenario, see Section B.3 in the Supplement.
3. D EPENDENCE ON FINETUNING DATASET
It is interesting to study how the improved resistance to forgetting with scale depends on the choice of ﬁnetuning dataset.
In particular, it is worth noting that, though quite different in format, CIFAR10/100 are semantically quite similar to subsets of ImageNet21k, our pretraining set.
To empirically investigate the dependence on ﬁnetuning dataset we have run additional experiments similar to those presented in Figure 1 for the Oxford Pet and Flowers, Caltech Birds, Stanford Cars, street view house numbers (SVHN) and DomainNet Clipart datasets.
Results are shown in Figure 5 and in Figure 2.
We note in particular that SVHN classes are numerals and so distinct from ImageNet21k classes, while the Clipart dataset consists of synthetic, rather than natural images.
See the Supplement for further details.
3. F ORGETTING IN PRETRAINED MODELS VS . MODELS TRAINED FROM SCRATCH
In the previous section, we saw that the amount of catastrophic forgetting in models pretrained on large datasets decreased with increasing model size, to the point that the largest models (ViT-B, with 86 .
7 M parameters and ResNet200 with 62 .
6 M parameters) suffer very little performance loss from second-task training.
In this section, we investigate whether this improvement in forgetting is simply a consequence of the improvement in performance on Task A.
That is, do pretraining and scale have beneﬁts with respect to forgetting beyond simply improving the ﬁne-tuned performance on downstream tasks?
To check this, we compare the performance and amount of forgetting in pretrained ResNet models to models trained from scratch.
We handicap the pretrained ResNets so that their maximum accuracy on on
Task A matches that of the corresponding trained-from-scratch ResNet; this is done simply by reducing the number of steps for which we ﬁne-tune.
Figure 6 shows the results of this experiment, with additional results and details in Appendix C.1.
While the trained-from-scratch ResNets achieve the same performance as their pretrained counterparts on Task A, their Task A performance degrades signiﬁcantly more than the pretrained ResNets when trained on Task B; pretrained ResNets forget less, and this improves with model scale.
This result suggests that pretraining and model scale confer beneﬁts upon models which go beyond simply improving performance.
From Figure 6 it is also clear (see also Figure 19 in the appendix, which plots the same data in a different way) that pretraining is required to see the beneﬁts of scale.
Simply scaling up the model size in trained-from-scratch ResNets does not yield clear improvements in their forgetting performance
We focused in this section exclusively on ResNets (rather than Vision Transformers) because obtaining decent performance on image classiﬁcation tasks with Vision Transformers seems to require pretraining.
In our experiments, for example, a trained-from-scratch ViT-S model was unable to achieve better than 75 % test accuracy on ﬁve-class CIFAR-10, while the same model, pretrained, achieved better than 98 % accuracy.
3. D EPENDENCE ON PRETRAINING TIME , PRETRAINING DATASET SIZE , AND FINETUNING TIME
Having seen in the previous sections that pretraining on large datasets improves models’ resilience to catastrophic forgetting, in this section we explore the dependence of this phenomenon on the duration of pretraining, the size of the pretraining dataset and the number of steps for which we ﬁnetune on Task A.
First, we study the inﬂuence of pretraining time on downstream forgetting.
This is motivated by previous studies (e g. Hernandez et al.
showing that downstream performance does not always monotonically improve with pretraining time and performance.
By contrast, in the case of forgetting we ﬁnd (see Figure 7 for ViT-S results) that as the pretraining process continues, both downstream performance and the forgetting frontier improve.
This was the case for all models (ViTs and ResNets) and downstream task sequences (CIFAR-10, CIFAR-100, etc.)
we studied.
We found no evidence of forgetting frontiers getting worse with increased pretraining.
Along similar lines, we show the inﬂuence of pretraining data scale on downstream forgetting in
Figure 8, pretraining a model on varying fractions of ImageNet21k.
While the full dataset contains over 14 million images, we ﬁnd that training on even one-sixteenth of the dataset (roughly the size of the standard ImageNet ILSVRC 2012 dataset) yields a model which only loses about 3% performance in forgetting on split-CIFAR-10.
This suggests that one does not need to pretrain models on unreasonably large datasets in order to reap beneﬁts for continual learning.
Finally, we study the forgetting frontier of pretrained models as we vary the amount of ﬁrst-task ﬁne-tuning (number of training steps).
This is partly inspired by a recent study (Andreassen et al., 2021) of robustness in pretrained models, which found that during ﬁne-tuning, pretrained models evolved some robustness to distribution shift which vanished at the end of ﬁne-tuning.
Figureshows forgetting frontiers for pretrained ViT-B and ResNet101 models, ﬁne-tuned for a number of steps between 100 and 5000.
The results in the ﬁgure suggest that in our setting, increasing the ﬁne-tuning steps to maximize performance on Task A does not appear to cause increased forgetting.
On the contrary, even training for about ten times more steps than it takes to achieve saturating performance does not impact forgetting.
3. R EPRESENTATION OVERLAP AND FORGETTING
In this section we take steps toward understanding why large pretrained models seem to be resistant to forgetting.
Following the work of Ramasesh et al.
(2021), we measure the similarity between a models representation of Task A and Task B data.
In detail, if we denote the P model features (penultimate layer activations) on an input, x , by f ( x ) ∈ R P , Ramasesh et al. (2021) propose measuring the similarity of f A :=
{ f ( x ) : x ∈ Task A } ∈
R | Task A |× P and f B :=
{ f ( x ) : x ∈ Task B } ∈ R | Task B |× P using the trace overlap ,
S AB = Tr(Θ AB Θ TAB
[Equation]
Here Θ AB = f
A f TB is the matrix of inner-products between Task A and Task B features.
Ramasesh et al.
(2021) show empirically that forgetting is maximal for intermediate values of S AB , and derive this property analytically in a simpliﬁed model.
In particular Ramasesh et al.
(2021) propose that a minimal trace overlap (corresponding to orthogonality of representations of different tasks) will lead to minimal forgetting.
Inspired by this, we measure representation overlaps in models ﬁnetuned on split CIFAR-10, taking representations at the end of Task A training.
We consider the 10x10 matrix, S , formed by taking the indices A and B to run over the different classes in CIFAR-10.
The numerator in the above expression can be thought of as an empirical (batch) estimate of the inner product between feature vectors for random draws from classes A and
B ; trace overlaps close to zero indicate that representations are nearly orthogonal.
Using the above deﬁnition, we return to the original split-CIFAR-10 setting of §3.1 and §3.3.
For each of the models used in those experiments (see Tables 1 and 2), we compute the representation overlap matrix.
We focus on two comparisons: (i) how does the representation overlap matrix differ between pretrained and trained-from-scratch ResNets?
and (ii) how does the orthogonality of representations vary as we scale the model size, for both ResNets and Vision Transformers?
Pretrained vs. trained-from-scratch representations : We show representative trace overlap matrices for both pretrained and trained-from-scratch ResNet101 models in Figure 9(left and center)
(see Figure 27 for additional models).
Each element of this matrix corresponds to the representation overlap between two batches, each drawn entirely from a single class.
The representation overlap between disjoint classes (off-diagonal class overlap matrix elements) is much smaller for the pretrained ResNets than those trained from scratch.
Averaging cross-class overlaps over class pairs from different tasks quantiﬁes this difference: for trained-from-scratch ResNet101, the average overlap is 0 .
907 , while for the pretrained model this value is only 0 . 466 .
For all sizes of ResNet, the pretrained models exhibit representations which have much smaller overlap than their trained-fromscratch counterparts.
Representation overlap across scales —Figure 9(right) shows the average cross-task overlap for both ResNets and Vision Transformers plotted against model size (number of parameters).
For pretrained Vision Transformers and ResNets, this overlap decreases with model size, while for trainedfrom-scratch ResNets, there is not a clear decreasing trend.
Interestingly, Vision Transformers appear to form more orthogonal representations than ResNets do, a fact which can also be seen in the overlap matrices presented in supplementary Figure 26, and perhaps explains their improved forgetting performance.
These observations suggest that pretrained models store representations of different classes with much less overlap than trained-from-scratch models, and that this orthogonality increases with scale.
As this mirrors the the observed trend in forgetting in these models, it is suggestive that the increased orthogonality of representations is a partial explanation for the forgetting resilience seen in large pretrained models.
D ISCUSSION
The central observation we have reported in this paper is that scaling up models pretrained on large datasets is effective at mitigating catastrophic forgetting in the settings studied.
Our experiments show that pretraining and scale can confer beneﬁts beyond simply the achievable downstream loss or accuracy values.
An important direction for future work is understanding what underlies the systematic improvement of continual learning with model scale.
The rate at which performance improves with model size in Figure 1, with power-law exponent close to − 1 , is reminiscent of the variance limited behavior discussed in Bahri et al.
Namely, when the number of model parameters N is much larger than the number of training examples (as is the case for split-CIFAR ﬁnetuning), performance is predicted to scale as N − 1 .
This scaling behavior is notably not seen in Figure 11 for language modeling— this is reminiscent of the disparate scaling behaviors observed in Sharma & Kaplan (2020) between language-modeling and image-classiﬁcation tasks.
Ideally, developing an understanding of this behavior would allow for predicting the rate of improvement to forgetting with model size. A CKNOWLEDGMENTS
The authors would like to thank Anselm Levskaya, Bill Mark, Anders Andreassen and Maithra
Raghu for their conversations during the completion of this work
A E XPERIMENTAL DETAILS A. V ISION A.1. S UPERVISED PRETRAINING
Here we detail the settings used to in supervised pretraining, resulting in the models used in the experiments of sections 3.1, 3.3, 3. and B.2.
We pretrain seven models: (ViT-B, ViT-S, ViT-xS,
R26x1, R50x1, R101x1, R152x1, R200x1) on ImageNet21k for 90 epochs.
Pretraining is done using the Adam optimizer ( β 1 = 0 .
9 and β 2 = 0 .
999 ); for all models, we use a batch size of 4096.
Our learning-rate schedule includes a warmup of 10k steps to a maximum learning rate η = 10 − 3 , followed by a linear decay to 10 − 5 .
For the Vision Transformer models, we use a weight decay penalty of 0 .
03 and a dropout rate of 0 .
1 ; for ResNet models, we do not use weight decay or dropout.
These settings are largely identical to those found in Dosovitskiy et al.
A note on ImageNet21k: We use the version of ImageNet21k available in TensorFlow Datasets 4.0. and apply minimal preprocessing, simply converting the labels to one-hot vectors and cropping the image to be 224x224 pixels in size.
During training, this crop is an Inception-style random crop, while during evaluation it is a deterministic central crop; during training we additionally augment the data by performing random horizontal ﬂips and apply label smoothing, with values 0 .
9999 and 0 . 0001 .
Importantly, ImageNet21k is class-imbalanced and heavy-tailed, with the uneven distribution of label occurrences shown in Figure 10.
We do not balance the dataset.
A.1. V ISION T RANSFORMER MODEL DETAILS
For full description of the Vision Transformer architecture, see the original paper by Dosovitskiy et al.
Abbreviated details are given here.
We process 2D images of height H , width W , and
C channels, x ∈ R H × W × C by ﬁrst reshaping them into a 1D sequence of ﬂattened patches–in this work, we exclusively use patch size 16.
Each patch is mapped (via a learned linear transformation) to an embedding vector of dimension D (for our models, D xS = 256 , D S = 516 , and D B = 768 , as in Table 1).
As both our pretraining and ﬁnetuning is supervised, we prepend a (learnable) ‘[class]‘ token embedding to this sequence of patches, add a learnable 1D position embedding, and pass the sequence through the Transformer encoder.
The Transformer encoder alternates between multiheaded self-attention layers and MLP (plus residual connection) layers, with layernorm applied before both the attention and MLP layers.
For classiﬁcation, we use a classiﬁcation head which acts on only the transformed ‘[class]‘ token representation.
During pretraining, this classiﬁcation head is an single-hidden-layer MLP, as in the original Vision Transformer work; during ﬁnetuning, the classiﬁcation head is a linear layer (in the split-CIFAR-10 setting, we use a multi-head setup with a different classiﬁcation layer for each task in the sequence).
We initialize all classiﬁcation layers with zeros.
[Table]
A.1. CIFAR-100 D ISTRIBUTION S HIFT T ASK
Here we provide more information about the CIFAR-100 distribution-shift task, which was introduced in Ramasesh et al.
and forms the basis of our experiments in section A.1.3 of the main text.
The idea behind this task is to mimic a way in which catastrophic forgetting might occur in practice [Table] in image-classiﬁcation settings where the class labels remain constant but the distribution of the images changes.
A key features of this type of setting is that the identity of the task is not known to the model at inference time [Table] so task-speciﬁc components (such as multiple readout layers) are not allowed.
For example, in a medical setting, one might seek to classify images as displaying signs of a disease or not, and the system might be trained sequentially on images taken at various hospitals.
While the CIFAR-100 dataset features images divided into 100 distinct classes, these classes are grouped into 20 superclasses .
The CIFAR-100 distribution shift task sequence involves classifying an image by its superclass; individual tasks in the sequence draw images from a single subclass for each superclass.
As a concrete example, in the experiments in the main text we use the ﬁve superclasses aquatic mammals , fruits and vegetables , household electrical devices , trees , and vehicles-1 1 .
For the ﬁrst task in the sequence, Task A, we uses the classes dolphin (for aquatic mammals), apple (for fruits and vegetables), lamp (for household electrical devices), maple tree (for trees), and bicycle (for vehicles-1).
For the second task, Task B, we use the classes whale (for aquatic mammals), orange (for fruits and vegetables), television (for household electrical devices), willow (for trees), and motorcycle (for vehicles-2).
A. F INETUNING EXPERIMENTAL DETAILS
In this section we provide details on the ﬁnetuning procedures for the experiments in the main text.
All ﬁnetuning was done using stochastic gradient descent with momentum ( β = 0 . 9 ), clipping gradients at unity; batch size is ﬁxed at 512.
In the ﬁrst task, we used a schedule with a 5-step linear ramp to a maximum learning rate followed by a cosine decay to zero over the full ﬁrst-task training.
In the second task, we used constant learning rates.
Section 3.1, Figure 1: Finetuning both the Vision Transformers and ResNets was done forsteps.
For Vision Transformers, the ﬁrst-task training used a warm-up from η = 0 .
01 to η = 0 .
For each model we perform four different runs, with second-task (constant) learning rates 0 .
007 , and 0 .
For ResNets, the task-A learning rate warm-up was from η = 0 .
002 to η = 0 .
The (constant) second-task learning rates we used were 0 .
001 , 0 . 0003 , and 0 . 0001 .
Section 3.1, Figure 2:
In the 10-task split-CIFAR100 sequence (left), each task was trained for 200 steps; the initial task used a learning-rate schedule with a 5-step warmup from η = 0 .
01 to η = 0 .
05 followed by a cosine decay to 0 over the 200 steps, while all subsequent steps used constant η = 0 .
In the 2-task 50-class split-CIFAR100 sequence (right), the settings are identical to those of the Vision Transformer training in Figure 1.
Section 3.1, Figure 3: For ResNets, we used a ﬁrst-task warmup from η = 0 .
001 to η = 0 .
005 , and trained each task for 300 steps.
Constant second-task learning rates were 0 .
001 , 0 . 0003 , and 0 . 0001 .
For Vision Transformers, we trained each task for 200 steps, using a ﬁrst-task warmup from η = 0 .
01 to η = 0 .
Constant second-task learning rates were 0 .
01 , 0 . 003 , and 0 .
Section 3.2, Figure 4: The pretrained models in this ﬁgure were the same as those in Figure 1.
For the trained-from-scratch models, we used a linear warm-up from η = 0 .
006 to η = 0 .
03 , and trained each task for 15,000 steps.
Constant second-task learning rates were 0 .
001 , 0 . 0003 , and 0 . 0001 .
Note that training here is much longer than for the pretrained models.
Section 3.3, Figure 5: Vision Transformer training settings are identical to those of Figure 1.
Section 3.3, Figure 6:
In the left panel, Vision Transformer training settings are identical to those of
Figure 1.
In the right two panels, settings are also identical to those of Figure 1, with the exception of the number of ﬁnetuning steps, as indicated in the ﬁgure legend.
Section 3.4, Figure 7: We ﬁnetune for 300 steps on each task, with a maximum learning rate of η = 0 .
(we omit the linear warm-up for these models, but keep the cosine decay).
Constant second-task learning rates are η = 0 . 0002 , η = 0 .
0001 , η = 0 .
00004 , and η = 0 . 00002 .
A. L ANGUAGE
Models are decoder-only transformers which were pretrained using Adafactor and learning rate 1 . 0 , context length 1024 and batch size 256 .
We ﬁnetuned with batch size of 8 and set a constant learning rate equal to the learning rate at the end of pretraining rescaled by the relative batch size, 8 / 256 .
B A DDITIONAL EXPERIMENTS B. F ORGETTING IN LANGUAGE MODELS
Transfer learning from unsupervised pretraining is commonplace in Natural Language Processing.
It has become standard to pretrain models on data crawled from the internet and then ﬁnetune on smaller, downstream tasks (Devlin et al., 2019; Radford et al., 2019; Raffel et al., 2020; Brown et al., 2020).
This bears similarities with both the unsupervised pretraining and the experiments in Figurewhere a single head was used to train on different distributions (subclasses).
Indeed, as mentioned in the main text, part of the conceptual motivation for this work came from the observation that large language models with diverse pretraining perform well on tasks with no ﬁnetuning – ostensibly retaining information for many gradient steps.
Here, we include some preliminary empirical results probing catastrophic forgetting directly in large language models.
We consider decoder only transformer models pretrained on data scraped from the web, including
C4 (Raffel et al., 2020).
These models range in size from 57 million to 8.6 billion non-embedding parameters and were originally trained as part of the BIG-bench collaboration (2021).
In Figure 11, we consider sequentially ﬁnetuning the models on Task A: IMDb Reviews for 1.5k steps followed by Task B: english Wikipedia for 10k steps.
The longer ﬁnetuning time for Task B is intended to highlight forgetting, as these models are relatively robust.
We ﬁnd that, as in the case of pretrained vision models, the best joint Task A, Task B performance improves with scale.
B. F ORGETTING IN VISION MODELS WITH UNSUPERVISED PRETRAINING
In the main text, the models we studied were pretrained in a supervised manner on ImageNet21k.
However, part of the conceptual motivation for this work came from the properties of language models trained in an unsupervised fashion.
Furthermore, recently, unsupervised pretraining methods have yielded vision models which achieve good downstream performance; among these are
SimCLR (Chen et al., 2020), MoCo (He et al., 2020), DINO (Caron et al., 2021), and BYOL (Grill et al., 2020).
Given the increased prevalence of such unsupervised pretraining methods, we investigate in this section whether unsupervised pretraining, combined with scale, provides resistance to forgetting similarly to supervised pretraining.
We take publicly-available ResNet models which were pretrained using SimCLR on the ImageNet ILSVRC-2012 dataset (Russakovsky et al., 2015), and ﬁnetune them on the two-task split CIFAR-10 sequence.
The resulting forgetting frontiers, shown in Figure 12, show that as in supervised pretraining, SimCLR pretraining yields models which are resilient to forgetting, especially as scale is increased.
We leave a thorough exploration of the continual-learning properties of other unsupervised pretraining methods for future work.
B. F ORGETTING IN TASK SEQUENCES WITH MORE THAN TWO SUBTASKS
In this section, we extend our results on vision past the two-task setting which was the primary focus of the main text.
Speciﬁcally, we use a 10-class, 10-subtask split of the CIFAR100 dataset to study how forgetting in this scenario varies as a function of model and dataset scale.
The speciﬁc split we use is shown in Table 3.
As in the main text, we study the performance of Vision Transformer and
ResNet models.
Task A
Task B
Task C
Task D
Task E
Apple
Bowl
Chair
Dolphin
Lamp
Aquarium Fish
Boy
Chimpanzee
Elephant
Lawnmower
Baby
Bridge
Clock
Flatﬁsh
Leopard
Bear
Bus
Cloud
Forest
Lion
Beaver
Butterﬂy
Cockroach
Fox
Lizard
Bed
Camel
Couch
Girl
Lobster
Bee
Can
Crab
Hamster
Man
Beetle
Castle
Crocodile
House
Maple tree
Bicycle
Caterpillar
Cup
Kangaroo
Motorcycle
Bottle
Cattle
Dinosaur
Keyboard
Mountain
Task F
Task G
Task H
Task I
Task J
Mouse
Plain
Rose
Squirrel
Train
Mushroom
Plate
Sea
Streetcar
Trout
Oak tree
Poppy
Seal
Sunﬂower
Tulip
Orange
Porcupine
Shark
Sweet pepper
Turtle
Orchid
Possum
Shrew
Table
Wardrobe
Otter
Rabbit
Skunk
Tank
Whale
Palm tree
Raccoon
Skyscraper
Telephone
Willow tree
Pear
Ray
Snail
Television
Wolf
Pickup truck
Road
Snake
Tiger
Woman
Pine tree
Rocket
Spider
Tractor
Worm
We ﬁnd that:
• As in the two-task setting, both the forgetting frontiers and the best average accuracy improve with model scale as a power-law in the number of model parameters.
• When we scale the size of the pre-training dataset, we also observe an improvement with scale, though with some saturation around the full ImageNet21k dataset..
Model scaling : Model-scaling plots showing the best average accuracy as a function of model parameters are show in Figure 13.
As in the two-task split CIFAR10 sequence, these scaling plots are one way to show that the continual-learning performance of both architectures improves with scale.
The power-law exponents are different in this 10-task setting than they are for the two-task setting, but interestingly all are close to unity.
Another way to see the improvement in continual-learning ability with model scale is to plot forgetting frontiers, which we do in Figures 15 and 16.
In this setting with more than two tasks, plotting a single forgetting frontier is not possible, but we plot two-task frontiers for all pairs of subtasks (with 10 subtasks, there are 10 × 9 / 2 = 45 pairs).
These frontiers are constructed in exactly the same way as the two-task frontiers, showing the test accuracies for each subtask throughout the entirety of the 10-task training.
As the ﬁgures show, forgetting frontiers of larger models dominate the forgetting frontiers of smaller models of the same architecture.
This complements the previous scaling-plot as a demonstration of improving continual-learning with scale, though the behavior here is not as clean as it was in the two-task setting.
Data scaling: A data-scaling plot showing the best average accuracy for a ViT-S model as a function of pretraining dataset size is shown in Figure 14.
Forgetting frontiers are shown in Figure 17.
B. F ORGETTING IN DOWNSTREAM TASK SEQUENCES ON DATASETS OTHER THAN CIFAR10 AND CIFAR
In this section, we study the continual-learning performance of ImageNet21k-models on downstream task sequences beyond CIFAR10 and CIFAR100.
Speciﬁcally, we use the following datasets:
• The Oxford-IIIT pet dataset (Parkhi et al., 2012), a 37-category dataset of images of pets, labeled by breed, with roughly 200 images per class.
In our experiments, we create a two-task split of this dataset, with the ﬁrst 19 breeds (alphabetically by English label) comprising task A and the second 18 breeds comprising task B. • The Oxford Flowers 102 dataset (Nilsback & Zisserman, 2008), a 102-category dataset consisting of ﬂowers common in the the UK, labeled by species.
In our experiments, we create a two-task split of this dataset, with the ﬁrst 51 ﬂower species (alphabetically by English species name) in task A and the second 51 ﬂower species in task B. •
The Street View House Numbers (SVHN) dataset (Netzer et al., 2011), a 10-category dataset where images consist of single-digit numbers from house numbers.
We use the the two-task split in which task A consists of digits 0,1,2,3,4 and task B consists of digits 5,6,7,8,9.
• The Caltech-UCSD Birds 200 (CUB-200) dataset (Welinder et al., 2010), a 200-category dataset consisting of photos of birds, labeled by species.
As above, we create a two-task split by alphabetically sorting English species names and taking task A to consist of the ﬁrst 100 categories, and task B to consist of the second 100 categories.
• The Cars196 dataset, a 196-category dataset consisting of natural images of cars (Krause et al., 2013), with class labels speciﬁc models (including years) of cars, e.g. 2012 Tesla Model S or 2012 BMW M3 coupe.
We split this into two tasks by using the standard label numbers in TensorFlow datasets, with labels 0-97 in task A and 98-195 in
task B. •
The Domainnet/Clipart dataset, a subset of Domainnet (Peng et al., 2019), consists of clipart drawings (unlike all of the above datasets, which consist of natural images) ofdifferent categories including tennis racquets, diamonds, and umbrellas.
We create a twotask split alphabetically, with 173 categories in task A and 172 categories in task B.
We use the same basic setup as in the split-CIFAR10 task sequence, in which the ﬁrst task is trained with a 5-step warmup followed by a cosine decay, while the second task is trained with a constant learning rate.
The speciﬁc learning rates we use for each of these datasets are given in table 4.
Forgetting frontiers and best-average-error scaling plots are shown in Figure 2.
From these plots, we can see that the improvement in forgetting with scale is not unique to the CIFAR10 and CIFAR100 task sequences we used in the main text, but occurs for all of these datasets as well.
Scaling plots show that this improvement in average-task performance is a power-law function of the model parameters, but with exponents which vary across datasets.
The broader implication of these plots is that the improvement in forgetting performance with scale is not limited to downstream datasets which are essentially in-distribution of the pretraining dataset (e.g., CIFAR10 and CIFAR100 are very similar to images found in ImageNet21k).
As a proxy to quantify how ‘in-distribution’ the downstream datasets are, in Table 5, we show performances for both a ResNet and ViT model when we train just a linear classiﬁer on top of weights frozen after the ImageNet21k pretraining.
We take the closeness of this head-only performance to the full ﬁnetuned performance as a proxy for the dataset being in-distribution to ImageNet21k; using this proxy, the Oxford-IIIT Pet dataset, Oxford Flowers dataset, CIFAR10, CIFAR100, and CUB Birds 200 dataset are roughly in-distribution, while Domainnet/Clipart, Cars196, and SVHN are not.
But, as we emphasized above, even for these ‘out-of-distribution‘ datasets, forgetting is largely mitigated by scale in the pretrained models.
For completeness, we show head-only and full-ﬁnetuning performances for all ResNet and ViT models on the ﬁrst task (task A) of our split-CIFAR10 sequence in Table 6.
[Table]
Oxford-IIIT pet
[Table]
rate
[Table]
Finetuning steps
[Table]
Oxford Flowers
Task
[Table]
learning rate
[Table]
rates
[Table]
Finetuning steps
[Table]
SVHN
[Table]
rate
[Table]
Caltech-UCSD Birds
[Table]
rates
[Table]
Finetuning steps
[Table]
Domainnet/Clipart
[Table]
Finetuning steps
[Table]
Cars
[Table]
rates
[Table]
[Table]
[Table]
B. F ORGETTING ON THE PRE - TRAINING TASK DURING FINETUNING
Given the strong performance of large-scale pretrained models on continual learning tasks we have seen in the main text [Table] it is natural to wonder to what extent performance drops on the original pretraining tasks throughout ﬁnetuning.
One intuition might suggest that because the pretraining dataset is so large, performance drops on the pretraining task will be minimal throughout ﬁnetuning.
However, in our experiments [Table] we found that this was not the case.
In Table 7, we show the performance drops on the pretraining tasks for all models, in two downstream tasks: CIFAR10 and SVHN.
In most cases, we ﬁnd signﬁcant performance drops, though these are much more pronounced when ﬁnetuning on the SVHN dataset than when ﬁnetuning on CIFAR10.
[Table]
C S UPPORTING RESULTS
In the following subsections, we present results which are directly supporting experiments or ﬁgures in the main text, but not new experiments.
C. C OMPARING PRETRAINED AND TRAINED - FROM - SCRATCH R ES N ETS
In the main text, Figure 6, we showed a comparison between the forgetting frontiers (on the split
CIFAR10 task sequence) of trained-from-scratch ResNet models and pretrained ResNet models, where we handicapped the pretrained ResNet models (by training them for fewer steps) so that they matched task-A performance of the trained-from-scratch ResNets.
This was done to show that even when task-A performance is the same, pretrained ResNets forget far less than their trained-fromscratch counterparts.
For space, in the main text we only showed three ResNet models: ResNet26,
ResNet101, and ResNet200.
Here, we show the forgetting frontiers for all ResNet models (Figure 18), and we also show (Figure 19) the data plotted slightly differently—with all model sizes plotted on the same graph—to make it apparent that while the forgetting frontiers of the pretrained
ResNet models clearly improve with model size, the forgetting frontiers of trained-from-scratch
ResNet models do not.
Figure 19 also includes a scaling plot of the best average error vs. model parameters, supporting the conclusion that in trained-from-scratch ResNets, scaling does not appear to beneﬁt the continual learning performance.
C. L EARNING CURVES FOR SPLIT -CIFAR10 VISION EXPERIMENTS
In Figure 1 of the main text we plot forgetting frontiers of pretrained Vision Transformers and
ResNets on the split-CIFAR10 task sequence.
Corresponding to these frontiers are the learning curves (accuracies vs. step) shown in Figures 21 and 20.
[Figure] [Figure]
C. L EARNING CURVES FOR THE LANGUAGE EXPERIMENT
For completeness, Figure 22 displays the token accuracy on task A as a function of the number of steps, corresponding to the frontiers shown in Figure 11 of the main text.
C. F ORGETTING FRONTIERS COLORED BY LEARNING RATE
In the forgetting frontiers of Figure 1 of the main text, the frontiers are colored by model size; each model, however, was trained with several different learning rates during the second task.
In
Figures 23 and 24, we plot the same frontiers, coloring them by learning rate on the left, and by proper time (learning rate multiplied by step number) on the right.
We also plot a scaling curve in Figure 25 of the average accuracy at a ﬁxed learning rate and step ( η = 0 .
0001 and t = 200 )
, to show that the power-law scaling holds even if one looks at a ﬁxed hyperparameter choice.
[Figure] [Figure]
[Figure]
C. R EPRESENTATION OVERLAP MATRICES FOR ALL R ES N ET AND V I T MODELS
In the main text, we described the computation of the classwise representation overlap matrices for our models at the end of Task A training, showing an example of such a matrix for both a pretrained and a trained-from-scratch ResNet101 model.
In Figure 27, we show representation overlap matrices for all of the ResNet models, which bears out our statement in the main text that pretrained ResNet representations were much closer to orthogonal than their trained-from-scratch counterparts.
In
Figure 26, we show representation overlap matrices for the (pretrained) Vision Transformer models.
As with the pretrained ResNet models, the cross-class representation overlaps become closer to orthogonal as we increase the model size.
In these ﬁgures, we also plot the representation overlaps for pre-trained models before being trained on task A (which was not plotted in the main text).
From these plots one can see that while pretraining on ImageNet21k introduces some level of orthogonality in the model representations of
CIFAR10 images, this orthogonality becomes much more pronounced after training the model on task A of the CIFAR10 split.