E FFECT OF MODEL AND PRETRAINING SCALE ON CATASTROPHIC FORGETTING IN NEURAL NETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Catastrophic forgetting presents a challenge in developing deep learning models capable of continual learning, i.e. learning tasks sequentially.
Recently, both computer vision and natural-language processing have witnessed great progress through the use of large-scale pretrained models.
In this work, we present an empirical study of catastrophic forgetting in this pretraining paradigm.
Our experiments indicate that large, pretrained ResNets and Transformers are signiﬁcantly more resistant to forgetting than randomly-initialized, trained-from-scratch models; this robustness systematically improves with scale of both model and pretraining dataset size.
We take initial steps towards characterizing what aspect of model representations allows them to perform continual learning so well, ﬁnding that in the pretrained models, distinct class representations grow more orthogonal with scale.
Our results suggest that, when possible, scale and a diverse pretraining dataset can be useful ingredients in mitigating catastrophic forgetting.
Continual learning is a current challenge in designing machine learning systems: when trained on multiple tasks in sequence, models tend to suffer performance losses on earlier tasks—this is known as catastrophic forgetting (McCloskey & Cohen, 1989).
Catastrophic forgetting occurs in many settings, from supervised learning for classiﬁcation (Zenke et al., 2017) to reinforcement learning for games (Kirkpatrick et al., 2017) and downstream applications such as translation (Bapna & Firat, 2019) or medical diagnoses (Gupta et al., 2021).
While many methods have been proposed to help mitigate forgetting, none of these have fully solved the problem; see Kemker et al.
(2017) for a review.
Increasing model and dataset size have been key to the success of deep learning (Kolesnikov et al., 2020; BIG-bench collaboration, 2021;
Radford et al., 2019; Zhai et al., 2021).
A recent trend has been the use of increasingly large models pretrained on diverse datasets, which are subsequently ﬁne-tuned or evaluated without ﬁne-tuning on downstream tasks of interest; see Bommasani et al.
(2021) and references therein.
A particularly striking realization of this has been large pretrained language models, which show consistent improvement across scales (Kaplan et al., 2020) and can perform well on diverse tasks with no ﬁnetuning (Brown et al., 2020).
From a continual learning perspective, this behavior is especially dramatic as, without ﬁnetuning, any training signal for these tasks appears mixed into a drastically larger pretraining dataset, and thus more than likely appeared many gradient steps before evaluation.
This observation in large part inspired our work.
Here, we empirically study continual learning in large pretrained models.
Our results suggest that pretrained models are signiﬁcantly more resistant to catastrophic forgetting than their randomlyinitialized counterparts, and that this resistance improves with the scale of both the model and pretraining data.
Most of our experiments concern vision—speciﬁcally image classiﬁcation; but we also conduct a preliminary investigation of language models.
Our key observations are as follows:
• Resistance to catastrophic forgetting improves with model and pretraining dataset scale (Figure 1).
• Large, pretrained image-classiﬁcation models are signiﬁcantly more resistant to forgetting than models trained from random initialization (Figure 4).
• Both supervised and unsupervised, SimCLR
(Chen et al., 2020), pretraining improve forgetting (Figure 7).
• We perform initial experiments aimed at exploring why scale and pretraining reduce catastrophic forgetting, showing that distinct class representations in pretrained models are more orthogonal than in trained-from-scratch models (Figure 8).
• We observe similar phenomena in language models (Figure 9).
R ELATED WORK
Catastrophic forgetting Catastrophic forgetting was identiﬁed as a problem for neural networks as early as the late 1980s, as it occurred in the single-hidden-layer networks that were studied at the time (see Delange et al.
(2021) for a review).
Within the catastrophic forgetting literature, a number of potential solutions to the problem have been devised; some involve replaying previous-task examples during training on later tasks (Rebufﬁ et al., 2016; Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018; Shin et al., 2017; Riemer et al., 2018), while others like elastic weight consolidation (Kirkpatrick et al., 2017) limit the extent to which parameters can change once learned (Zenke et al., 2017; Lee et al., 2017; Farajtabar et al., 2019).
Perhaps most similar to our work are proposed mitigation methods which exploit the overparameterization of modern neural networks to perform continual learning of multiple tasks, e.g. by using pruning in Mallya & Lazebnik (2018) or task-speciﬁc subnetworks in Wortsman et al.
and recent works studying the impact of pretraining Anonymous (2022) and overparameterization via network width Mirzadeh et al.
(2021) on forgetting.
Large-scale models
The observation—initially in language models (Kaplan et al., 2020)—that model performance scales predictably with model and dataset size spurred an increased interest in training and deploying increasingly large pretrained models.
Large-scale models emerged in natural-language processing with models such as GPT (Radford et al., 2019) and BERT (Devlin et al., 2019), which are pretrained in an unsupervised manner and thus able to make use of the large corpora of natural text available on the internet.
More recently, image-classiﬁcation models have also beneﬁted from pretraining on large datasets (Kolesnikov et al., 2020), both supervised and unsupervised; a key example is the Vision Transformer architecture of Dosovitskiy et al.
(2020), which—despite not using the inductive bias provided by convolutions—attains downstream performance comparable to convolutional networks after pretraining on ImageNet-21k (Deng et al., 2009) or JFT-300M (Sun et al., 2017).
On many current NLP and computer vision benchmarks, large pretrained models achieve top performance.
All of the experiments in our work follow the pretrain-then-ﬁnetune method; recently, another paradigm, side-tuning, was proposed by Zhang et al.
(2020), which avoids forgetting altogether by leaving the pretrained model ﬁxed and employing auxiliary networks for downstream tasks.
E XPERIMENTAL SETUP
Tasks: Most of the experiments in this paper are conducted in the standard, task split setting (Kirkpatrick et al., 2017; Zenke et al., 2017).
We consider split CIFAR-10 task sequences – the ten class dataset is split and sequentially trained on two tasks of 5 classes each.
We also consider sequential 10 and 50 class splits of CIFAR-100.
We also consider the more recent input-distribution-shift CIFAR-100 setup (Ramasesh et al., 2021).
In this case each task has the same target types made up a ﬁxed subset of the 20 CIFAR-100 superclasses (e.g. aquatic mammals, ﬁsh, ...).
The sequence of tasks consists of modifying which subclasses are sampled from.
For example Task A may have aquatic mammals drawn from beaver and dolphin, while Task B may have otter and whale.
This setup allows one to probe forgetting without changing the model head, in a setting where task identity is not known to the model.
For our language model experiments, we consider next token prediction generative tasks and use the
IMDb Reviews (Maas et al., 2011) and english Wikipedia (Foundation) datasets.
Vision models:
In most of our experiments we use the Vision Transformer (ViT) and ResNet architectures.
For the Vision Transformers, we use the ViT-S and ViT-B conﬁgurations from Dosovitskiy et al.
(2020) and a smaller conﬁguration we call ViT-xS – ranging from 5.7-86.7 million parameters.
For ResNets, we use ﬁve models: ResNet26, ResNet50, ResNet101, ResNet152, and ResNet200 – ranging from 14.0-62.6 million parameters; as in Kolesnikov et al.
(2020), these models use group normalization (Wu & He, 2018) and weight standarization (Qiao et al., 2019) in place of batch normalization.
Details can be found in Tables 1 and 2 in Appendix A.1.2.
Pretraining: Unless otherwise speciﬁed, we pretrain our image models in a supervised manner on the ImageNet21k dataset, which contains approximately 14 million images in about 26000 distinct categories using the Adam optimizer (Kingma & Ba, 2017).
Further details are in the Appendix.
Finetuning: When ﬁne-tuning on multiple tasks in sequence, we use the SGD optimizer with momentum β = 0 .
The ﬁrst task training employs a cosine decay schedule with a linear warmup; later tasks in the sequence are trained with ﬁxed learning rates.
For the split-CIFAR-10 task sequences, we use a multi-head setup in which each head is initialized to zeros.
R ESULTS 3. F ORGETTING IMPROVES WITH MODEL SCALE
Our main ﬁnding is that for both pretrained Vision Transformers and pretrained ResNets, catastrophic forgetting is mitigated to a large extent by scale: that is, larger models suffer less from forgetting.
This is shown in Figure 1 for the two-task split CIFAR-10 sequence: Task A { airplane, automobile, bird, cat, deer } →
Task B { dog, frog, horse, ship, truck } .
In order to make a fair comparison, we plot the test accuracies of the model on both tasks at different steps of Task B ﬁne-tuning for a variety of learning rates.
This allows us to see, for each model, the maximum performance achievable on the new task for a given performance on the initial task.
We call this the forgetting frontier.
Figure 1 shows that the forgetting frontier for both ViT and ResNet models improves signiﬁcantly with scale.
While a 5.7M-parameter ViT-xS model loses about 6.5% in Task A accuracy over the course of learning to perform Task B, an 86.7M-parameter ViT-B model loses less than half a percent.
To put this in context, in a previous studies of split-CIFAR-10 task sequences, a ResNet model lost close to 40% accuracy with no mitigation methods, 5% when using a replay buffer, and 20% when using elastic weight consolidation (see Figure 1 of Ramasesh et al.
for an example).
It is also important to note that this effect of scale and pretraining on improving forgetting frontiers does not seem to be architecture-dependent.
Both vision transformers—based on self-attention— and ResNets—based on convolutions—exhibit improvements in their forgetting frontiers as they grow larger, as long as they are pretrained.
We see this improvement in forgetting performance beyond the simple two-task split-CIFARscenario.
Across a 10-task sequence of 10-class split-CIFAR-100 tasks (Figure 2 left), sequential 50-class splits of CIFAR-100 (Figure 2 right), and the CIFAR-100 single-head input-distributionshift task (Figure 3), large-scale pretrained models exhibit resistance to forgetting, and continual learning ability appears to improve systematically with scale.
3. F ORGETTING IN PRETRAINED MODELS VS . MODELS TRAINED FROM SCRATCH
In the previous section, we saw that the amount of catastrophic forgetting in models pretrained on large datasets decreased with increasing model size, to the point that the largest models (ViT-B, with 86 .
7 M parameters and ResNet200 with 62 .
6 M parameters) suffer very little performance loss from second-task training.
In this section, we investigate whether this improvement in forgetting is simply a consequence of the improvement in performance on Task A.
That is, do pretraining and scale have beneﬁts with respect to forgetting beyond simply improving the ﬁne-tuned performance on downstream tasks?
To check this, we compare the performance and amount of forgetting in pretrained ResNet models to models trained from scratch.
We handicap the pretrained ResNets so that their maximum accuracy on on
Task A matches that of the corresponding trained-from-scratch ResNet; this is done simply by reducing the number of steps for which we ﬁne-tune.
Figure 4 shows the results of this experiment, with additional results and details in Appendix B.1.
While the trained-from-scratch ResNets achieve the same performance as their pretrained counterparts on Task A, their Task A performance degrades signiﬁcantly more than the pretrained ResNets when trained on Task B; pretrained ResNets forget less, and this improves with model scale.
This result suggests that pretraining and model scale confer beneﬁts upon models which go beyond simply improving performance.
From Figure 4 it is also clear (see also Figure 12 in the appendix, which plots the same data in a different way) that pretraining is required to see the beneﬁts of scale.
Simply scaling up the model size in trained-from-scratch ResNets does not yield clear improvements in their forgetting performance.
We focused in this section exclusively on ResNets (rather than Vision Transformers) because obtaining decent performance on image classiﬁcation tasks with Vision Transformers seems to require pretraining.
In our experiments, for example, a trained-from-scratch ViT-S model was unable to achieve better than 75 % test accuracy on ﬁve-class CIFAR-10, while the same model, pretrained, achieved better than 98 % accuracy.
3. D EPENDENCE ON PRETRAINING TIME , PRETRAINING DATASET SIZE , AND FINETUNING TIME
Having seen in the previous sections that pretraining on large datasets improves models’ resilience to catastrophic forgetting, in this section we explore the dependence of this phenomenon on the duration of pretraining, the size of the pretraining dataset, and the number of steps for which we ﬁnetune on Task A.
First, we study the inﬂuence of pretraining time on downstream forgetting.
This is motivated by previous studies (e.g. Hernandez et al.
showing that downstream performance does not always monotonically improve with pretraining time and performance.
By contrast, in the case of forgetting we ﬁnd (see Figure 5 for ViT-S results) that as the pretraining process continues, both downstream performance and the forgetting frontier improve.
This was the case for all models (ViTs and ResNets) and downstream task sequences (CIFAR-10, CIFAR-100, etc.)
we studied.
We found no evidence of forgetting frontiers getting worse with increased pretraining.
Along similar lines, we show the inﬂuence of pretraining data scale on downstream forgetting in
Figure 6, pretraining a model on varying fractions of ImageNet21k.
While the full dataset contains over 14 million images, we ﬁnd that training on even one-sixteenth of the dataset (roughly the size of the standard ImageNet ILSVRC 2012 dataset) yields a model which only loses about 3% performance in forgetting on split-CIFAR-10.
This suggests that one does not need to pretrain models on unreasonably large datasets in order to reap beneﬁts for continual learning.
Finally, we study the forgetting frontier of pretrained models as we vary the amount of ﬁrst-task ﬁne-tuning (number of training steps).
This is partly inspired by a recent study (Andreassen et al., 2021) of robustness in pretrained models, which found that during ﬁne-tuning, pretrained models evolved some robustness to distribution shift which vanished at the end of ﬁne-tuning.
Figureshows forgetting frontiers for pretrained ViT-B and ResNet101 models, ﬁne-tuned for a number of steps between 100 and 5000.
The results in the ﬁgure suggest that in our setting, increasing the ﬁne-tuning steps to maximize performance on Task A does not appear to cause increased forgetting.
On the contrary, even training for about ten times more steps than it takes to achieve saturating performance does not impact forgetting.
3. F ORGETTING IN MODELS WITH UNSUPERVISED PRETRAINING
In previous sections, the models we studied were pretrained in a supervised manner on ImageNet21k.
Part of the conceptual motivation for this work came from the properties of language models trained in an unsupervised fashion.
Furthermore, recently, unsupervised pretraining methods have yielded vision models which achieve good downstream performance; among these are
SimCLR (Chen et al., 2020), MoCo (He et al., 2020), DINO (Caron et al., 2021), and BYOL (Grill et al., 2020).
Given the increased prevalence of such unsupervised pretraining methods, we investigate in this section whether unsupervised pretraining, combined with scale, provides resistance to forgetting similarly to supervised pretraining.
We take publicly-available ResNet models which were pretrained using SimCLR on the ImageNet ILSVRC-2012 dataset (Russakovsky et al., 2015), and ﬁnetune them on the two-task split CIFAR-10 sequence.
The resulting forgetting frontiers, shown in Figure 7, show that as in supervised pretraining, SimCLR pretraining yields models which are resilient to forgetting, especially as scale is increased.
We leave a thorough exploration of the continual-learning properties of other unsupervised pretraining methods for future work.
3. R EPRESENTATION OVERLAP AND FORGETTING
In this section we take steps toward understanding why large pretrained models seem to be resistant to forgetting.
Following the work of Ramasesh et al.
(2021), we measure the similarity between a models representation of Task A and Task B data.
In detail, if we denote the P model features (penultimate layer activations) on an input, x , by f ( x ) ∈ R P , Ramasesh et al. (2021) propose measuring the similarity of f A :=
{ f ( x ) : x ∈ Task A } ∈
R | Task A |× P and f B :=
{ f ( x ) : x ∈ Task B } ∈ R | Task B |× P using the trace overlap ,
[Equation]
[Equation]
Here Θ AB = f
A f TB is the matrix of inner-products between Task A and Task B features.
Ramasesh et al.
(2021) show empirically that forgetting is maximal for intermediate values of S AB , and derive this property analytically in a simpliﬁed model.
In particular Ramasesh et al.
(2021) propose that a minimal trace overlap (corresponding to orthogonality of representations of different tasks) will lead to minimal forgetting.
Inspired by this, we measure representation overlaps in models ﬁnetuned on split CIFAR-10, taking representations at the end of Task A training.
We consider the 10x10 matrix, S , formed by taking the indices A and B to run over the different classes in CIFAR-10.
The numerator in the above expression can be thought of as an empirical (batch) estimate of the inner product between feature vectors for random draws from classes A and
B ; trace overlaps close to zero indicate that representations are nearly orthogonal.
Using the above deﬁnition, we return to the original split-CIFAR-10 setting of §3.1 and §3.2.
For each of the models used in those experiments (see Tables 1 and 2), we compute the representation overlap matrix.
We focus on two comparisons: (i) how does the representation overlap matrix differ between pretrained and trained-from-scratch ResNets?
and (ii) how does the orthogonality of representations vary as we scale the model size, for both ResNets and Vision Transformers?
Pretrained vs. trained-from-scratch representations : We show representative trace overlap matrices for both pretrained and trained-from-scratch ResNet101 models in Figure 8(left and center)
(see Figure 13 for additional models).
Each element of this matrix corresponds to the representation overlap between two batches, each drawn entirely from a single class.
The representation overlap between disjoint classes (off-diagonal class overlap matrix elements) is much smaller for the pretrained ResNets than those trained from scratch.
Averaging cross-class overlaps over class pairs from different tasks quantiﬁes this difference: for trained-from-scratch ResNet101, the average overlap is 0 .
907 , while for the pretrained model this value is only 0 . 466 .
For all sizes of ResNet, the pretrained models exhibit representations which have much smaller overlap than their trained-fromscratch counterparts.
Representation overlap across scales —Figure 8(right) shows the average cross-task overlap for both ResNets and Vision Transformers plotted against model size (number of parameters).
For pretrained Vision Transformers and ResNets, this overlap decreases with model size, while for trainedfrom-scratch ResNets, there is not a clear decreasing trend.
Interestingly, Vision Transformers appear to form more orthogonal representations than ResNets do, a fact which can also be seen in the overlap matrices presented in supplementary Figure 14, and perhaps explains their improved forgetting performance.
These observations suggest that pretrained models store representations of different classes with much less overlap than trained-from-scratch models, and that this orthogonality increases with scale.
As this mirrors the the observed trend in forgetting in these models, it is suggestive that the increased orthogonality of representations is a partial explanation for the forgetting resilience seen in large pretrained models.
F ORGETTING IN LANGUAGE MODELS
Transfer learning from unsupervised pretraining is commonplace in Natural Language Processing.
It has become standard to pretrain models on data crawled from the internet and then ﬁnetune on smaller, downstream tasks (Devlin et al., 2019; Radford et al., 2019; Raffel et al., 2020; Brown et al., 2020).
This bears similarities with both the unsupervised pretraining and the experiments in Figure 3 where a single head was used to train on different distributions (subclasses).
Indeed, as mentioned above, part of the conceptual motivation for this work came from the observation that large language models with diverse pretraining perform well on tasks with no ﬁnetuning – ostensibly retaining information for many gradient steps.
In this subsection and in the Supplement, we include some preliminary empirical results probing catastrophic forgetting directly in large language models.
We consider decoder only transformer models pretrained on data scraped from the web, including
C4 (Raffel et al., 2020).
These models range in size from 57 million to 8.6 billion non-embedding parameters and were originally trained as part of the BIG-bench collaboration (2021).
In Figure 9, we consider sequentially ﬁnetuning the models on Task A: IMDb Reviews for 1.5k steps followed by Task B: english Wikipedia for 10k steps.
The longer ﬁnetuning time for Task B is intended to highlight forgetting, as these models are relatively robust.
We ﬁnd that, as in the case of pretrained vision models, the best joint Task A, Task B performance improves with scale.
D ISCUSSION
The central observation we have reported in this paper is that scaling up models pretrained on large datasets is effective at mitigating catastrophic forgetting in the settings studied.
While it was known that large pretrained models can achieve impressive performances in downstream tasks, our experiments show that pretraining and scale can confer beneﬁts beyond simply the achievable downstream loss or accuracy values.
A natural question is what relationship between pretraining and ﬁnetuning tasks is required to confer beneﬁt.
In the vision setting, we considered ImageNet pretraining followed by CIFAR classiﬁcation, so both pretraining and ﬁnetuning distributions consisted of natural images (albeit with different resolutions and collection techniques).
In the language setting, ﬁnetining and pretraining tasks both consisted of English language modeling.
It would be interesting to explore to what degree distributional overlap of pretraining and ﬁnetuning data underlies the improved resistence to forgetting.
Our work suggests natural follow-up investigations to further understand continual learning in large pretrained models.
First, while we have taken initial steps towards identifying the reason for the resistance to catastrophic forgetting we observed – noting that increased orthogonality of representations is a plausible mechanism – we have not performed extensive experiments showing that this is causally linked to the forgetting behavior.
Second, we explored unsupervised pretraining only via the SimCLR method; conﬁrming that our results hold in other label-free methods such as DINO,
BYOL, MoCo, and others would also be a valuable study.
Another natural direction for future work is understanding what underlies the systematic improvement with model scale (e.g. Figures 1 and 9 right).
The rate at which performance improves with model size in Figure 1, with power-law exponent close to − 1 , is reminiscent of the variance limited behavior discussed in Bahri et al.
Namely when the number of parameters is much larger than the training dataset size (as is the case for split-CIFAR ﬁntetuning) performance is predicted to scale as N − 1 .
This behavior is noticeably not seen in Figure 9 for language modeling – this is reminiscent of the disparate scaling behaviors observed in Sharma & Kaplan (2020) between language modeling and classiﬁcation tasks.
More generally, understanding what underlies and predicting the rate of improvement to forgetting with model size is an important direction for future research.
A E XPERIMENTAL DETAILS A. V ISION A.1. S UPERVISED PRETRAINING
Here we detail the settings used to in supervised pretraining, resulting in the models used in the experiments of sections 3.1, 3.2, 3.3, and 3.4.
We pretrain seven models: (ViT-B, ViT-S, ViT-xS,
R26x1, R50x1, R101x1, R152x1, R200x1) on ImageNet21k for 90 epochs.
Pretraining is done using the Adam optimizer ( β 1 = 0 .
9 and β 2 = 0 .
999 ); for all models, we use a batch size of 4096.
Our learning-rate schedule includes a warmup of 10k steps to a maximum learning rate η = 10 − 3 , followed by a linear decay to 10 − 5 .
For the Vision Transformer models, we use a weight decay penalty of 0 .
03 and a dropout rate of 0 .
1 ; for ResNet models, we do not use weight decay or dropout.
These settings are largely identical to those found in Dosovitskiy et al.
A note on ImageNet21k: We use the version of ImageNet21k available in TensorFlow Datasets 4.0. and apply minimal preprocessing, simply converting the labels to one-hot vectors and cropping the image to be 224x224 pixels in size.
During training, this crop is an Inception-style random crop, while during evaluation it is a deterministic central crop; during training we additionally augment the data by performing random horizontal ﬂips and apply label smoothing, with values 0 .
9999 and 0 . 0001 .
Importantly, ImageNet21k is class-imbalanced and heavy-tailed, with the uneven distribution of label occurrences shown in Figure 10.
We do not balance the dataset.
[Table]
we
[Table]
before
[Table]
A.1. V ISION T RANSFORMER MODEL DETAILS
For full description of the Vision Transformer architecture, see the original paper by Dosovitskiy et al.
Abbreviated details are given here.
We process 2D images of height H , width W , and
C channels, x ∈ R H × W × C by ﬁrst reshaping them into a 1D sequence of ﬂattened patches–in this work, we exclusively use patch size 16.
Each patch is mapped (via a learned linear transformation) to an embedding vector of dimension D (for our models, D xS = 256 , D S = 516 , and D B = 768 , as in Table 1).
As both our pretraining and ﬁnetuning is supervised, we prepend a (learnable) ‘[class]‘ token embedding to this sequence of patches, add a learnable 1D position embedding, and pass the sequence through the Transformer encoder.
The Transformer encoder alternates between multiheaded self-attention layers and MLP (plus residual connection) layers, with layernorm applied before both the attention and MLP layers.
For classiﬁcation, we use a classiﬁcation head which acts on only the transformed ‘[class]‘ token representation.
During pretraining, this classiﬁcation head is an single-hidden-layer MLP, as in the original Vision Transformer work; during ﬁnetuning, the classiﬁcation head is a linear layer (in the split-CIFAR-10 setting, we use a multi-head setup with a different classiﬁcation layer for each task in the sequence).
We initialize all classiﬁcation layers with zeros.
[Table] [Table]
A.1. CIFAR-100 D ISTRIBUTION S HIFT T ASK
Here we provide more information about the CIFAR-100 distribution-shift task, which was introduced in Ramasesh et al.
and forms the basis of our experiments in section A.1.3 of the main text.
The idea behind this task is to mimic a way in which catastrophic forgetting might occur in practice, in image-classiﬁcation settings where the class labels remain constant but the distribution of the images changes.
A key features of this type of setting is that the identity of the task is not known to the model at inference time, so task-speciﬁc components (such as multiple readout layers) are not allowed.
For example, in a medical setting, one might seek to classify images as displaying signs of a disease or not, and the system might be trained sequentially on images taken at various hospitals.
While the CIFAR-100 dataset features images divided into 100 distinct classes, these classes are grouped into 20 superclasses .
The CIFAR-100 distribution shift task sequence involves classifying an image by its superclass; individual tasks in the sequence draw images from a single subclass for each superclass.
As a concrete example, in the experiments in the main text we use the ﬁve superclasses aquatic mammals , fruits and vegetables , household electrical devices , trees , and vehicles-1 1 .
For the ﬁrst task in the sequence, Task A, we uses the classes dolphin (for aquatic mammals), apple (for fruits and vegetables), lamp (for household electrical devices), maple tree (for trees), and bicycle (for vehicles-1).
For the second task, Task B, we use the classes whale (for aquatic mammals), orange (for fruits and vegetables), television (for household electrical devices), willow (for trees), and motorcycle (for vehicles-2).
A. F INETUNING EXPERIMENTAL DETAILS
In this section we provide details on the ﬁnetuning procedures for the experiments in the main text.
All ﬁnetuning was done using stochastic gradient descent with momentum ( β = 0 . 9 ), clipping gradients at unity; batch size is ﬁxed at 512.
In the ﬁrst task, we used a schedule with a 5-step linear ramp to a maximum learning rate followed by a cosine decay to zero over the full ﬁrst-task training.
In the second task, we used constant learning rates.
Section 3.1, Figure 1: Finetuning both the Vision Transformers and ResNets was done forsteps.
For Vision Transformers, the ﬁrst-task training used a warm-up from η = 0 .
01 to η = 0 .
For each model we perform four different runs, with second-task (constant) learning rates 0 .
007 , and 0 .
For ResNets, the task-A learning rate warm-up was from η = 0 .
002 to η = 0 .
The (constant) second-task learning rates we used were 0 .
001 , 0 . 0003 , and 0 . 0001 .
Section 3.1, Figure 2:
In the 10-task split-CIFAR100 sequence (left), each task was trained for 200 steps; the initial task used a learning-rate schedule with a 5-step warmup from η = 0 .
01 to η = 0 .
05 followed by a cosine decay to 0 over the 200 steps, while all subsequent steps used constant η = 0 .
In the 2-task 50-class split-CIFAR100 sequence (right), the settings are identical to those of the Vision Transformer training in Figure 1.
Section 3.1, Figure 3: For ResNets, we used a ﬁrst-task warmup from η = 0 .
001 to η = 0 .
005 , and trained each task for 300 steps.
Constant second-task learning rates were 0 .
001 , 0 . 0003 , and
[Table]
For Vision Transformers, we trained each task for 200 steps, using a ﬁrst-task warmup from η = 0 .
01 to η = 0 .
Constant second-task learning rates were 0 .
01 , 0 . 003 , and 0 .
Section 3.2, Figure 4: The pretrained models in this ﬁgure were the same as those in Figure 1.
For the trained-from-scratch models, we used a linear warm-up from η = 0 .
006 to η = 0 .
03 , and trained each task for 15,000 steps.
Constant second-task learning rates were 0 .
001 , 0 . 0003 , and 0 . 0001 .
Note that training here is much longer than for the pretrained models.
Section 3.3, Figure 5: Vision Transformer training settings are identical to those of Figure 1.
Section 3.3, Figure 6:
In the left panel, Vision Transformer training settings are identical to those of
Figure 1.
In the right two panels, settings are also identical to those of Figure 1, with the exception of the number of ﬁnetuning steps, as indicated in the ﬁgure legend.
Section 3.4, Figure 7: We ﬁnetune for 300 steps on each task, with a maximum learning rate of η = 0 .
(we omit the linear warm-up for these models, but keep the cosine decay).
Constant second-task learning rates are η = 0 . 0002 , η = 0 .
0001 , η = 0 .
00004 , and η = 0 . 00002 .
A. L ANGUAGE
Models are decoder-only transformers which were pretrained using Adafactor and learning rate 1 . 0 , context length 1024 and batch size 256 .
We ﬁnetuned with batch size of 8 and set a constant learning rate equal to the learning rate at the end of pretraining rescaled by the relative batch size, 8 / 256 .
B A DDITIONAL RESULTS B. C OMPARING PRETRAINED AND TRAINED - FROM - SCRATCH R ES N ETS
In the main text, we showed forgetting frontiers on the split-CIFAR-10 task sequence for selected
ResNet models: ResNet26, ResNet101, and ResNet200.
Here, we show the forgetting frontiers for all ResNets (11), and we also plot the data in a slightly different way (12) to make it apparent that while the forgetting frontiers of the pretrained ResNet models clearly improve with model size, the forgetting frontiers of trained-from-scratch ResNet models do not.
B. H EAD - ONLY TRAINING
One way to probe the quality of representations generated by the pre-trained models are to train a simple linear classiﬁer on the initial ﬁve-class split of CIFARR10, freezing the representations after pre-training (on ImageNet21k).
As shown in Table 3, even with freezing the model weights, a linear classiﬁer can achieve high accuracy on the downstream tasks
. B. R EPRESENTATION OVERLAP MATRICES FOR ALL R ES N ET AND V I T MODELS
In the main text, we described the computation of the classwise representation overlap matrices for our models at the end of Task A training, showing an example of such a matrix for both a pretrained and a trained-from-scratch ResNet101 model.
In Figure 13, we show representation overlap matrices for all of the ResNet models, which bears out our statement in the main text that pretrained ResNet representations were much closer to orthogonal than their trained-from-scratch counterparts.
In
Figure 14, we show representation overlap matrices
As with the pretrained ResNet models, the cross-class representation overlaps become closer to orthogonal as we increase the model size.
[Figure]
Task A
Task B
Task C
Task D
Task E
Apple
Bowl
Chair
Dolphin
Lamp
Aquarium Fish
Boy
Chimpanzee
Elephant
Lawnmower
Baby
Bridge
Clock
Flatﬁsh
Leopard
Bear
Bus
Cloud
Forest
Lion
Beaver
Butterﬂy
Cockroach
Fox
Lizard
Bed
Camel
Couch
Girl
Lobster
Bee
Can
Crab
Hamster
Man
Beetle
Castle
Crocodile
House
Maple tree
Bicycle
Caterpillar
Cup
Kangaroo
Motorcycle
Bottle
Cattle
Dinosaur
Keyboard
Mountain
Task F
Task G
Task H
Task I
Task J
Mouse
Plain
Rose
Squirrel
Train
Mushroom
Plate
Sea
Streetcar
Trout
Oak tree
Poppy
Seal
Sunﬂower
Tulip
Orange
Porcupine
Shark
Sweet pepper
Turtle
Orchid
Possum
Shrew
Table
Wardrobe
Otter
Rabbit
Skunk
Tank
Whale
Palm tree
Raccoon
Skyscraper
Telephone
Willow tree
Pear
Ray
Snail
Television
Wolf
Pickup truck
Road
Snake
Tiger
Woman
Pine tree
Rocket
Spider
Tractor
Worm
B. M ORE LANGUAGE RESULTS
For completeness, Figure 15 displays the token accuracy on task A as a function of the number of steps.
B. E XTENSION PAST THE TWO - TASK SETTING
In this section, we extend our results on vision past the two-task setting which was the primary focus of the main text.
Speciﬁcally, we use a 10-class, 10-subtask split of the CIFAR100 dataset to study how forgetting in this scenario varies as a function of model and dataset scale.
The speciﬁc split we use is shown in Table 4.
As in the main text, we study the performance of Vision Transformer and
ResNet models.
We ﬁnd that:
• As in the two-task setting, both the forgetting frontiers and the best average accuracy improve with model scale as a power-law in the number of model parameters.
• When we scale the size of the pre-training dataset, we also observe an improvement with scale, though with some saturation around the full ImageNet21k dataset..
Model scaling: Model-scaling plots showing the best average accuracy as a function of model parameters are show in Figure 16.
As in the two-task split CIFAR10 sequence, these scaling plots are one way to show that the continual-learning performance of both architectures improves with scale.
The power-law exponents are different in this 10-task setting than they are for the two-task setting, but interestingly all are close to unity.
Another way to see the improvement in continual-learning ability with model scale is to plot forgetting frontiers, which we do in Figures 18 and 19.
In this setting with more than two tasks, plotting a single forgetting frontier is not possible, but we plot two-task frontiers for all pairs of subtasks (with 10 subtasks, there are 10 × 9 / 2 = 45 pairs).
These frontiers are constructed in exactly the same way as the two-task frontiers, showing the test accuracies for each subtask throughout the entirety of the 10-task training.
As the ﬁgures show, forgetting frontiers of larger models dominate the forgetting frontiers of smaller models of the same architecture.
This complements the previous scaling-plot as a demonstration of improving continual-learning with scale, though the behavior here is not as clean as it was in the two-task setting.
Data scaling: A data-scaling plot showing the best average accuracy for a ViT-S model as a function of pretraining dataset size is shown in Figure 17.
Forgetting frontiers are shown in Figure 20.
C L EARNING RATE AND STEP
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]