Original: A NATOMY OF C ATASTROPHIC F ORGETTING : H IDDEN R EPRESENTATIONS AND T ASK S EMANTICS
Revised: A NATOMY OF C ATASTROPHIC F ORGETTING : H IDDEN R EPRESENTATIONS AND T ASK S EMANTICS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Catastrophic forgetting is a recurring challenge to developing versatile deep learning models.
Revised: Catastrophic forgetting is a recurring challenge to developing versatile deep learning models.

Original: Despite its ubiquity, there is limited understanding of its connections to neural network (hidden) representations and task semantics.
Revised: Despite its ubiquity, there is limited understanding of its connections to neural network (hidden) representations and task semantics.

Original: In this paper, we address this important knowledge gap.
Revised: In this paper, we address this important knowledge gap.

Original: Through quantitative analysis of neural representations, we ﬁnd that deeper layers are disproportionately responsible for forgetting, with sequential training resulting in an erasure of earlier task representational subspaces.
Revised: Through quantitative analysis of neural representations, we ﬁnd that deeper layers are disproportionately responsible for forgetting, with sequential training resulting in an erasure of earlier task representational subspaces.

Original: Methods to mitigate forgetting stabilize these deeper layers, but show diversity on precise effects, with some increasing feature reuse while others store task representations orthogonally, preventing interference.
Revised: Methods to mitigate forgetting stabilize these deeper layers, but show diversity on precise effects, with some increasing feature reuse while others store task representations orthogonally, preventing interference.

Original: These insights also enable the development of an analytic argument and empirical picture relating forgetting to task semantic similarity, where we ﬁnd that maximal forgetting occurs for task sequences with intermediate similarity.
Revised: These insights also enable the development of an analytic argument and empirical picture relating forgetting to task semantic similarity, where we ﬁnd that maximal forgetting occurs for task sequences with intermediate similarity.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: While the past few years have seen the development of increasingly versatile machine learning systems capable of learning complex tasks (Stokes et al., 2020; Raghu & Schmidt, 2020; Wu et al., 2019b), catastrophic forgetting remains a core capability challenge.
Revised: While the past few years have seen the development of increasingly versatile machine learning systems capable of learning complex tasks (Stokes et al., 2020; Raghu & Schmidt, 2020; Wu et al., 2019b), catastrophic forgetting remains a core capability challenge.

Original: Catastrophic forgetting is the ubiquitous phenomena where machine learning models trained on non-stationary data distributions suffer performance losses on older data instances.
Revised: Catastrophic forgetting is the ubiquitous phenomena where machine learning models trained on non-stationary data distributions suffer performance losses on older data instances.

Original: More speciﬁcally, if our machine learning model is trained on a sequence of tasks, accuracy on earlier tasks drops signiﬁcantly.
Revised: More speciﬁcally, if our machine learning model is trained on a sequence of tasks, accuracy on earlier tasks drops signiﬁcantly.

Original: The catastrophic forgetting problem manifests in many sub-domains of machine learning including continual learning (Kirkpatrick et al., 2017), multi-task learning (Kudugunta et al., 2019)
Revised: The catastrophic forgetting problem manifests in many sub-domains of machine learning including continual learning (Kirkpatrick et al., 2017), multi-task learning (Kudugunta et al., 2019)

Original: , standard supervised learning through input distribution shift (Toneva et al., 2019; Snoek et al., 2019; Rabanser et al., 2019; Recht et al., 2019) and data augmentation (Gontijo-Lopes et al., 2020).
Revised: , standard supervised learning through input distribution shift (Toneva et al., 2019; Snoek et al., 2019; Rabanser et al., 2019; Recht et al., 2019) and data augmentation (Gontijo-Lopes et al., 2020).

Original: Mitigating catastrophic forgetting has been an important research focus (Goodfellow et al., 2013;
Revised: Mitigating catastrophic forgetting has been an important research focus (Goodfellow et al., 2013;

Original: Kirkpatrick et al., 2017; Lee et al., 2017; Li et al., 2019; Serr`a et al., 2018; Ritter et al., 2018;
Revised: Kirkpatrick et al., 2017; Lee et al., 2017; Li et al., 2019; Serr`a et al., 2018; Ritter et al., 2018;

Original: Rolnick et al., 2019), but many methods are only effective in speciﬁc settings (Kemker et al., 2018), and progress is hindered by limited understanding of catastrophic forgetting‘s fundamental properties.
Revised: Rolnick et al., 2019), but many methods are only effective in speciﬁc settings (Kemker et al., 2018), and progress is hindered by limited understanding of catastrophic forgetting‘s fundamental properties.

Original: How does catastrophic forgetting affect the hidden representations of neural networks?
Revised: How does catastrophic forgetting affect the hidden representations of neural networks?

Original: Are earlier tasks forgotten equally across all parameters?
Revised: Are earlier tasks forgotten equally across all parameters?

Original: Are there underlying principles common across methods to mitigate forgetting?
Revised: Are there underlying principles common across methods to mitigate forgetting?

Original: How is catastrophic forgetting affected by (semantic) similarities between sequential tasks?
Revised: How is catastrophic forgetting affected by (semantic) similarities between sequential tasks?

Original: This paper takes steps to answering these questions, speciﬁcally:
Revised: This paper takes steps to answering these questions, speciﬁcally:

Original: With experiments on split CIFAR-10, a novel distribution-shift CIFAR-100 variant, CelebA and ImageNet we analyze neural network layer representations, ﬁnding that higher layers are disproportionately responsible for catastrophic forgetting, the sequential training process erasing earlier task subspaces.
Revised: With experiments on split CIFAR-10, a novel distribution-shift CIFAR-100 variant, CelebA and ImageNet we analyze neural network layer representations, ﬁnding that higher layers are disproportionately responsible for catastrophic forgetting, the sequential training process erasing earlier task subspaces.

Original: We investigate different methods for mitigating forgetting, ﬁnding that while all stabilize higher layer representations, some methods encourage greater feature reuse in higher layers, while others store task representations as orthogonal subspaces, preventing interference.
Revised: We investigate different methods for mitigating forgetting, ﬁnding that while all stabilize higher layer representations, some methods encourage greater feature reuse in higher layers, while others store task representations as orthogonal subspaces, preventing interference.

Original: We study the connection between forgetting and task semantics, ﬁnding that semantic similarity between subsequent tasks consistently controls the degree of forgetting. 4.
Revised: We study the connection between forgetting and task semantics, ﬁnding that semantic similarity between subsequent tasks consistently controls the degree of forgetting. 4.

Original: Informed by the representation results, we construct an analytic model that relates task similarity to representation interference and forgetting.
Revised: Informed by the representation results, we construct an analytic model that relates task similarity to representation interference and forgetting.

Original: This provides a quantitative empirical measure oftask similarity, and together these show that forgetting is most severe for tasks with intermediate similarity.
Revised: This provides a quantitative empirical measure of

Original: [Figure]
Revised: [Figure]

Revised: task similarity, and together these show that forgetting is most severe for tasks with intermediate similarity.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Mitigation strategies Developing mitigation strategies for catastophic forgetting is an active area of research (Kemker et al., 2018).
Revised: Mitigation strategies Developing mitigation strategies for catastophic forgetting is an active area of research (Kemker et al., 2018).

Original: Popular approaches based on structural regularization include
Revised: Popular approaches based on structural regularization include

Original: Elastic Weight Consolidation (Kirkpatrick et al., 2017) and Synaptic Intelligence (Zenke et al., 2017).
Revised: Elastic Weight Consolidation (Kirkpatrick et al., 2017) and Synaptic Intelligence (Zenke et al., 2017).

Original: An alternative, functional regularization approach is based on storing and replaying earlier data in a replay buffer (Schaul et al., 2015; Robins, 1995; Rolnick et al., 2019).
Revised: An alternative, functional regularization approach is based on storing and replaying earlier data in a replay buffer (Schaul et al., 2015; Robins, 1995; Rolnick et al., 2019).

Original: We study how these mitigation methods affect the internal network representations to better understand their role in preventing catastrophic forgetting.
Revised: We study how these mitigation methods affect the internal network representations to better understand their role in preventing catastrophic forgetting.

Original: Understanding catastrophic forgetting Our work is similar in spirit to existing empirical studies aimed at better understanding the catastrophic forgetting phenomenon (Goodfellow et al., 2013;
Revised: Understanding catastrophic forgetting Our work is similar in spirit to existing empirical studies aimed at better understanding the catastrophic forgetting phenomenon (Goodfellow et al., 2013;

Original: Toneva et al., 2019).
Revised: Toneva et al., 2019).

Original: We focus speciﬁcally on understanding how layerwise representations change, and on the relation between forgetting and task semantics, which have not previously been explored.
Revised: We focus speciﬁcally on understanding how layerwise representations change, and on the relation between forgetting and task semantics, which have not previously been explored.

Original: This semantic aspect is related to recent work by Nguyen et al.
Revised: This semantic aspect is related to recent work by Nguyen et al.

Original: (2019) examining the inﬂuence of task sequences on forgetting.
Revised: (2019) examining the inﬂuence of task sequences on forgetting.

Original: It also builds on work showing that learning in both biological and artiﬁcial neural networks is affected by semantic properties of the training data (Saxe et al., 2019;
Revised: It also builds on work showing that learning in both biological and artiﬁcial neural networks is affected by semantic properties of the training data (Saxe et al., 2019;

Original: Mandler & McDonough, 1993).
Revised: Mandler & McDonough, 1993).

Original: Fine-tuning and transfer learning While not studied in the context of catastrophic forgetting, layerwise learning dynamics have been investigated in settings other than continual learning.
Revised: Fine-tuning and transfer learning While not studied in the context of catastrophic forgetting, layerwise learning dynamics have been investigated in settings other than continual learning.

Original: For example, Raghu et al.
Revised: For example, Raghu et al.

Original: (2017) showed that layerwise network representations roughly converge bottomup ; (from input to output).
Revised: (2017) showed that layerwise network representations roughly converge bottomup ; (from input to output).

Original: Neyshabur et al.
Revised: Neyshabur et al.

Original: (2020) observed similar phenomena in a transfer-learning setting with images, as have others recently in transformers for NLP (Wu et al., 2020; Merchant et al., 2020).
Revised: (2020) observed similar phenomena in a transfer-learning setting with images, as have others recently in transformers for NLP (Wu et al., 2020; Merchant et al., 2020).

Original: Furthermore, the observation that early layers in networks learn general features like edge detectors, while latter layers learn more task speciﬁc features is a well-known result in computer vision (Erhan et al., 2009), and here we quantitatively study its ramiﬁcations for sequential training and catastrophic forgetting.
Revised: Furthermore, the observation that early layers in networks learn general features like edge detectors, while latter layers learn more task speciﬁc features is a well-known result in computer vision (Erhan et al., 2009), and here we quantitatively study its ramiﬁcations for sequential training and catastrophic forgetting.

Original:  S ETUP
Revised:  S ETUP

Original: Tasks:
Revised: Tasks:

Original: We conduct this study over many different tasks and datasets: (i) Split CIFAR-10, where the ten class dataset is split into two tasks of 5 classes each (ii) input distribution shift CIFAR-100, where each task is to distinguish between the CIFAR-100 superclasses, but input data for each task is a different subset of the constituent classes of the superclass (see Appendix A.3) (iii) CelebA attribute prediction: the two tasks have input data either men or women, and we predict either smile or mouth open (iv) ImageNet superclass prediction, similar to CIFAR100.
Revised: We conduct this study over many different tasks and datasets: (i) Split CIFAR-10, where the ten class dataset is split into two tasks of 5 classes each (ii) input distribution shift CIFAR-100, where each task is to distinguish between the CIFAR-100 superclasses, but input data for each task is a different subset of the constituent classes of the superclass (see Appendix A.3) (iii) CelebA attribute prediction: the two tasks have input data either men or women, and we predict either smile or mouth open (iv) ImageNet superclass prediction, similar to CIFAR100.

Original: Models: We perform experiments with three common neural network architectures used in image classiﬁcation — VGG (Simonyan & Zisserman, 2014), ResNet (He et al., 2015) and DenseNet (Huang et al., 2016).
Revised: Models: We perform experiments with three common neural network architectures used in image classiﬁcation — VGG (Simonyan & Zisserman, 2014), ResNet (He et al., 2015) and DenseNet (Huang et al., 2016).

Original: Examples of catastrophic forgetting in these models are shown in Figure 1. 
Revised: Examples of catastrophic forgetting in these models are shown in Figure 1. [Figure]

Original:  C ATASTROPHIC F ORGETTING AND H IDDEN L AYER R EPRESENTATIONS
Revised:  C ATASTROPHIC F ORGETTING AND H IDDEN L AYER R EPRESENTATIONS

Original: We begin by investigating how catastrophic forgetting manifests in the hidden representations of the neural network.
Revised: We begin by investigating how catastrophic forgetting manifests in the hidden representations of the neural network.

Original: Do all parameters (and layers) forget equally?
Revised: Do all parameters (and layers) forget equally?

Original: Or are speciﬁc components of the network particularly responsible for the drop in accuracy in sequential training?
Revised: Or are speciﬁc components of the network particularly responsible for the drop in accuracy in sequential training?

Original: Through a combination of layer freezing experiments, and representational analysis, we ﬁnd that higher layers[Figure] (layers closest to output) are disproportionately responsible for catastrophic forgetting, with lower layers (layers closest to input) remaining representationally stable through sequential training.
Revised: Through a combination of layer freezing experiments, and representational analysis, we ﬁnd that higher layers (layers closest to output) are disproportionately responsible for catastrophic forgetting, with lower layers (layers closest to input) remaining representationally stable through sequential training.

Original: Further analysis of the representational subspaces of Task 1, Task 1 (post Task 2 training) and Task 2, provide insights on both feature reuse and interference.
Revised: Further analysis of the representational subspaces of Task 1, Task 1 (post Task 2 training) and Task 2, provide insights on both feature reuse and interference.

Original: 4. F REEZING L AYER R EPRESENTATIONS
Revised: 4. F REEZING L AYER R EPRESENTATIONS

Original: To study the effect of individual layers on forgetting, we measure the effect of freezing layer representations on Task 2 accuracy (Figure 2).
Revised: To study the effect of individual layers on forgetting, we measure the effect of freezing layer representations on Task 2 accuracy (Figure 2).

Original: Speciﬁcally, we freeze a contiguous block of layers (starting from the lowest layer) after training on Task 1, and only train the remaining layers on Task 2.
Revised: Speciﬁcally, we freeze a contiguous block of layers (starting from the lowest layer) after training on Task 1, and only train the remaining layers on Task 2.

Original: Across different architectures and tasks, we observe that lower layers can be reliably frozen with very little impact on Task 2 accuracy.
Revised: Across different architectures and tasks, we observe that lower layers can be reliably frozen with very little impact on Task 2 accuracy.

Original: This suggests the possibility of lower layer features being reused between both tasks, with higher layers being the main contributor to catastrophic forgetting.
Revised: This suggests the possibility of lower layer features being reused between both tasks, with higher layers being the main contributor to catastrophic forgetting.

Original: 4. R EPRESENTATIONAL S IMILARITY THROUGH S EQUENTIAL T RAINING
Revised: 4. R EPRESENTATIONAL S IMILARITY THROUGH S EQUENTIAL T RAINING

Original: The results of Figure 2 illustrate that lower layer representations on Task 1 can be reused for good performance on Task 2.
Revised: The results of Figure 2 illustrate that lower layer representations on Task 1 can be reused for good performance on Task 2.

Original: To determine if this is what actually occurs during the training process, we turn to Centered Kernel Alignment (CKA) (Kornblith et al., 2019), a neural network representation similarity measure.
Revised: To determine if this is what actually occurs during the training process, we turn to Centered Kernel Alignment (CKA) (Kornblith et al., 2019), a neural network representation similarity measure.

Original: CKA and other related algorithms (Raghu et al., 2017; Morcos et al., 2018) provide a scalar score (between 0 and 1) determining how similar a pair of (hidden) layer representations are, and have been used to study many properties of deep neural networks (Gotmare et al., 2018; Kudugunta et al., 2019; Wu et al., 2019a).
Revised: CKA and other related algorithms (Raghu et al., 2017; Morcos et al., 2018) provide a scalar score (between 0 and 1) determining how similar a pair of (hidden) layer representations are, and have been used to study many properties of deep neural networks (Gotmare et al., 2018; Kudugunta et al., 2019; Wu et al., 2019a).

Original: Speciﬁcally, letting X ∈ R n × p and Y ∈ R n × p be (centered) layer activation matrices of (the same) n datapoints and p neurons, CKA computes
Revised: Speciﬁcally, letting X ∈ R n × p and Y ∈ R n × p be (centered) layer activation matrices of (the same) n datapoints and p neurons, CKA computes

Original: [Equation]
Revised: [Equation]

Original: for HSIC Hilbert-Schmidt Independence Criterion (Gretton et al., 2005).
Revised: for HSIC Hilbert-Schmidt Independence Criterion (Gretton et al., 2005).

Original: We use linear-kernel CKA.
Revised: We use linear-kernel CKA.

Original: In Figure 3, we plot the results of computing CKA on Task 1 layer representations before and after Task 2 training.
Revised: In Figure 3, we plot the results of computing CKA on Task 1 layer representations before and after Task 2 training.

Original: Across architectures and tasks, we observe that the lower layers have high representation similarity, suggesting that lower layer Task 1 features are reused in Task 2.
Revised: Across architectures and tasks, we observe that the lower layers have high representation similarity, suggesting that lower layer Task 1 features are reused in Task 2.

Original: The higher layers however show signiﬁcant decreases in representation similarity, suggesting they disproportionately contribute to catastrophic forgetting.
Revised: The higher layers however show signiﬁcant decreases in representation similarity, suggesting they disproportionately contribute to catastrophic forgetting.

Original: These conclusions are further supported by additional layer reset experiments in Appendix Figure 12, which shows that rewinding higher layer parameters from post-Task 2 training to their pre-Task 2 training values signiﬁcantly improves Task 1 performance.
Revised: These conclusions are further supported by additional layer reset experiments in Appendix Figure 12, which shows that rewinding higher layer parameters from post-Task 2 training to their pre-Task 2 training values signiﬁcantly improves Task 1 performance.

Original: 4. F EATURE R EUSE AND S UBSPACE E RASURE
Revised: 4. F EATURE R EUSE AND S UBSPACE E RASURE

Original: Further insights on how the representations of lower and higher layers evolve during sequential training is given through a subspace similarity analysis.
Revised: Further insights on how the representations of lower and higher layers evolve during sequential training is given through a subspace similarity analysis.

Original: Letting X ∈ R n × p be the (centered) layer activation matrix of n examples by p neurons, we compute the PCA decomposition of X , i.e. the  eigenvectors ( v 1 , v 2 , ... ) and eigenvalues ( λ 1 , λ 2 ... ) of X T X .
Revised: Letting X ∈ R n × p be the (centered) layer activation matrix of n examples by p neurons, we compute the PCA decomposition of X , i.e. the [Figure] [Figure] eigenvectors ( v 1 , v 2 , ... ) and eigenvalues ( λ 1 , λ 2 ... ) of X T X .

Original: Letting V k be the matrix formed from the top k principal directions, v 1 , ..., v k as columns, and U k the corresponding matrix for a different activation matrix Y , we compute [Equation]
Revised: Letting V k be the matrix formed from the top k principal directions, v 1 , ..., v k as columns

Original: [Figure]
Revised: [Figure]

Revised: and U k the corresponding matrix for a different activation matrix Y , we compute

Original: [Figure]
Revised: [Equation]

Original: This measures the overlap in the subspaces spanned by ( v 1 , ..., v k ) and ( u 1 , ..., u k ) .
Revised: This measures the overlap in the subspaces spanned by ( v 1 , ..., v k ) and ( u 1 , ..., u k ) .

Original: Concretely, if X and Y correspond to layer activation matrices for two different tasks, SubspaceSim k ( X, Y ) measures how similarly the top k representations for those tasks are stored in the network.
Revised: Concretely, if X and Y correspond to layer activation matrices for two different tasks, SubspaceSim k ( X, Y ) measures how similarly the top k representations for those tasks are stored in the network.

Original: Figure 4 shows the result of computing SubspaceSim k ( X, Y ) for X, Y being the layer activation matrices in (i) (Task 1, Task 2) (purple line) (ii) (Task 1, Task 1 post-Task 2 training) (blue line) (iii) (Task 2, Task 1 post-Task 2 training) (blue line).
Revised: Figure 4 shows the result of computing SubspaceSim k ( X, Y ) for X, Y being the layer activation matrices in (i) (Task 1, Task 2) (purple line) (ii) (Task 1, Task 1 post-Task 2 training) (blue line) (iii) (Task 2, Task 1 post-Task 2 training) (blue line).

Original: Supporting the CKA analysis, lower layers have high subspace similarity between Task 1 and Task 2 (feature reuse), but higher layers have low similarity (representations change signiﬁcantly during Task 2 training.)
Revised: Supporting the CKA analysis, lower layers have high subspace similarity between Task 1 and Task 2 (feature reuse), but higher layers have low similarity (representations change signiﬁcantly during Task 2 training.)

Original: Furthermore, (Task 1, Task 1 post-Task 2 training) also shows low subspace similarity in the higher layers, indicating that the representational change in later layers is not restricted to Task 2 data, but also alters the networks representation of the initial task.
Revised: Furthermore, (Task 1, Task 1 post-Task 2 training) also shows low subspace similarity in the higher layers, indicating that the representational change in later layers is not restricted to Task 2 data, but also alters the networks representation of the initial task.

Original: Strikingly, (Task 2, Task 1 post-Task 2 training) shows high similarity in all layers, despite being different tasks .
Revised: Strikingly, (Task 2, Task 1 post-Task 2 training) shows high similarity in all layers, despite being different tasks .

Original: In summary, these results illustrate that during sequential training, effective feature reuse happens in the lower layers, but in the higher layers, after Task 2 training, Task 1 representations are mapped into the same subspace as Task 2.
Revised: In summary, these results illustrate that during sequential training, effective feature reuse happens in the lower layers, but in the higher layers, after Task 2 training, Task 1 representations are mapped into the same subspace as Task 2.

Original: Speciﬁcally, Task 2 training causes subspace erasure of Task 1 in the higher layers.
Revised: Speciﬁcally, Task 2 training causes subspace erasure of Task 1 in the higher layers.

Original: (Additional details and results are in Appendix Section A.5.)
Revised: (Additional details and results are in Appendix Section A.5.)

Original: F ORGETTING M ITIGATION M ETHODS AND R EPRESENTATIONS
Revised: [Figure] [Figure] [Figure]

Original: Informed by this understanding of how catastrophic forgetting manifests in the hidden layers (feature reuse in lower layers and subspace erasure in higher layers), we next study methods to mitigate
Revised: a c

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original:  [Figure] [Table] [Figure] [Table]
Revised:  F ORGETTING M ITIGATION M ETHODS AND R EPRESENTATIONS

Original: [Figure] [Table] [Figure] [Table]
Revised: Informed by this understanding of how catastrophic forgetting manifests in the hidden layers (feature reuse in lower layers and subspace erasure in higher layers), we next study methods to mitigate forgetting.

Original: [Figure]

Original: [Table] [Figure] [Table] [Figure]
Revised: Do popularly used mitigation methods act to stabilize higher layers — given possible alternate approaches such as weight orthogonalization (Appendix C)?

Original: [Table]

Original: [Figure] [Table] [Figure] [Table]
Revised: We investigate these questions across different successful mitigation strategies (i) EWC (elastic weight consolidation)

Original: [Figure] [Table] [Figure] [Table]
Revised: (Kirkpatrick et al., 2017), a regularization-based approach (ii) replay buffer (Schaul et al., 2015; Ratcliff, 1990; Rolnick et al., 2019), a rehearsal-styled approach (iii) SI (Synaptic

Original: [Figure]

Original: [Table] [Figure] [Table] [Figure]
Revised: Intelligence) (Zenke et al., 2017), another popular regularization-based approach.

Original: [Table]

Original: [Figure] [Table] [Figure] [Table]
Revised: In Figure 5, we plot the results of computing CKA on layer representations of Task 1 before/after training on Task 2, with different amounts of mitigation applied, and across different mitigation methods, tasks and architectures (full results in Appendix B.7.)

Original: [Figure] [Table] [Figure] [Table]
Revised: Across all of these settings, we observe that even a small amount of mitigation results in a signiﬁcant increase in similarity (stabilization) in the higher layers.

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure] forgetting. Do popularly used mitigation methods act to stabilize higher layers — given possible alternate approaches such as weight orthogonalization (Appendix C)? To answer this
Revised: However, there remains a central open question — is this increase in similarity due to feature reuse in the higher layers between Task 1 and Task 2, or does the network store Task 1 and

Original: [Figure]
Revised: Task 2 representations in non-interfering, orthogonal subspaces ?

Original:  we compute the Subspace Similarity measure introduced in Section 4.3, with results in Figure 6 (and additional experiments in Appendix Section B.5).
Revised: To answer this, we compute the Subspace Similarity measure introduced in Section 4.3, with results in Figure 6 (and additional experiments in Appendix Section B.5).

Original: These results reveal key differences between the mitigation methods: replay buffer results in orthogonal network subspaces for higher layer representations of
Revised: These results reveal key differences between the mitigation methods: replay buffer results in orthogonal network subspaces for higher [Figure] [Figure] Performance Drop on Task

Original:  Task 1 and Task 2, while EWC and SI promote feature reuse even in the higher layers.
Revised: Animals Artifacts [Table] layer representations of Task 1 and Task 2, while EWC and SI promote feature reuse even in the higher layers.

Original: In more detail:
Revised: In more detail:

Original: • Comparing the top row of Figure 6 to Figure 4 (the same plot without mitigation), we see that mitigation methods increase the subspace similarity of (Task 1, Task 1 Post-Task 2) and (Task 1, Task 2) representations in earlier/middling layers (more feature reuse).
Revised: • Comparing the top row of Figure 6 to Figure 4 (the same plot without mitigation), we see that mitigation methods increase the subspace similarity of (Task 1, Task 1 Post-Task 2) and (Task 1, Task 2) representations in earlier/middling layers (more feature reuse).

Original: • Most interestingly, comparing the different mitigation methods across top row Figure 6, we see that while EWC and SI maintain high subspace similarity of (Task 2, Task 

Revised: • Most interestingly, comparing the different mitigation methods across top row Figure 6, we see that while EWC and SI maintain high subspace similarity of (Task 2, Task

Original: Post-Task 2) even in the highest layers, replay signiﬁcantly decreases this value in the highest layers.
Revised: Post-Task 2) even in the highest layers, replay signiﬁcantly decreases this value in the highest layers.

Original: This suggests that replay performs mitigation through use of orthogonal subspaces, while EWC/SI encourage feature reuse even in the highest layers.
Revised: This suggests that replay performs mitigation through use of orthogonal subspaces, while EWC/SI encourage feature reuse even in the highest layers.

Original: In the bottom row we show subspace similarity of (Task 2, Task 

Revised: In the bottom row we show subspace similarity of (Task 2, Task

Original: Post-Task 2) for varying strengths of the different mitigation methods.
Revised: Post-Task 2) for varying strengths of the different mitigation methods.

Original: We observe a clear decrease in subspace similarity in higher layers even when a little replay is used (again supporting orthogonal subspaces), while EWC/SI maintain or increase this (feature reuse even in higher layers.)
Revised: We observe a clear decrease in subspace similarity in higher layers even when a little replay is used (again supporting orthogonal subspaces), while EWC/SI maintain or increase this (feature reuse even in higher layers.)

Original: We investigate these questions across different successful mitigation strategies (i) EWC (elastic weight consolidation)

Original: (Kirkpatrick et al., 2017), a regularization-based approach (ii) replay buffer

Original: [Figure]

Original: [Figure]

Original: Performance Drop on Task

Original: [Figure]

Original: [Table]

Original: (Schaul et al., 2015; Ratcliff, 1990; Rolnick et al., 2019), a rehearsal-styled approach (iii) SI (Synaptic

Original: Intelligence) (Zenke et al., 2017), another popular regularization-based approach.

Original: These mitigation methods have been speciﬁcally designed for the multi-head setting, so our experiments look at applying them to both split CIFAR-10 and split CIFAR-100.

Original: In Figure 5, we plot the results of computing CKA on layer representations of Task 1 before/after training on Task 2, with different amounts of mitigation applied, and across different mitigation methods, tasks and architectures (full results in Appendix B.7.)

Original: Across all of these settings, we observe that even a small amount of mitigation results in a signiﬁcant increase in similarity (stabilization) in the higher layers.

Original: However, there remains a central open question — is this increase in similarity due to feature reuse in the higher layers between Task 1 and Task 2, or does the network store Task 1 and

Original: Task 2 representations in non-interfering, orthogonal subspaces ?

Original:  S EMANTICS
Revised:  S EMANTICS

Original: With the insights on how catastrophic forgetting relates to the hidden representations, and the effects on network internals of different mitigation methods, we turn to investigating how the semantic similarity between the sequential tasks affects forgetting.
Revised: With the insights on how catastrophic forgetting relates to the hidden representations, and the effects on network internals of different mitigation methods, we turn to investigating how the semantic similarity between the sequential tasks affects forgetting.

Original: Speciﬁcally the prior results on lower layer feature reuse suggest a key question: does similarity between tasks result in less forgetting?
Revised: Speciﬁcally the prior results on lower layer feature reuse suggest a key question: does similarity between tasks result in less forgetting?

Original: Surprisingly, the answer is quite nuanced, depending on how precisely the different tasks are represented in the network.
Revised: Surprisingly, the answer is quite nuanced, depending on how precisely the different tasks are represented in the network.

Original: 6. F ORGETTING , T ASK S EMANTICS AND A S IMILARITY P UZZLE
Revised: 6. F ORGETTING , T ASK S EMANTICS AND A S IMILARITY P UZZLE

Original: Our study of the relationship between task semantic similarity and forgetting reveals a surprising puzzle.
Revised: Our study of the relationship between task semantic similarity and forgetting reveals a surprising puzzle.

Original: In Figure 8 and Table 1, we observe that depending on the task setup, a semantically similar  task may be forgotten less or more than a less similar task.
Revised: In Figure 8 and Table 1, we observe that depending on the task setup, a semantically similar [Figure] task may be forgotten less or more than a less similar task.

Original: This behavior is consistent across different settings and datasets.
Revised: This behavior is consistent across different settings and datasets.

Original: Speciﬁcally, in Figure 8a and 8d, Task 1 and Task 2 consist of binary classiﬁcations between animals and objects from CIFAR-10, where we ﬁnd that the similar categories [Figure]
Revised: Speciﬁcally, in Figure 8a and 8d, Task 1 and Task 2 consist of binary classiﬁcations between animals and objects from CIFAR-10, where we ﬁnd that the similar categories forget less.

Original: forget less. But in Figure 8b and 8e, a four category classiﬁcation between animals and objects followed by a binary classiﬁcation task results in the similar categories being forgotten more.
Revised: But in Figure 8b and 8e, a four category classiﬁcation between animals and objects followed by a binary classiﬁcation task results in the similar categories being forgotten more.

Original: Table 1 shows an analogous result on ImageNet.
Revised: Table 1 shows an analogous result on ImageNet.

Original: This effect of similar categories being forgotten more can also be observed in the CIFAR-100 distribution shift setting, where input distribution shift to the
Revised: This effect of similar categories being forgotten more can also be observed in the CIFAR-100 distribution shift setting, where input distribution shift to the

Original: CIFAR-100 superclasses results in more similar categories being forgotten more, e.g. input shifts to natural superclasses (Figure 8c) or artiﬁcal superclasses (Figure 8f).
Revised: CIFAR-100 superclasses results in more similar categories being forgotten more, e.g. input shifts to natural superclasses (Figure 8c) or artiﬁcal superclasses (Figure 8f).

Original: Further details and supporting experiments can be found in Appendix B.9.
Revised: Further details and supporting experiments can be found in Appendix B.9.

Original: In the subsequent sections, we resolve this puzzle through (i) the development of an analytic model that formalizes the subtle connection between task similarity and forgetting (ii) an ensuing measure of task similarity, trace overlap , empirically quantiﬁes task similarity (iii) showing maximal forgetting happens with intermediate task similarity.
Revised: In the subsequent sections, we resolve this puzzle through (i) the development of an analytic model that formalizes the subtle connection between task similarity and forgetting (ii) an ensuing measure of task similarity, trace overlap , empirically quantiﬁes task similarity (iii) showing maximal forgetting happens with intermediate task similarity.

Original: 6. A N A NALYTIC M ODEL OF F ORGETTING S EMANTICS
Revised: 6. A N A NALYTIC M ODEL OF F ORGETTING S EMANTICS

Original: We ﬁrst formalize a model with a single classiﬁcation head (like in distribution shift CIFAR-100), with the corresponding multi-head setup in Appendix C. We write the network output as f ( x )
Revised: We ﬁrst formalize a model with a single classiﬁcation head (like in distribution shift CIFAR-100), with the corresponding multi-head setup in Appendix C. We write the network output as f ( x )

Original: = (cid:80) pµ
Revised: = (cid:80) pµ

Original: =1 θ µ g µ ( w ; x ) , where θ are the last layer weights and g are features.
Revised: =1 θ µ g µ ( w ; x ) , where θ are the last layer weights and g are features.

Original: We consider training sequentially on two tasks.
Revised: We consider training sequentially on two tasks.

Original: For the ﬁrst task, we train normally.
Revised: For the ﬁrst task, we train normally.

Original: For the second task, inspired by our empirical observation above that forgetting is driven by deeper layers, we freeze the the earlier layer features, g µ ( w ; x ) =
Revised: For the second task, inspired by our empirical observation above that forgetting is driven by deeper layers, we freeze the the earlier layer features, g µ ( w ; x ) =

Original: g µ ( ˆ w ; x ) when training on the second task (here ˆ w represent the weights after training on the ﬁrst task).
Revised: g µ ( ˆ w ; x ) when training on the second task (here ˆ w represent the weights after training on the ﬁrst task).

Original: If we train with loss function L ( f, y ) and learning rate η , the network output SGD update at timestep t is
Revised: If we train with loss function L ( f, y ) and learning rate η , the network output SGD update at timestep t is

Original: [Equation]
Revised: [Equation]

Original: Here D (2) train is the second-task training data and Θ( x, x (cid:48) )
Revised: Here D (2) train is the second-task training data and Θ( x, x (cid:48) )

Original: = (cid:80) pµ
Revised: = (cid:80) pµ

Original: =1 g µ ( ˆ w ; x ) g µ ( ˆ w ; x (cid:48) ) is the overlap (inner product) between the model last-layer representations on data point x and x (cid:48) .
Revised: =1 g µ ( ˆ w ; x ) g µ ( ˆ w ; x (cid:48) ) is the overlap (inner product) between the model last-layer representations on data point x and x (cid:48) .

Original: If the overlap Θ( x, x (cid:48) ) is small, then the change in the logits is small and forgetting is minimal, speciﬁcally:
Revised: If the overlap Θ( x, x (cid:48) ) is small, then the change in the logits is small and forgetting is minimal, speciﬁcally:

Original: Lemma 1 Let f t be the logits of a neural network trained via the frozen feature model.
Revised: Lemma 1 Let f t be the logits of a neural network trained via the frozen feature model.

Original: Let x be an element of the initial task test set.
Revised: Let x be an element of the initial task test set.

Original: Let us further denote the vector of the feature overlap applied to the second task training set as (cid:126) Θ( x ) :
Revised: Let us further denote the vector of the feature overlap applied to the second task training set as (cid:126) Θ( x ) :

Original: =
Revised: =

Original: { Θ( x, x (cid:48) ) : x (cid:48) ∈ X (2) train } and the loss vector as (cid:126)L = { L ( f ( x (cid:48) ) , y (cid:48) ) :
Revised: { Θ( x, x (cid:48) ) : x (cid:48) ∈ X (2) train } and the loss vector as (cid:126)L = { L ( f ( x (cid:48) ) , y (cid:48) ) :

Original: x (cid:48) , y (cid:48)
Revised: x (cid:48) , y (cid:48)

Original: ∈ D (2) train } .
Revised: ∈ D (2) train } .

Original: With this, the change in the original task logits is bounded as 
Revised: With this, the change in the original task logits is bounded as | ∆

Original: [Equation]
Revised: [Equation]

Original: Thus representational similarity is necessary for forgetting to occur, and in this model, sufﬁciently similar and sufﬁciently dissimilar tasks have minimal forgetting, while intermediate similar tasks have maximal forgetting.
Revised: Thus representational similarity is necessary for forgetting to occur, and in this model, sufﬁciently similar and sufﬁciently dissimilar tasks have minimal forgetting, while intermediate similar tasks have maximal forgetting.

Original: 6. C ATASTROPHIC F ORGETTING ACROSS VARYING T ASK S IMILARITIES
Revised: 6. C ATASTROPHIC F ORGETTING ACROSS VARYING T ASK S IMILARITIES

Original: The analytical result enables us to make precise a measure of task similarity, through using the feature overlap matrix Θ( x (cid:48) , x ) .
Revised: The analytical result enables us to make precise a measure of task similarity, through using the feature overlap matrix Θ( x (cid:48) , x ) .

Original: Concretely, letting Θ 11 be the overlap matrix on Task 1, Θ 22 on Task 2, and Θ 12 on Task 1 and Task 2, we deﬁne: 
Revised: Concretely, letting Θ 11 be the overlap matrix on Task 1, Θ 22 on Task [Figure] and Θ 12 on Task 1 and Task 2, we deﬁne: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Trace overlap is directly analogous to CKA, where instead of comparing the similarity of different features on the same data, we are comparing the similarity of common features on different data.
Revised: Trace overlap is directly analogous to CKA, where instead of comparing the similarity of different features on the same data, we are comparing the similarity of common features on different data.

Original: Increasing task dissimilarity can help reduce forgetting In Lemma 1, we see that very dissimilar tasks, with small overlap matrix, exhibit minimal forgetting.
Revised: Increasing task dissimilarity can help reduce forgetting In Lemma [Figure] we see that very dissimilar tasks, with small overlap matrix, exhibit minimal forgetting.

Original: In Figure
Revised: In Figure

Original: [Figure]
Revised: [Figure]

Original: we test this in the context of sequential binary classiﬁcation.
Revised: we test this in the context of sequential binary classiﬁcation.

Original: We introduce an other category to the classiﬁcation problem (of images from classes not included in the training tasks)
Revised: We introduce an other category to the classiﬁcation problem (of images from classes not included in the training tasks)

Original: [Figure]
Revised: [Figure]

Original: which encourages the model to represent Task 1 and Task 2 dissimilarly (lower trace overlap in Figure 9b)
Revised: which encourages the model to represent Task 1 and Task 2 dissimilarly (lower trace overlap in Figure 9b)

Original: [Figure]
Revised: [Figure]

Original: and reduces forgetting (Figure 9a)
Revised: and reduces forgetting (Figure 9a)

Original: [Figure]
Revised: [Figure]

Original: Maximal forgetting at intermediate similarity To empirically study the conclusion from the model that intermediate-similar tasks have maximal forgetting, we use mixup (Zhang et al., 2017), a standard data augmentation technique, to interpolate between the two tasks.
Revised: Maximal forgetting at intermediate similarity To empirically study the conclusion from the model that intermediate-similar tasks have maximal forgetting, we use mixup (Zhang et al., 2017), a standard data augmentation technique, to interpolate between the two tasks.

Original: Speciﬁcally, we ﬁrst train a model on binary classiﬁcation (with an other category to encourage dissimilarity) and for λ ∈ [0 , 1] , generate an mixup dataset D ( λ ) train =
Revised: Speciﬁcally, we ﬁrst train a model on binary classiﬁcation (with an other category to encourage dissimilarity) and for λ ∈ [0 , 1] , generate an mixup dataset D ( λ ) train =

Original: { ( λx (2) +(1 − λ ) x (1) , λy (2) +(1 − λ ) y (1) ) } .
Revised: { ( λx (2) +(1 − λ ) x (1) , λy (2) +(1 − λ ) y (1) ) } .

Original: We compare model behavior across a range of mixup fractions, ﬁnding that the task similarity as measured through the trace overlap between task1 and task2 varies monotonically with mixup fraction (Figure 10b), quantifying  that varying the mixing parameter, λ , tunes the degree of task similarity.
Revised: We compare model behavior across a range of mixup fractions, ﬁnding that the task similarity as measured through the trace overlap between task1 and task2 varies monotonically with mixup fraction (Figure 10b), quantifying [Figure] that varying the mixing parameter, λ , tunes the degree of task similarity.

Original: Figure 10a shows that the most extreme forgetting happens for intermediate similarity (i.e. mixing fraction ∼ 0 .
Revised: Figure 10a shows that the most extreme forgetting happens for intermediate similarity (i e. mixing fraction ∼ 0 .

Original: 1 − 0 . 2 ).
Revised: 1 − 0 . 2 ).

Original: Semantic similarity revisited Armed with these quantitative measures, we revisit the task semantics puzzle in Figure 8 where (i) for sequential binary classiﬁcation, semantically more similar categories[Figure] were forgotten less, while in (ii) four class classiﬁcation followed by binary classiﬁcation and input distribution shift, similar classes were forgotten more.
Revised: Semantic similarity revisited Armed with these quantitative measures, we revisit the task semantics puzzle in Figure 8 where (i) for sequential binary classiﬁcation, semantically more similar categories were forgotten less, while in (ii) four class classiﬁcation followed by binary classiﬁcation and input distribution shift, similar classes were forgotten more.

Original: Figure 10c (solid dots) shows that in (i) the net didn’t represent objects and animals differently, and the slightly more similar task is forgotten less.
Revised: Figure 10c (solid dots) shows that in (i) the net didn’t represent objects and animals differently, and the slightly more similar task is forgotten less.

Original: But in (ii) Figure 10c (hollow dots), the model encodes animals and objects differently, minimizing forgetting on the dissimilar tasks, consistent with maximal forgetting for intermediate similarity.
Revised: But in (ii) Figure 10c (hollow dots), the model encodes animals and objects differently, minimizing forgetting on the dissimilar tasks, consistent with maximal forgetting for intermediate similarity.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have studied some of the fundamental properties of catastrophic forgetting, answering important open questions on how it arises and interacts with hidden representations, mitigation strategies and task semantics.
Revised: In this paper, we have studied some of the fundamental properties of catastrophic forgetting, answering important open questions on how it arises and interacts with hidden representations, mitigation strategies and task semantics.

Original: By using representational analysis techniques, we demonstrated that forgetting is not evenly distributed throughout the deep learning model but concentrated at the higher layers, which change signiﬁcantly and erase earlier task subspaces through sequential training.
Revised: By using representational analysis techniques, we demonstrated that forgetting is not evenly distributed throughout the deep learning model but concentrated at the higher layers, which change signiﬁcantly and erase earlier task subspaces through sequential training.

Original: Consistent with this, we ﬁnd mitigation methods all stabilize higher layer representations, but vary on whether they enforce more feature reuse, or store tasks in orthogonal subspaces.
Revised: Consistent with this, we ﬁnd mitigation methods all stabilize higher layer representations, but vary on whether they enforce more feature reuse, or store tasks in orthogonal subspaces.

Original: Informed by these insights, we formulate an analytic model to investigate connections between forgetting and task semantics.
Revised: Informed by these insights, we formulate an analytic model to investigate connections between forgetting and task semantics.

Original: This gives a quantitative measure of task similarity and shows that intermediate similarity in sequential tasks leads to maximal forgetting.
Revised: This gives a quantitative measure of task similarity and shows that intermediate similarity in sequential tasks leads to maximal forgetting.

Original: In summary, these results help provide a foundation for a deeper understanding of catastrophic forgetting and suggest new approaches for developing and measuring mitigation methods.
Revised: In summary, these results help provide a foundation for a deeper understanding of catastrophic forgetting and suggest new approaches for developing and measuring mitigation methods.

Original: A. T RAINING
Revised: A. T RAINING

Original: All networks are trained using cross-entropy loss with SGD with momentum ( β = 0.9), using a batch size 128.
Revised: All networks are trained using cross-entropy loss with SGD with momentum ( β = 0.9), using a batch size 128.

Original: We do not use learning-rate schedules here, leaving that investigation to future work.
Revised: We do not use learning-rate schedules here, leaving that investigation to future work.

Original: To better correspond with practical situations, however, we choose a learning rate and total number of epochs such that interpolation (of the training set) occurs.
Revised: To better correspond with practical situations, however, we choose a learning rate and total number of epochs such that interpolation (of the training set) occurs.

Original: For the split CIFAR-10 task, this typically corresponds to training for 30 epochs per task with a learning rate of 0.01 (VGG), 0.03 (ResNet), and 0.03 (DenseNet).
Revised: For the split CIFAR-10 task, this typically corresponds to training for 30 epochs per task with a learning rate of 0.01 (VGG), 0.03 (ResNet), and 0.03 (DenseNet).

Original: We use weight decay with strength 1e-4, and do not apply any data augmentation.
Revised: We use weight decay with strength 1e-4, and do not apply any data augmentation.

Original: For the split CIFAR-100 task, we usually train for 60 or 90 epochs per task, with other the other hyperparameters identical to the CIFAR-10 case.
Revised: For the split CIFAR-100 task, we usually train for 60 or 90 epochs per task, with other the other hyperparameters identical to the CIFAR-10 case.

Original: For multi-head networks (used in split CIFAR-10), each head is initialized with weights drawn from a normal distribution with variance 1 /n f , where n f is the number of features; biases are initialized to zero.
Revised: For multi-head networks (used in split CIFAR-10), each head is initialized with weights drawn from a normal distribution with variance 1 /n f , where n f is the number of features; biases are initialized to zero.

Original: We do not copy the parameters from the old head to the new when switching tasks.
Revised: We do not copy the parameters from the old head to the new when switching tasks.

Original: For the split CIFAR-10 setup, the plots shown in the main text refer to experiments in which the initial task was classifying between the ﬁve categories airplane , automobile , bird , cat , and deer ; and the second task comprising dog , frog , horse , ship , and truck .
Revised: For the split CIFAR-10 setup, the plots shown in the main text refer to experiments in which the initial task was classifying between the ﬁve categories airplane , automobile , bird , cat , and deer ; and the second task comprising dog , frog , horse , ship , and truck .

Original: To verify that our results were not unique to the particular split of CIFAR-10 we chose, we used three other splits: airplane , bird , deer , frog , ship (task 1) and automobile , cat , dog , horse , truck (task 2); automobile , cat , dog , horse , truck (task 1) and airplane , bird , deer , frog , ship (task 2); and dog , frog , horse , ship , truck (task 1) and airplane , automobile , bird , cat , deer (task 2).
Revised: To verify that our results were not unique to the particular split of CIFAR-10 we chose, we used three other splits: airplane , bird , deer , frog , ship (task 1) and automobile , cat , dog , horse , truck (task 2); automobile , cat , dog , horse , truck (task 1) and airplane , bird , deer , frog , ship (task 2); and dog , frog , horse , ship , truck (task 1) and airplane , automobile , bird , cat , deer (task 2).

Original: A. CIFAR-100 D ISTRIBUTION S HIFT T ASK
Revised: A. CIFAR-100 D ISTRIBUTION S HIFT T ASK

Original: Here we provide the concrete example of the CIFAR-100 distribution-shift task shown in Figure 1.
Revised: Here we provide the concrete example of the CIFAR-100 distribution-shift task shown in Figure 1.

Original: This is meant to capture it the scenario where catastrophic forgetting arises through input distribution shift — the input data to the neural network undergoes a distribution shift, causing the neural network to perform poorly on earlier data distributions (Arivazhagan et al., 2019;
Revised: This is meant to capture it the scenario where catastrophic forgetting arises through input distribution shift — the input data to the neural network undergoes a distribution shift, causing the neural network to perform poorly on earlier data distributions (Arivazhagan et al., 2019;

Original: Snoek et al., 2019; Rabanser et al., 2019).
Revised: Snoek et al., 2019; Rabanser et al., 2019).

Original: As mentioned in the main text, in this task, the model must identify, for CIFARimages, which of the 20 superclasses the image belongs to.
Revised: As mentioned in the main text, in this task, the model must identify, for CIFARimages, which of the 20 superclasses the image belongs to.

Original: The difference in tasks comes from the difference in subclasses which make up the superclass.
Revised: The difference in tasks comes from the difference in subclasses which make up the superclass.

Original: As an example, we take the ﬁve superclasses aquatic mammals , fruits and vegetables , household electrical devices , trees , and vehicles-1 1 , with the corresponding task 1 subclasses (1) dolphin, (2) apple, (3) lamp, (4) maple tree, and (5) bicycle and task 2 subclasses (1) whale, (2) orange, (3) television, (4) willow, and (5) motorcycle.
Revised: As an example, we take the ﬁve superclasses aquatic mammals , fruits and vegetables , household electrical devices , trees , and vehicles-1 1 , with the corresponding task 1 subclasses (1) dolphin, (2) apple, (3) lamp, (4) maple tree, and (5) bicycle and task 2 subclasses (1) whale, (2) orange, (3) television, (4) willow, and (5) motorcycle.

Original: A key feature of this setting is that task-speciﬁc components (including multiple heads) are precluded since the model does not know the task identity either at inference or time.
Revised: A key feature of this setting is that task-speciﬁc components (including multiple heads) are precluded since the model does not know the task identity either at inference or time.

Original: While we do not explore it here, this setup also allows for continuously varying the data distribution, another situation likely to occur in practice.
Revised: While we do not explore it here, this setup also allows for continuously varying the data distribution, another situation likely to occur in practice.

Original: A. R EPRESENTATIONAL S IMILARITY AND C ENTERED K ERNEL A LIGNMENT
Revised: A. R EPRESENTATIONAL S IMILARITY AND C ENTERED K ERNEL A LIGNMENT

Original: To understand properties of neural network hidden representations, we turn to representational similarity algorithms.
Revised: To understand properties of neural network hidden representations, we turn to representational similarity algorithms.

Original: A key challenge in analyzing neural network hidden representations is the lack of alignment — no natural correspondence exists between hidden neurons across different neural networks.
Revised: A key challenge in analyzing neural network hidden representations is the lack of alignment — no natural correspondence exists between hidden neurons across different neural networks.

Original: Representational similarity algorithms propose different ways to overcome this — one of the ﬁrst such algorithms, SVCCA (Raghu et al., 2017; Morcos et al., 2018), uses a Canonical
Revised: Representational similarity algorithms propose different ways to overcome this — one of the ﬁrst such algorithms, SVCCA (Raghu et al., 2017; Morcos et al., 2018), uses a Canonical

Original: Correlation Analysis (CCA) step to align neurons (enable invariance) through linear transformations.
Revised: Correlation Analysis (CCA) step to align neurons (enable invariance) through linear transformations.

Original: We use centered kernel alignment (CKA), proposed by Kornblith et al.
Revised: We use centered kernel alignment (CKA), proposed by Kornblith et al.

Original: , to measure the similarity between two representations of the same dataset which is invariant to orthogonal transformation and isotropic scaling (but not arbitrary linear transformations).
Revised: , to measure the similarity between two representations of the same dataset which is invariant to orthogonal transformation and isotropic scaling (but not arbitrary linear transformations).

Original: Given a dataset of m examples, we compare two representations X and Y of that dataset (say, from two different neural networks or from the same neural network with different parameters), with n x and n y features respectively; that is, X ∈ R m × n x and Y ∈ R m × n y .
Revised: Given a dataset of m examples, we compare two representations X and Y of that dataset (say, from two different neural networks or from the same neural network with different parameters), with n x and n y features respectively; that is, X ∈ R m × n x and Y ∈ R m × n y .

Original: Then the linear-kernel CKA similarity between the two representations is given by
Revised: Then the linear-kernel CKA similarity between the two representations is given by

Original: [Equation]
Revised: [Equation]

Original: A. S UBSPACE S IMILARITY
Revised: A. S UBSPACE S IMILARITY

Original: For the subspace similarity computations we pick a ﬁxed threshold k for all layers so that the ﬁrst k principal components capture 80% of variance in the ﬁnal layer.
Revised: For the subspace similarity computations we pick a ﬁxed threshold k for all layers so that the ﬁrst k principal components capture 80% of variance in the ﬁnal layer.

Original: This gives k = 3 for Resnet on
Revised: This gives k = 3 for Resnet on

Original: CIFAR-10, k = 4 for Resnet on CIFAR-100, k = 4 for VGG on CIFAR-10 and k = 7 for VGG on
Revised: CIFAR-10, k = 4 for Resnet on CIFAR-100, k = 4 for VGG on CIFAR-10 and k = 7 for VGG on

Original: CIFAR-100 and k = 4 for DenseNet on CIFAR-10 and k = 5 for DenseNet on CIFAR-100.
Revised: CIFAR-100 and k = 4 for DenseNet on CIFAR-10 and k = 5 for DenseNet on CIFAR-100.

Original: [Figure]
Revised: [Figure]

Original: a
Revised: c

Original: [Figure]
Revised: [Figure]

Original: S i m il a r i t y
Revised: S i m il a r i t y

Original: [Figure]
Revised: [Figure]

Original: A. M ITIGATION M ETHODS A.6. E LASTIC W EIGHT C ONSOLIDATION
Revised: A. M ITIGATION M ETHODS A.6. E LASTIC W EIGHT C ONSOLIDATION

Original: Developed by Kirkpatrick et al.
Revised: Developed by Kirkpatrick et al.

Original: in 2017, elastic weight consolidation (EWC) adds a term to the loss function of the new task to get the regularized loss function L ( (cid:126)θ ) :
Revised: in 2017, elastic weight consolidation (EWC) adds a term to the loss function of the new task to get the regularized loss function L ( (cid:126)θ ) :

Original: [Equation]
Revised: [Equation]

Original: Here, (cid:126)θ are the model parameters; L B ( (cid:126)θ ) is the unregularized loss function of the new task; λ is a parameter which controls the strength of the regularization; F i is the diagonal of the Fisher information matrix with respect to the parameters; and (cid:126)θ ∗ A are the model parameters after having been trained on task A.
Revised: Here, (cid:126)θ are the model parameters; L B ( (cid:126)θ ) is the unregularized loss function of the new task; λ is a parameter which controls the strength of the regularization; F i is the diagonal of the Fisher information matrix with respect to the parameters; and (cid:126)θ ∗ A are the model parameters after having been trained on task A.

Original: We compute the Fisher information via using the squared gradients of the log-likelihood, averaged over a subset of the trainset.
Revised: We compute the Fisher information via using the squared gradients of the log-likelihood, averaged over a subset of the trainset.

Original: We use the empirical Fisher as an approximation.
Revised: We use the empirical Fisher as an approximation.

Original: Following standard practice, in our experiments we estimate the Fisher information using a subset of the full training data; for all of our models on CIFAR-10, 200 samples proved sufﬁcient to be reasonably conﬁdent of a converged estimate.
Revised: Following standard practice, in our experiments we estimate the Fisher information using a subset of the full training data; for all of our models on CIFAR-10, 200 samples proved sufﬁcient to be reasonably conﬁdent of a converged estimate.

Original: We do not include a delay between the start of secondtask training and the application of the EWC penalty.
Revised: We do not include a delay between the start of secondtask training and the application of the EWC penalty.

Original: Naturally, we do not apply the EWC penalty to the parameters of the head in the multi-head setting.
Revised: Naturally, we do not apply the EWC penalty to the parameters of the head in the multi-head setting.

Original: A.6. S YNAPTIC I NTELLIGENCE
Revised: A.6. S YNAPTIC I NTELLIGENCE

Original: Developed by Zenke et al.
Revised: Developed by Zenke et al.

Original: (2017), synaptic intelligence, like EWC, is a structural regularization method which adds an elastic penalty to the second task’s loss function: 
Revised: (2017), synaptic intelligence, like EWC, is a structural regularization method which adds an elastic penalty to the second task’s loss function: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The difference from EWC is in how the elastic penalties are calculated.
Revised: The difference from EWC is in how the elastic penalties are calculated.

Original: In synaptic intelligence, for each parameter, a contribution to the loss is tracked via the product of the gradient of the loss and the stepwise change in the parameter value.
Revised: In synaptic intelligence, for each parameter, a contribution to the loss is tracked via the product of the gradient of the loss and the stepwise change in the parameter value.

Original: The coefﬁcient is then constructed such that changing the parameter by an amount equal to its change during task-1 training will incur a loss penalty equal to the change in loss from the initial value.
Revised: The coefﬁcient is then constructed such that changing the parameter by an amount equal to its change during task-1 training will incur a loss penalty equal to the change in loss from the initial value.

Original: Our implementation does not differ from the standard implementation.
Revised: Our implementation does not differ from the standard implementation.

Original: B S UPPORTING EXPERIMENTS
Revised: B S UPPORTING EXPERIMENTS

Original: Here we present experiments omitted from the main text for space, which help ﬂesh out some of the conclusions.
Revised: Here we present experiments omitted from the main text for space, which help ﬂesh out some of the conclusions.

Original: This section is roughly organized with layerwise experiments ﬁrst, followed by semantic experiments.
Revised: This section is roughly organized with layerwise experiments ﬁrst, followed by semantic experiments.

Original: B. L AYER RESET EXPERIMENTS
Revised: B. L AYER RESET EXPERIMENTS

Original: As described in the main text, we perform an additional experiment to check that deeper representations get corrupted more than shallower representations.
Revised: As described in the main text, we perform an additional experiment to check that deeper representations get corrupted more than shallower representations.

Original: This experiment, corresponding to
Revised: This experiment, corresponding to

Original: Figure 12, shows that when layers are reset to their pre-forgetting value, performance improves more dramatically when deeper layers are reset ﬁrst.
Revised: Figure 12, shows that when layers are reset to their pre-forgetting value, performance improves more dramatically when deeper layers are reset ﬁrst.

Original: B. T ASK - SPECIFIC STAGES
Revised: B. T ASK - SPECIFIC STAGES

Original: Our observations that forgetting is driven primarily by the higher layers suggests that in a setting with known task identities (i.e. in the split CIFAR-10 setting described in the main text but not the
Revised: Our observations that forgetting is driven primarily by the higher layers suggests that in a setting with known task identities (i.e. in the split CIFAR-10 setting described in the main text but not the

Original: CIFAR-100 distribution-shift setting), making the deeper stages of the network task-speciﬁc would likely mitigate a large portion of forgetting on the old tasks while still allowing for full performance on the new tasks.
Revised: CIFAR-100 distribution-shift setting), making the deeper stages of the network task-speciﬁc would likely mitigate a large portion of forgetting on the old tasks while still allowing for full performance on the new tasks.

Original: In particular, since the changes in deeper represntations are much greater than their shallower counterparts, only a few task-speciﬁc stages should be necessary to get signiﬁcant performance increases on the old task.
Revised: In particular, since the changes in deeper represntations are much greater than their shallower counterparts, only a few task-speciﬁc stages should be necessary to get signiﬁcant performance increases on the old task.

Original: For each of the three network architectures we considered
Revised: For each of the three network architectures we considered

Original: [Figure]
Revised: [Figure]

Original: Together, these results demonstrate that higher layers are disproportionately responsible for forgetting
Revised: Together, these results demonstrate that higher layers are disproportionately responsible for forgetting.

Original: (VGG, ResNet, and DenseNet), we investigated the performance of these task-speciﬁc networks on the split CIFAR task.
Revised: (VGG, ResNet, and DenseNet), we investigated the performance of these task-speciﬁc networks on the split CIFAR task.

Original: We intend these results to be further evidence of the role of deeper layers in forgetting, not a proposed mitigation scheme
Revised: We intend these results to be further evidence of the role of deeper layers in forgetting, not a proposed mitigation scheme

Original: [Figure]
Revised: [Figure]

Original: Results, shown in ﬁgure 13, show that for all architectures, performance dramatically rises with the number of task-speciﬁc stages.
Revised: Results, shown in ﬁgure 13, show that for all architectures, performance dramatically rises with the number of task-speciﬁc stages.

Original: For all architectures, having the deepest two stages be task-speciﬁc recovers a signiﬁcant fraction of the forgotten task performance.
Revised: For all architectures, having the deepest two stages be task-speciﬁc recovers a signiﬁcant fraction of the forgotten task performance.

Original: B. L AYER RESET AND RETRAIN EXPERIMENTS
Revised: B. L AYER RESET AND RETRAIN EXPERIMENTS

Original: As an additional probe of the degree to which catastrophic forgetting impacts the representations of various network layers, we perform a reset-and-retrain experiment on the split CIFAR-10 task, using the same training setup as in Figure 3 of the main text.
Revised: As an additional probe of the degree to which catastrophic forgetting impacts the representations of various network layers, we perform a reset-and-retrain experiment on the split CIFAR-10 task, using the same training setup as in Figure 3 of the main text.

Original: The reset-and-retrain experiment is designed to probe the degree to which network representations get corrupted and can no longer be used by a network of the same architecture to productively classify images.
Revised: The reset-and-retrain experiment is designed to probe the degree to which network representations get corrupted and can no longer be used by a network of the same architecture to productively classify images.

Original: To investigate this, after having trained the network on Task 2, we freeze parameters of stages 1 through N to their post-taskvalues, reset the parameters of stages N + 1 onwards to their post-task-1 values, and then retrain those parameters on the Task 1 loss function.
Revised: To investigate this, after having trained the network on Task 2, we freeze parameters of stages 1 through N to their post-taskvalues, reset the parameters of stages N + 1 onwards to their post-task-1 values, and then retrain those parameters on the Task 1 loss function.

Original: This is a natural, operational measure of the degree to which layer 1 through N representations have been corrupted.
Revised: This is a natural, operational measure of the degree to which layer 1 through N representations have been corrupted.

Original: The results of this experiment, shown in Figure 14, tell a similar story to that of Figure ??
Revised: The results of this experiment, shown in Figure 14, tell a similar story to that of Figure ??

Original: of the main text.
Revised: of the main text.

Original: In particular, we see even retraining the network with all but the last couple of layers frozen recovers nearly the pre-forgetting performance of the network.
Revised: In particular, we see even retraining the network with all but the last couple of layers frozen recovers nearly the pre-forgetting performance of the network.

Original: Some of the performances in Figure 14 even outperform the pre-forgetting network performance, suggesting that network representations of intermediate layers may actually be improved by training on Task 2.
Revised: Some of the performances in Figure 14 even outperform the pre-forgetting network performance, suggesting that network representations of intermediate layers may actually be improved by training on Task 2.

Original: Further suggestive results are shown in Figure 15, in which we retrain the full network again on Task 1 after the training on Task 2.
Revised: Further suggestive results are shown in Figure 15, in which we retrain the full network again on Task 1 after the training on Task 2.

Original: For ResNet and, to some degree, VGG, training on the second task clearly improves performance on the ﬁrst.
Revised: For ResNet and, to some degree, VGG, training on the second task clearly improves performance on the ﬁrst.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: the model only exposed to task-0 data.
Revised: model only exposed to task-0 data.

Original: This effect is consistent across different splits of the original dataset.
Revised: This effect is consistent across different splits of the original dataset.

Original: Of the architectures we studied, this effect is most pronounced for ResNet, but also exists to some degree for VGG and DenseNet models.
Revised: Of the architectures we studied, this effect is most pronounced for ResNet, but also exists to some degree for VGG and DenseNet models.

Original: B. R EPRESENTATION CHANGES IN THE CIFAR-100 D ISTRIBUTION SHIFT EXPERIMENTS
Revised: B. R EPRESENTATION CHANGES IN THE CIFAR-100 D ISTRIBUTION SHIFT EXPERIMENTS

Original: Here we show the change in representations due to training on distribution-shifted CIFAR-100 for the VGG (Figure 16) and DenseNet (Figure 17) networks, to accompany the ResNet results in the main text.
Revised: Here we show the change in representations due to training on distribution-shifted CIFAR-100 for the VGG (Figure 16) and DenseNet (Figure 17) networks, to accompany the ResNet results in the main text.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  i
Revised: S i

Original: [Figure]
Revised: [Figure]

Original: il 
Revised: il a

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: B. A DDITIONAL E XPERIMENTS FOR S UBSPACE S IMILARITY OF M ITIGATION M ETHODS
Revised: B. A DDITIONAL E XPERIMENTS FOR S UBSPACE S IMILARITY OF M ITIGATION M ETHODS

Original: We include additional experiments on computing subspace similarities for different mitigation methods across different architectures.
Revised: We include additional experiments on computing subspace similarities for different mitigation methods across different architectures.

Original: The results support the orthogonal storage of Task 1, Taskrepresentations by replay, while EWC and SI enforce greater feature reuse in the higher layers. 
Revised: The results support the orthogonal storage of Task 1, Taskrepresentations by replay, while EWC and SI enforce greater feature reuse in the higher layers. [Figure] t

Original: [Figure]
Revised: [Figure]

Original: B. L INEAR REGRESSION ON FORGOTTEN NETWORK ACTIVATIONS
Revised: B. L INEAR REGRESSION ON FORGOTTEN NETWORK ACTIVATIONS

Original: To further probe the manner in which catastrophic forgetting affects the internal representation of networks, we perform the following experiment (in the split CIFAR-10 setting).
Revised: To further probe the manner in which catastrophic forgetting affects the internal representation of networks, we perform the following experiment (in the split CIFAR-10 setting).

Original: We train a (multihead) model on the ﬁrst task for some time, resulting in model M 1 , and then train on the second task for some time, resulting in model M 2 .
Revised: We train a (multihead) model on the ﬁrst task for some time, resulting in model M 1 , and then train on the second task for some time, resulting in model M 2 .

Original: Due to forgetting, the model M 2 of course performs signiﬁcantly worse than does M 1 on the ﬁrst task.
Revised: Due to forgetting, the model M 2 of course performs signiﬁcantly worse than does M 1 on the ﬁrst task.

Original: But, to probe how much information has been lost internally, we train a linear model to classify between the ﬁrst-task categories using only the internal activations of M 2 on the ﬁrst-task images.
Revised: But, to probe how much information has been lost internally, we train a linear model to classify between the ﬁrst-task categories using only the internal activations of M 2 on the ﬁrst-task images.

Original: We measure both the amount of performance which we can recover using this linear regression, and the weight placed by the linear model on various stages of the network.
Revised: We measure both the amount of performance which we can recover using this linear regression, and the weight placed by the linear model on various stages of the network.

Original: Model performances are given in Table 2.
Revised: Model performances are given in Table 2.

Original: For all three architectures, the performance of the model before forgetting is naturally the highest, with a performance drop of up to forty percent due to forgetting.
Revised: For all three architectures, the performance of the model before forgetting is naturally the highest, with a performance drop of up to forty percent due to forgetting.

Original: Remarkably, however, training a linear model on the internal post-forgetting activations recovers a substantial fraction of the performance loss: for ResNet and VGG, the linear model on post-forgetting activations is only a percent less accurate than the model before forgetting.
Revised: Remarkably, however, training a linear model on the internal post-forgetting activations recovers a substantial fraction of the performance loss: for ResNet and VGG, the linear model on post-forgetting activations is only a percent less accurate than the model before forgetting.

Original: As a control, we also train a linear model on the activations of networks at initialization (denoted the
Revised: As a control, we also train a linear model on the activations of networks at initialization (denoted the

Original: [Table]
Revised: [Table]

Original: random lift in the table).
Revised: random lift in the table).

Original: Because the number of activations is much higher than the number of pixels in a CIFAR-10 image, this control is necessary to ensure that the boost in performance we see is not simply due to performing a high-dimensional lift which renders the data linearly separable.
Revised: Because the number of activations is much higher than the number of pixels in a CIFAR-10 image, this control is necessary to ensure that the boost in performance we see is not simply due to performing a high-dimensional lift which renders the data linearly separable.

Original: As the table shows, however, this random lift performance is quite poor.
Revised: As the table shows, however, this random lift performance is quite poor.

Original: Figure 19 shows the weights placed by the linear model on the various internal activations of the network.
Revised: Figure 19 shows the weights placed by the linear model on the various internal activations of the network.

Original: We use the activations both before and after forgetting.
Revised: We use the activations both before and after forgetting.

Original: Before forgetting, a linear model trained on the activations places most importance on the ﬁnal layers, as expected; however, after forgetting, the model learns to use information stored in earlier-layer representations of the network.
Revised: Before forgetting, a linear model trained on the activations places most importance on the ﬁnal layers, as expected; however, after forgetting, the model learns to use information stored in earlier-layer representations of the network.

Original: The reduced weighting of the ﬁnal layers is consistent with the experiments we describe in the main text showing that deeper representations suffer the most from forgetting.
Revised: The reduced weighting of the ﬁnal layers is consistent with the experiments we describe in the main text showing that deeper representations suffer the most from forgetting.

Original: [Figure]
Revised: [Figure]

Original: L i n e a r w e i g
Revised: L i n e a r w e i g

Original: h t
Revised: h t

Original: h
Revised: h

Original: i s t
Revised: i s t

Original: o g r a m
Revised: o g r a m

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: r w e i g h t
Revised: r w e i g h t

Original: h i s t
Revised: h i s t

Original: o g r a m
Revised: o g r a m

Original: VGG: After forgetting    8 10 12 14 16 Layer 0.
Revised: VGG: After forgetting    8 10 12 14 16 Layer 0.

Original: ResNet: After forgetting   8 Layer 0.
Revised: ResNet: After forgetting   8 Layer 0.

Original: DenseNet:
Revised: DenseNet:

Original: After forgetting
Revised: After forgetting

Original: B. CKA MEASUREMENTS ON MITIGATION METHODS
Revised: B. CKA MEASUREMENTS ON MITIGATION METHODS

Original: This section gives CKA similarity measurements to accompany those in the main text presented in ﬁgure 5.
Revised: This section gives CKA similarity measurements to accompany those in the main text presented in ﬁgure 5.

Original: The main text ﬁgure showed results for ResNet; here we show corresponding results for all architectures on both split CIFAR10 and split CIFAR100 tasks, in ﬁgures 20 and 21, repsectively.
Revised: The main text ﬁgure showed results for ResNet; here we show corresponding results for all architectures on both split CIFAR10 and split CIFAR100 tasks, in ﬁgures 20 and 21, repsectively.

Original: These plots show that the results discussed in the main text apply broadly across architectures and datasets.
Revised: These plots show that the results discussed in the main text apply broadly across architectures and datasets.

Original: B. O THER CATEGORY IN R ES N ET AND D ENSE N ET
Revised: B. O THER CATEGORY IN R ES N ET AND D ENSE N ET

Original: In Figure 9 of the main text, we showed for the VGG architecture that adding an ‘other’ category (by sampling from all images in the dataset which are not in either of the main tasks) can mitigate forgetting by reducing the similarity between the model’s representations of the task 1 data versus[Table] [Table] [Table]
Revised: In Figure 9 of the main text, we showed for the VGG architecture that adding an ‘other’ category (by sampling from all images in the dataset which are not in either of the main tasks) can mitigate forgetting by reducing the similarity between the model’s representations of the task 1 data versus

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: C K A S
Revised: C K A S

Original: i m il a r i t
Revised: i m il a r i t

Original: y
Revised: y

Original: [Figure]
Revised: [Figure]

Original: [Table][Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: K A S i m il a r i t y
Revised: A S i m il a r i t y

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure][Table]
Revised: [Figure] [Figure]

Original: the task 2 data.
Revised: the task 2 data.

Original: As our analytic model suggested, increasing this dissimilarity should help forgetting.
Revised: As our analytic model suggested, increasing this dissimilarity should help forgetting.

Original: Figures 22 and 23 show corresponding plots for the Resnet and Densenet architectures.
Revised: Figures 22 and 23 show corresponding plots for the Resnet and Densenet architectures.

Original: B. A DDITIONAL SEMANTIC EXPERIMENTS
Revised: B. A DDITIONAL SEMANTIC EXPERIMENTS

Original: In this section, comprising ﬁgures 24, 25, and 26, we present additional realizations of the semantic experiments presented in Section 6, Figure 8.
Revised: In this section, comprising ﬁgures 24, 25, and 26, we present additional realizations of the semantic experiments presented in Section 6, Figure 8.

Original: In particular we present results for sequential binary classiﬁcation tasks, four-class classiﬁcation followed by two-class classiﬁcation, and the CIFARdistribution shift task.
Revised: In particular we present results for sequential binary classiﬁcation tasks, four-class classiﬁcation followed by two-class classiﬁcation, and the CIFARdistribution shift task.

Original: We ﬁnd consistent results across different choices of objects and animals and across VGG, ResNet, and DenseNet architectures.
Revised: We ﬁnd consistent results across different choices of objects and animals and across VGG, ResNet, and DenseNet architectures.

Original: In particular, we ﬁnd that when models are encouraged to distinguish objects and animals, dissimilar categories lead to less forgetting.
Revised: In particular, we ﬁnd that when models are encouraged to distinguish objects and animals, dissimilar categories lead to less forgetting.

Original: In contrast when models are trained with only a single semantic category, there is no pressure to distinguish objects from animals and similar categories lead to less forgetting.
Revised: In contrast when models are trained with only a single semantic category, there is no pressure to distinguish objects from animals and similar categories lead to less forgetting.

Original: We observe one exception to the intuitive semantic groupings of categories.
Revised: We observe one exception to the intuitive semantic groupings of categories.

Original: For VGG in Setup 2 (Figure 25) the truck category behaves similarly to animal classes in that the performance suffers more when the second task is animal classiﬁcation then when it is object classiﬁcation.
Revised: For VGG in Setup 2 (Figure 25) the truck category behaves similarly to animal classes in that the performance suffers more when the second task is animal classiﬁcation then when it is object classiﬁcation.

Original: C F ROZEN - FEATURE MODEL : FURTHER DETAILS
Revised: C F ROZEN - FEATURE MODEL : FURTHER DETAILS

Original: Here we consider extensions and applications of the analytic model introduced in Section 6.2.
Revised: Here we consider extensions and applications of the analytic model introduced in Section 6.2.

Original: Both the single and multi-head models relate the change in predictions during training on a second task to
Revised: Both the single and multi-head models relate the change in predictions during training on a second task to

Original: [Figure]
Revised: [Figure]

Original: the overlap between features, Θ( x, x (cid:48) )
Revised: the overlap between features, Θ( x, x (cid:48) )

Original: = (cid:80) µ g µ ( x ) g µ ( x (cid:48) ) , evaluated on the initial and second task data.
Revised: = (cid:80) µ g µ ( x ) g µ ( x (cid:48) ) , evaluated on the initial and second task data.

Original: Multi-head model
Revised: Multi-head model

Original: In the main text we focused for simplicity on an analytic model of single head forgetting.
Revised: In the main text we focused for simplicity on an analytic model of single head forgetting.

Original: Here we construct a solvable model for the multi-head setup.
Revised: Here we construct a solvable model for the multi-head setup.

Original: We consider a model
Revised: We consider a model

Original: [Figure]
Revised: [Figure]

Original: consisting of non-linear features g ( w ; x ) , a linear layer with weights, θ , and a read out head h
Revised: consisting of non-linear features g ( w ; x ) , a linear layer with weights, θ , and a read out head h

Original: ( i ) .
Revised: ( i ) .

Original: [Equation]
Revised: [Equation]

Original: Here µ runs over the feature dimensions, while a runs over the output dimensions of our additional linear layer.
Revised: Here µ runs over the feature dimensions, while a runs over the output dimensions of our additional linear layer.

Original: For the initial task we use the head h (1) , while for the second task, we swap the head and use h (2) .
Revised: For the initial task we use the head h (1) , while for the second task, we swap the head and use h (2) .

Original: Again we consider a model trained without restrictions on Task 1 using head h (1) .
Revised: Again we consider a model trained without restrictions on Task 1 using head h (1) .

Original: For Task 2, we ﬁrst swap the head and then perform head only training on head h (2) .
Revised: For Task 2, we ﬁrst swap the head and then perform head only training on head h (2) .

Original: After training the head, we
Revised: After training the head, we

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: [Figure]

Original: freeze the head, h (2) , and the features g ( w ; x )
Revised: freeze the head, h (2) , and the features g ( w ; x )

Original: =
Revised: =

Original: g ( ˆ w ; x ) , where ˆ w is the value of the feature weights after training on the initial task.
Revised: g ( ˆ w ; x ) , where ˆ w is the value of the feature weights after training on the initial task.

Original: This model is again inspired by our observation that forgetting is driven by changes in the latter network layers.
Revised: This model is again inspired by our observation that forgetting is driven by changes in the latter network layers.

Original: The multi head model is additionally inspired by our observing that after head ﬁrst training, the CKA similarity between the second task head before and after second task training is relatively high (see Table 3).
Revised: The multi head model is additionally inspired by our observing that after head ﬁrst training, the CKA similarity between the second task head before and after second task training is relatively high (see Table 3).

Original: After training the second task head, we continue training the weights θ .
Revised: After training the second task head, we continue training the weights θ .

Original: The model output evolves as
Revised: The model output evolves as

Original: [Equation]
Revised: [Equation]

Original: Again we ﬁnd that if the representation overlap matrix Θ( x, x (cid:48) ) = (cid:80) µ g µ ( ˆ w ; x ) g µ ( ˆ w ; x (cid:48) ) is small between the features evaluated on the initial task and second task data then the predictions do not change signiﬁcantly.
Revised: Again we ﬁnd that if the representation overlap matrix Θ( x, x (cid:48) ) = (cid:80) µ g µ ( ˆ w ; x ) g µ ( ˆ w ; x (cid:48) ) is small between the features evaluated on the initial task and second task data then the predictions do not change signiﬁcantly.

Original: If this overlap is zero, then the predictions are constant.
Revised: If this overlap is zero, then the predictions are constant.

Original: In the multi-head setup we are considering here, we see that the change in predictions is further proportional to the similarity between the model heads, h T ( j ) h ( i ) .
Revised: In the multi-head setup we are considering here, we see that the change in predictions is further proportional to the similarity between the model heads, h T ( j ) h ( i ) .

Original: Finally, we note that we have considered a ﬁnal linear layer before the readout head for simplicity.
Revised: Finally, we note that we have considered a ﬁnal linear layer before the readout head for simplicity.

Original: One can instead include a ReLU non-linearity without changing the essential point, that the change in model output is governed by the overlap matrix, Θ .
Revised: One can instead include a ReLU non-linearity without changing the essential point, that the change in model output is governed by the overlap matrix, Θ .

Original: Rotating representations in the analytic model.
Revised: Rotating representations in the analytic model.

Original: We can use our analytic model (Section 6.2) to investigate how forgetting depends on representation similarity.
Revised: We can use our analytic model (Section 6.2) to investigate how forgetting depends on representation similarity.

Original: We again consider sequential binary tasks (car vs plane) followed by (cat vs horse).
Revised: We again consider sequential binary tasks (car vs plane) followed by (cat vs horse).

Original: We then tune the overlap of our initial task and ﬁnal task features by explicitly rotating the frozen features for the second task to produce new features g (cid:48)
Revised: We then tune the overlap of our initial task and ﬁnal task features by explicitly rotating the frozen features for the second task to produce new features g (cid:48)

Original: ( θ ; ˆ w ; x ) .
Revised: ( θ ; ˆ w ; x ) .

Original: [Equation]
Revised: [Equation]

Original: Here, if we have P features, R ( θ ) ∈ R P
Revised: Here, if we have P features, R ( θ ) ∈ R P

Original: × P is a one-parameter family of rotation matrices designed such that R (0)
Revised: × P is a one-parameter family of rotation matrices designed such that R (0)

Original: = 

Revised: =

Original: and g T ( ˆ w ; x ) g (cid:48)
Revised: and g T ( ˆ w ; x ) g (cid:48)

Original: ( π/ 2; ˆ w ; x (cid:48) ) for x ∈ X (1) test and x (cid:48)
Revised: ( π/ 2; ˆ w ; x (cid:48) ) for x ∈ X (1) test and x (cid:48)

Original: ∈ X (2) train is small.
Revised: ∈ X (2) train is small.

Original: Explicitly, we take
Revised: Explicitly, we take

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Effect of rotating representation
Revised: rotating representation

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: where r ij ( θ ) is the two dimensional rotation matrix between axes i and j , and V is the orthogonal matrix appearing in the singular value decomposition of the feature by data matrix on the initial task, g
Revised: where r ij ( θ ) is the two dimensional rotation matrix between axes i and j , and V is the orthogonal matrix appearing in the singular value decomposition of the feature by data matrix on the initial task, g

Original: ( X (1) test )
Revised: ( X (1) test )

Original: =
Revised: =

Original: UDV .
Revised: UDV .

Original: We ﬁnd that explicitly enforcing task dissimilarity via a large rotation minimizes the effect of forgetting (Figure 27).
Revised: We ﬁnd that explicitly enforcing task dissimilarity via a large rotation minimizes the effect of forgetting (Figure 27).

Original: Stopping forgetting without stabilizing weights We saw in Section 4 that both EWC and replay buffers stabilize deeper network representations.
Revised: Stopping forgetting without stabilizing weights We saw in Section 4 that both EWC and replay buffers stabilize deeper network representations.

Original: As mentioned above, this need not be the case a priori.
Revised: As mentioned above, this need not be the case a priori.

Original: As an illustrative example, we construct a setup where sequential training leads to no forgetting despite a signiﬁcant change to ﬁnal layer weights.
Revised: As an illustrative example, we construct a setup where sequential training leads to no forgetting despite a signiﬁcant change to ﬁnal layer weights.

Original: We consider the single head model of Section 6. where the Task 1 and Task 2 representations are completely orthogonal, Θ( x, x (cid:48) )
Revised: We consider the single head model of Section 6. where the Task 1 and Task 2 representations are completely orthogonal, Θ( x, x (cid:48) )

Original: = 

Revised: =

Original: for x ∈ X (1) test and x (cid:48) ∈ X (2) train .
Revised: for x ∈ X (1) test and x (cid:48) ∈ X (2) train .

Original: In this case the Task 1 predictions remain constant throughout all of Task 2 training, despite signiﬁcant changes to the ﬁnal layer weights and predictions on the second task.
Revised: In this case the Task 1 predictions remain constant throughout all of Task 2 training, despite signiﬁcant changes to the ﬁnal layer weights and predictions on the second task.

Original: In Figurewe present an example of this setup.
Revised: In Figurewe present an example of this setup.

Original: The representations are again made orthogonal by explicitly rotating the second task features by the rotation matrix, R ( π/ 2) deﬁned in Equation equation 11.
Revised: The representations are again made orthogonal by explicitly rotating the second task features by the rotation matrix, R ( π/ 2) deﬁned in Equation equation 11.

Original: D A DDITIONAL EXPERIMENTS
Revised: D A DDITIONAL EXPERIMENTS

Original: This section presents two additional experiments which are slightly unrelated to the main thrust of the paper.
Revised: This section presents two additional experiments which are slightly unrelated to the main thrust of the paper.

Original: We examine how the degree to which a network forgets is inﬂuenced by its width, and also how training the head independently of the rest of the network, in a multi-head setting, can mitigate forgetting.
Revised: We examine how the degree to which a network forgets is inﬂuenced by its width, and also how training the head independently of the rest of the network, in a multi-head setting, can mitigate forgetting.

Original: D. F ORGETTING VERSUS NETWORK WIDTH
Revised: D. F ORGETTING VERSUS NETWORK WIDTH

Original: [Figure]
Revised: [Figure]

Original: Inspired by recent work of Gilboa & Gur-Ari (2019) showing that wider networks can learn better features than their narrower counterparts
Revised: Inspired by recent work of Gilboa & Gur-Ari (2019) showing that wider networks can learn better features than their narrower counterparts

Original: [Figure]
Revised: [Figure]

Original: we sought to determine whether wider networks also exhibit less catastrophic forgetting than narrower ones.
Revised: we sought to determine whether wider networks also exhibit less catastrophic forgetting than narrower ones.

Original: We tested forgetting versus width for VGG,
Revised: We tested forgetting versus width for VGG,

Original: DenseNet
Revised: DenseNet

Original: [Figure]
Revised: [Figure]

Original: and ResNet networks on the split CIFAR-10 task in the multi-head setting.
Revised: and ResNet networks on the split CIFAR-10 task in the multi-head setting.

Original: For VGG and
Revised: For VGG and

Original: ResNet, the width was varied by changing the number of channels in the initial convolutional layer and scaling the rest of the network layers concurrently while maintaining the network shape; for
Revised: ResNet, the width was varied by changing the number of channels in the initial convolutional layer and scaling the rest of the network layers concurrently while maintaining the network shape; for

Original: DenseNet, the width was varied by by changing the growth rate.
Revised: DenseNet, the width was varied by by changing the growth rate.

Original: Surprisingly, though we found that the performances of the networks on each task improved as they grew wider, the amount of forgetting in each network, measured by the percent drop in accuracy (on task 1) due to training on task, exhibited a minimal dependence on network width.
Revised: Surprisingly, though we found that the performances of the networks on each task improved as they grew wider, the amount of forgetting in each network, measured by the percent drop in accuracy (on task 1) due to training on task, exhibited a minimal dependence on network width.

Original: These results are shown in Figure 29.
Revised: These results are shown in Figure 29.

Original: D. H EADFIRST TRAINING
Revised: D. H EADFIRST TRAINING

Original: When training multi-head networks on sequences of tasks, we found that the networks suffered a smaller performance drop on the original task, i.e. forgot less, if only the new task head was trained for a few epochs prior to training the full network.
Revised: When training multi-head networks on sequences of tasks, we found that the networks suffered a smaller performance drop on the original task, i.e. forgot less, if only the new task head was trained for a few epochs prior to training the full network.

Original: In Figure 30, this effect is shown for all three network architectures on the split CIFAR-10 task.
Revised: In Figure 30, this effect is shown for all three network architectures on the split CIFAR-10 task.

Original: Training only the head for up to ﬁve epochs seemed to improve the original-task performance without sacriﬁcing performance on the new task.
Revised: Training only the head for up to ﬁve epochs seemed to improve the original-task performance without sacriﬁcing performance on the new task.

Original: This suggests that coadaptation between the freshly-initialized head and the rest of the model contributes to forgetting when there is no headﬁrst training.
Revised: This suggests that coadaptation between the freshly-initialized head and the rest of the model contributes to forgetting when there is no headﬁrst training.

Original: [Figure]
Revised: [Figure]

Original: T a s k 1 A cc u r a c y
Revised: T a s k 1 A cc u r a c y

Original: E F IGURES FOR A NON R EVIEWERF I MAGE N ET R ESULTS FOR T ASK S EMANTICS
Revised: E F IGURES FOR A NON R EVIEWERF I MAGE N ET R ESULTS FOR T ASK S EMANTICS

Original: Figures corresponding to results tables in the main text.
Revised: Figures corresponding to results tables in the main text.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]