Original: C ONTINUAL L EARNING IN R ECURRENT N EURAL N ETWORKS
Revised: C ONTINUAL L EARNING IN R ECURRENT N EURAL N ETWORKS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While a diverse collection of continual learning (CL) methods has been proposed to prevent catastrophic forgetting, a thorough investigation of their effectiveness for processing sequential data with recurrent neural networks (RNNs) is lacking.
Revised: While a diverse collection of continual learning (CL) methods has been proposed to prevent catastrophic forgetting, a thorough investigation of their effectiveness for processing sequential data with recurrent neural networks (RNNs) is lacking.

Original: Here, we provide the ﬁrst comprehensive evaluation of established CL methods on a variety of sequential data benchmarks.
Revised: Here, we provide the ﬁrst comprehensive evaluation of established CL methods on a variety of sequential data benchmarks.

Original: Speciﬁcally, we shed light on the particularities that arise when applying weight-importance methods, such as elastic weight consolidation, to RNNs.
Revised: Speciﬁcally, we shed light on the particularities that arise when applying weight-importance methods, such as elastic weight consolidation, to RNNs.

Original: In contrast to feedforward networks, RNNs iteratively reuse a shared set of weights and require working memory to process input samples.
Revised: In contrast to feedforward networks, RNNs iteratively reuse a shared set of weights and require working memory to process input samples.

Original: We show that the performance of weight-importance methods is not directly affected by the length of the processed sequences, but rather by high working memory requirements, which lead to an increased need for stability at the cost of decreased plasticity for learning subsequent tasks.
Revised: We show that the performance of weight-importance methods is not directly affected by the length of the processed sequences, but rather by high working memory requirements, which lead to an increased need for stability at the cost of decreased plasticity for learning subsequent tasks.

Original: We additionally provide theoretical arguments supporting this interpretation by studying linear RNNs.
Revised: We additionally provide theoretical arguments supporting this interpretation by studying linear RNNs.

Original: Our study shows that established CL methods can be successfully ported to the recurrent case, and that a recent regularization approach based on hypernetworks outperforms weight-importance methods, thus emerging as a promising candidate for CL in RNNs.
Revised: Our study shows that established CL methods can be successfully ported to the recurrent case, and that a recent regularization approach based on hypernetworks outperforms weight-importance methods, thus emerging as a promising candidate for CL in RNNs.

Original: Overall, we provide insights on the differences between CL in feedforward networks and RNNs, while guiding towards effective solutions to tackle CL on sequential data.
Revised: Overall, we provide insights on the differences between CL in feedforward networks and RNNs, while guiding towards effective solutions to tackle CL on sequential data.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The ability to continually learn from a non-stationary data distribution while transferring and protecting past knowledge is known as continual learning (CL).
Revised: The ability to continually learn from a non-stationary data distribution while transferring and protecting past knowledge is known as continual learning (CL).

Original: This ability requires neural networks to be stable to prevent forgetting, but also plastic to learn novel information, which is referred to as the stability-plasticity dilemma (Grossberg, 2007; Mermillod et al., 2013).
Revised: This ability requires neural networks to be stable to prevent forgetting, but also plastic to learn novel information, which is referred to as the stability-plasticity dilemma (Grossberg, 2007; Mermillod et al., 2013).

Original: To address this dilemma, a variety of methods which tackle CL for static data with feedforward networks have been proposed (for reviews refer to Parisi et al.
Revised: To address this dilemma, a variety of methods which tackle CL for static data with feedforward networks have been proposed (for reviews refer to Parisi et al.

Original: and van de Ven and Tolias (2019)).
Revised: and van de Ven and Tolias (2019)).

Original: However, CL for sequential data has only received little attention, despite recent work conﬁrming that recurrent neural networks (RNNs) also suffer from catastrophic forgetting (Schak and Gepperth, 2019).
Revised: However, CL for sequential data has only received little attention, despite recent work conﬁrming that recurrent neural networks (RNNs) also suffer from catastrophic forgetting (Schak and Gepperth, 2019).

Original: A set of methods that holds great promise to address this problem are regularization methods, which work by constraining the update of certain parameters.
Revised: A set of methods that holds great promise to address this problem are regularization methods, which work by constraining the update of certain parameters.

Original: These methods can be considered more versatile than competing approaches, since they do not require rehearsal of past data, nor an increase in model capacity, but can beneﬁt from either of the two (e.g., Nguyen et al., 2018; Yoon et al., 2018).
Revised: These methods can be considered more versatile than competing approaches, since they do not require rehearsal of past data, nor an increase in model capacity, but can beneﬁt from either of the two (e.g., Nguyen et al., 2018; Yoon et al., 2018).

Original: This makes regularization methods applicable to a broader variety of situations, e.g. when issues related to data privacy, storage, or limited computational resources during inference might arise.
Revised: This makes regularization methods applicable to a broader variety of situations, e.g. when issues related to data privacy, storage, or limited computational resources during inference might arise.

Original: The most well-known regularization methods are weight-importance methods , such as elastic weight consolidation (EWC, Kirkpatrick et al.
Revised: The most well-known regularization methods are weight-importance methods , such as elastic weight consolidation (EWC, Kirkpatrick et al.

Original: (2017a))
Revised: (2017a))

Original: and synaptic intelligence (SI, Zenke et al.
Revised: and synaptic intelligence (SI, Zenke et al.

Original: (2017)), which are based on assigning importance values to weights.
Revised: (2017)), which are based on assigning importance values to weights.

Original: Some of these have a direct probabilistic interpretation as prior-focused CL methods (Farquhar and Gal, 2018), for which solutions of upcoming tasks must lie in the posterior parameter distribution of the current task (cf.
Revised: Some of these have a direct probabilistic interpretation as prior-focused CL methods (Farquhar and Gal, 2018), for which solutions of upcoming tasks must lie in the posterior parameter distribution of the current task (cf.

Original: Fig. 1b), highlighting the stability-plasticity dilemma.
Revised: Fig. 1b), highlighting the stability-plasticity dilemma.

Original: Whether this dilemma differently affects feedforward networks and RNNs, and whether weight-importance based methods can be used off the shelf for sequential data has remained unclear.
Revised: Whether this dilemma differently affects feedforward networks and RNNs, and whether weight-importance based methods can be used off the shelf for sequential data has remained unclear.

Original: Here, we contribute to the development of CL approaches for sequential data in several ways.
Revised: Here, we contribute to the development of CL approaches for sequential data in several ways.

Original: • We provide a ﬁrst comprehensive comparison of CL methods applied to sequential data.
Revised: • We provide a ﬁrst comprehensive comparison of CL methods applied to sequential data.

Original: For this, we port a set of established CL methods for feedforward networks to RNNs and assess their performance thoroughly and fairly in a variety of settings.
Revised: For this, we port a set of established CL methods for feedforward networks to RNNs and assess their performance thoroughly and fairly in a variety of settings.

Original: • We identify elements that critically affect the stability-plasticity dilemma of weightimportance methods in RNNs.
Revised: • We identify elements that critically affect the stability-plasticity dilemma of weightimportance methods in RNNs.

Original: We empirically show that high requirements for working memory, i e. the need to store and manipulate information when processing individual samples, lead to a saturation of weight importance values, making the RNN rigid and hindering its potential to learn new tasks.
Revised: We empirically show that high requirements for working memory, i.e. the need to store and manipulate information when processing individual samples, lead to a saturation of weight importance values, making the RNN rigid and hindering its potential to learn new tasks.

Original: In contrast, this trade-off is not directly affected by the sheer recurrent reuse of the weights, related to the length of processed sequences.
Revised: In contrast, this trade-off is not directly affected by the sheer recurrent reuse of the weights, related to the length of processed sequences.

Original: We complement these observations with a theoretical analysis of linear RNNs.
Revised: We complement these observations with a theoretical analysis of linear RNNs.

Original: • We show that existing CL approaches can constitute strong baselines when compared in a standardized setting and if equivalent hyperparameter-optimization resources are granted.
Revised: • We show that existing CL approaches can constitute strong baselines when compared in a standardized setting and if equivalent hyperparameter-optimization resources are granted.

Original: Moreover, we show that a CL regularization approach based on hypernetworks (von Oswald et al., 2020) mitigates the limitations of weight-importance methods in RNNs.
Revised: Moreover, we show that a CL regularization approach based on hypernetworks (von Oswald et al., 2020) mitigates the limitations of weight-importance methods in RNNs.

Original: • We provide a code base 1 comprising all assessed methods as well as variants of four well known sequential datasets adapted to CL: the Copy Task (Graves et al., 2014), Sequential Stroke MNIST (Gulcehre et al., 2017), AudioSet (Gemmeke et al., 2017) and multilingual Part-of-Speech tagging (Nivre et al., 2016).
Revised: • We provide a code base 1 comprising all assessed methods as well as variants of four well known sequential datasets adapted to CL: the Copy Task (Graves et al., 2014), Sequential Stroke MNIST (Gulcehre et al., 2017), AudioSet (Gemmeke et al., 2017) and multilingual Part-of-Speech tagging (Nivre et al., 2016).

Original: Taken together, our experimental and theoretical results facilitate the development of CL methods that are suited for sequential data.
Revised: Taken together, our experimental and theoretical results facilitate the development of CL methods that are suited for sequential data.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Continual learning with sequential data.
Revised: Continual learning with sequential data.

Original: As in Parisi et al.
Revised: As in Parisi et al.

Original: (2019), we categorize CL methods for
Revised: (2019), we categorize CL methods for

Original: RNNs into regularization approaches, dynamic architectures and complementary memory systems.
Revised: RNNs into regularization approaches, dynamic architectures and complementary memory systems.

Original: Regularization approaches set optimization constraints on the update of certain network parameters without requiring a model of past input data.
Revised: Regularization approaches set optimization constraints on the update of certain network parameters without requiring a model of past input data.

Original: EWC, for example, uses weight importance values to limit further updates of weights that are considered essential for solving previous tasks (Kirkpatrick et al., 2017b).
Revised: EWC, for example, uses weight importance values to limit further updates of weights that are considered essential for solving previous tasks (Kirkpatrick et al., 2017b).

Original: Throughout this work, we utilize a more mathematically sound and less memoryintensive version of this algorithm, called Online EWC (Huszár, 2018; Schwarz et al., 2018).
Revised: Throughout this work, we utilize a more mathematically sound and less memoryintensive version of this algorithm, called Online EWC (Huszár, 2018; Schwarz et al., 2018).

Original: Although a highly popular approach in feedforward networks, it has remained unclear how suitable EWC is in the context of sequential processing.
Revised: Although a highly popular approach in feedforward networks, it has remained unclear how suitable EWC is in the context of sequential processing.

Original: Indeed, some studies report promising results in the context of natural language processing (NLP)
Revised: Indeed, some studies report promising results in the context of natural language processing (NLP)

Original: (Madasu and Rao, 2020; Thompson et al., 2019), while others ﬁnd that it performs poorly (Asghar et al., 2020; Cossu et al., 2020a; Li et al., 2020).
Revised: (Madasu and Rao, 2020; Thompson et al., 2019), while others ﬁnd that it performs poorly (Asghar et al., 2020; Cossu et al., 2020a; Li et al., 2020).

Original: Here, we conduct the ﬁrst thorough investigation of EWC’s performance on RNNs, and ﬁnd that it can often be a suitable choice.
Revised: Here, we conduct the ﬁrst thorough investigation of EWC’s performance on RNNs, and ﬁnd that it can often be a suitable choice.

Original: A related CL approach that also relies on weight importance values is SI (Zenke et al., 2017).
Revised: A related CL approach that also relies on weight importance values is SI (Zenke et al., 2017).

Original: Variants of SI have been used for different sequential datasets, but have not been systematically compared against other established methods (Yang et al., 2019; Masse et al., 2018;
Revised: Variants of SI have been used for different sequential datasets, but have not been systematically compared against other established methods (Yang et al., 2019; Masse et al., 2018;

Original: Lee, 2017).
Revised: Lee, 2017).

Original: Fixed expansion layers (Coop and Arel, 2012) are another method to limit the plasticity of weights and prevent forgetting, and in RNNs take the form of a sparsely activated layer between consecutive hidden states (Coop and Arel, 2013).
Revised: Fixed expansion layers (Coop and Arel, 2012) are another method to limit the plasticity of weights and prevent forgetting, and in RNNs take the form of a sparsely activated layer between consecutive hidden states (Coop and Arel, 2013).

Original: Lastly, some regularization approaches rely on the use of non-overlapping and orthogonal representations to overcome catastrophic forgetting (French, 1992; 1994; 1970).
Revised: Lastly, some regularization approaches rely on the use of non-overlapping and orthogonal representations to overcome catastrophic forgetting (French, 1992; 1994; 1970).

Original: Masse et al.
Revised: Masse et al.

Original: (2018), for example, proposed the use of context-dependent random subnetworks, where weight changes are regularized by limiting plasticity to task-speciﬁc subnetworks.
Revised: (2018), for example, proposed the use of context-dependent random subnetworks, where weight changes are regularized by limiting plasticity to task-speciﬁc subnetworks.

Original: This eliminates forgetting for disjoint networks but leads to a reduction of available capacity per task.
Revised: This eliminates forgetting for disjoint networks but leads to a reduction of available capacity per task.

Original: In concurrent work, Duncker et al.
Revised: In concurrent work, Duncker et al.

Original: (2020) introduced a learning rule which aims to optimize the use of the activity-deﬁned subspace in RNNs learning multiple tasks.
Revised: (2020) introduced a learning rule which aims to optimize the use of the activity-deﬁned subspace in RNNs learning multiple tasks.

Original: When tasks are different, catastrophic interference is avoided by forcing the use of task-speciﬁc orthogonal subspaces, whereas the reuse of dynamics is encouraged across tasks that are similar.
Revised: When tasks are different, catastrophic interference is avoided by forcing the use of task-speciﬁc orthogonal subspaces, whereas the reuse of dynamics is encouraged across tasks that are similar.

Original: Dynamic architecture approaches, which rely on the addition of neural resources to mitigate catastrophic forgetting, have also been applied to RNNs.
Revised: Dynamic architecture approaches, which rely on the addition of neural resources to mitigate catastrophic forgetting, have also been applied to RNNs.

Original: Cossu et al.
Revised: Cossu et al.

Original: (2020a) presented a combination of progressive networks (Rusu et al., 2016) and gating autoencoders (Aljundi et al., 2017), where an
Revised: (2020a) presented a combination of progressive networks (Rusu et al., 2016) and gating autoencoders (Aljundi et al., 2017), where an

Original: RNN module is added for each new task and the reconstruction error of task-speciﬁc autoencoders is used to infer the RNN module to be used.
Revised: RNN module is added for each new task and the reconstruction error of task-speciﬁc autoencoders is used to infer the RNN module to be used.

Original: Arguably, the main limitation of this type of approach is the increase in the number of parameters with the number of tasks, although methods have been presented that add resources for each new task only if needed (Tsuda et al., 2020).
Revised: Arguably, the main limitation of this type of approach is the increase in the number of parameters with the number of tasks, although methods have been presented that add resources for each new task only if needed (Tsuda et al., 2020).

Original: Finally, complementary memory systems have also been applied to the retention of sequential information.
Revised: Finally, complementary memory systems have also been applied to the retention of sequential information.

Original: In an early work, Ans et al.
Revised: In an early work, Ans et al.

Original: (2004) proposed a secondary network that generates patterns for rehearsing previously learned information.
Revised: (2004) proposed a secondary network that generates patterns for rehearsing previously learned information.

Original: Asghar et al.
Revised: Asghar et al.

Original: (2020) suggested using an external memory that is progressively increased when new information is encountered.
Revised: (2020) suggested using an external memory that is progressively increased when new information is encountered.

Original: Sodhani et al.
Revised: Sodhani et al.

Original: combined an external memory with Net2Net (Chen et al., 2016), such that the network capacity can be extended while maintaining memories.
Revised: combined an external memory with Net2Net (Chen et al., 2016), such that the network capacity can be extended while maintaining memories.

Original: The major drawback of complementary memory systems is that they either violate CL desiderata by storing past data, or rely on the ability to learn a generative model, a task that arguably scales poorly to complex data.
Revised: The major drawback of complementary memory systems is that they either violate CL desiderata by storing past data, or rely on the ability to learn a generative model, a task that arguably scales poorly to complex data.

Original: We discuss related work in a broader context in supplementary materials (SM D).
Revised: We discuss related work in a broader context in supplementary materials (SM D).

Original: Hypernetworks.
Revised: Hypernetworks.

Original: Introduced by Ha et al.
Revised: Introduced by Ha et al.

Original: (2017), the term hypernetwork refers to a neural network that generates the weights of another network.
Revised: (2017), the term hypernetwork refers to a neural network that generates the weights of another network.

Original: The idea can be traced back to Schmidhuber (1992), who already suggested that a recurrent hypernetwork could be used for learning to learn (Schmidhuber, 1993).
Revised: The idea can be traced back to Schmidhuber (1992), who already suggested that a recurrent hypernetwork could be used for learning to learn (Schmidhuber, 1993).

Original: Importantly, hypernetworks can make use of the fact that parameters in a neural network possess compressible structure (Denil et al., 2013; Han et al., 2015).
Revised: Importantly, hypernetworks can make use of the fact that parameters in a neural network possess compressible structure (Denil et al., 2013; Han et al., 2015).

Original: Indeed, Ha et al.
Revised: Indeed, Ha et al.

Original: (2017) showed that the number of trainable weights of feed-forward architectures can be reduced via hypernetworks.
Revised: (2017) showed that the number of trainable weights of feed-forward architectures can be reduced via hypernetworks.

Original: More recently, hypernetworks have been adapted for CL (He et al., 2019;
Revised: More recently, hypernetworks have been adapted for CL (He et al., 2019;

Original: von Oswald et al., 2020), but not for learning with sequential data.
Revised: von Oswald et al., 2020), but not for learning with sequential data.

Original:  M ETHODS
Revised:  M ETHODS

Original: Recurrent Neural Networks.
Revised: Recurrent Neural Networks.

Original: We consider discrete-time RNNs.
Revised: We consider discrete-time RNNs.

Original: At timestep t , the network’s output ˆy t and hidden state h t are given by ( ˆy t , h t ) =
Revised: At timestep t , the network’s output ˆy t and hidden state h t are given by ( ˆy t , h t ) =

Original: f step ( x t , h t − 1 , ψ ) , where x t denotes the input at time t and ψ the parameters of the network (Cho et al., 2014; Elman, 1990; Hochreiter and Schmidhuber, 1997a).
Revised: f step ( x t , h t − 1 , ψ ) , where x t denotes the input at time t and ψ the parameters of the network (Cho et al., 2014; Elman, 1990; Hochreiter and Schmidhuber, 1997a).

Original: In this work, we consider either vanilla RNNs (based on Elman (1990)),
Revised: In this work, we consider either vanilla RNNs (based on Elman (1990)),

Original: LSTMs (Hochreiter and Schmidhuber, 1997a) or BiLSTMs (Schuster and Paliwal, 1997).
Revised: LSTMs (Hochreiter and Schmidhuber, 1997a) or BiLSTMs (Schuster and Paliwal, 1997).

Original: Naive baselines.
Revised: Naive baselines.

Original: We consider the following naive baselines.
Revised: We consider the following naive baselines.

Original: Fine-tuning refers to training an
Revised: Fine-tuning refers to training an

Original: RNN sequentially on all tasks without any CL protection.
Revised: RNN sequentially on all tasks without any CL protection.

Original: Each task has a different output head (multi-head), and the heads of previously learned tasks are kept ﬁxed.
Revised: Each task has a different output head (multi-head), and the heads of previously learned tasks are kept ﬁxed.

Original: Multitask describes the parallel training on all tasks (no CL).
Revised: Multitask describes the parallel training on all tasks (no CL).

Original: To keep approaches comparable, the multitask baseline uses a multi-head output.
Revised: To keep approaches comparable, the multitask baseline uses a multi-head output.

Original: Because we focus on methods with a comparable number of parameters, we summarize approaches that allocate a different model per task in the From-scratch baseline, where a different model is trained separately for each task, noting that performance improvements are likely to arise in related methods (such as Cossu et al.
Revised: Because we focus on methods with a comparable number of parameters, we summarize approaches that allocate a different model per task in the From-scratch baseline, where a different model is trained separately for each task, noting that performance improvements are likely to arise in related methods (such as Cossu et al.

Original: (2020a))
Revised: (2020a))

Original: whenever knowledge transfer is possible.
Revised: whenever knowledge transfer is possible.

Original: Continual learning baselines.
Revised: Continual learning baselines.

Original: We consider a diverse set of established CL methods and investigate their performance in RNNs.
Revised: We consider a diverse set of established CL methods and investigate their performance in RNNs.

Original: Online EWC (Huszár, 2018; Kirkpatrick et al., 2017a; Schwarz et al., 2018) and SI (Zenke et al., 2017) are different weight-importance CL methods.
Revised: Online EWC (Huszár, 2018; Kirkpatrick et al., 2017a; Schwarz et al., 2018) and SI (Zenke et al., 2017) are different weight-importance CL methods.

Original: A simple weighted L regularization ensures that the neural network is more rigid in weight directions that are considered important for previous tasks, i.e., the loss for the K -th task is given by
Revised: A simple weighted L regularization ensures that the neural network is more rigid in weight directions that are considered important for previous tasks, i.e., the loss for the K -th task is given by

Original: [Equation]
Revised: [Equation]

Original: where λ is the regularization strength, ω
Revised: where λ is the regularization strength, ω

Original: i is the importance associated with ψ i (cf.
Revised: i is the importance associated with ψ i (cf.

Original: SM B.5 and
Revised: SM B.5 and

Original: B.6) and ˜ ψ ( K − 1) denotes the main network weights ψ that were checkpointed after learning task
Revised: B.6) and ˜ ψ ( K − 1) denotes the main network weights ψ that were checkpointed after learning task

Original: K − 1 .
Revised: K − 1 .

Original: We denote by HNET a different regularization approach based on hypernetworks that was recently proposed by von Oswald et al.
Revised: We denote by HNET a different regularization approach based on hypernetworks that was recently proposed by von Oswald et al.

Original: A hypernetwork (Ha et al., 2017) is a neural network ψ
Revised: A hypernetwork (Ha et al., 2017) is a neural network ψ

Original: = h ( e , θ ) with parameters θ and input embeddings e that generates the weights of a main network.
Revised: = h ( e , θ ) with parameters θ and input embeddings e that generates the weights of a [Table] [Figure] [Table]

Revised: +OilW9rI4cmSX7JND4pBTUiFVUiMNwsk9eSTP5MV6sJ6sV+ttNjpnZTs75Aes9y/1Zp7J</latexit>

Original:  This method sidesteps the problem of ﬁnding a compromise between tasks with a shared model ψ , by generating a task-speciﬁc model ψ ( k ) from a low-dimensional embedding space via a shared hypernetwork in which the weights θ and embeddings e are continually learned.
Revised: [Figure] main network. This method sidesteps the problem of ﬁnding a compromise between tasks with a shared model ψ , by generating a task-speciﬁc model ψ ( k ) from a low-dimensional embedding space via a shared hypernetwork in which the weights θ and embeddings e are continually learned.

Original: In contrast to von Oswald et al.
Revised: In contrast to von Oswald et al.

Original: , we focus here on RNNs as main networks: f step ( x t , h t − 1 , ψ ) = f step (cid:0) x t , h t − 1 , h
Revised: , we focus here on RNNs as main networks: f step ( x t , h t − 1 , ψ ) = f step (cid:0) x t , h t − 1 , h

Original: ( e , θ ) (cid:1) (Fig. 1a).
Revised: ( e , θ ) (cid:1) (Fig. 1a).

Original: Crucially, this method has the advantage of not being noticeably affected by the recurrent nature of the main network, since CL is delegated to a feedforward metamodel, where forgetting is avoided based on a simple L2-regularization of its output.For a fair [Table]
Revised: Crucially, this method has the advantage of not being noticeably affected by the recurrent nature of the main network, since CL is delegated to a feedforward metamodel, where forgetting is avoided based on a simple L2-regularization of its output.

Original: +OilW9rI4cmSX7JND4pBTUiFVUiMNwsk9eSTP5MV6sJ6sV+ttNjpnZTs75Aes9y/1Zp7J</latexit>

Original: [Figure] comparison, we ensure that the number of trainable parameters is comparable to other baselines by focusing on chunked hypernetworks (von Oswald et al., 2020), and enforcing: (cid:12)(cid:12) θ ∪ { e k } Kk =1 (cid:12)(cid:12) ≤ | ψ | .
Revised: For a fair comparison, we ensure that the number of trainable parameters is comparable to other baselines by focusing on chunked hypernetworks (von Oswald et al., 2020), and enforcing: (cid:12)(cid:12) θ ∪ { e k } Kk =1 (cid:12)(cid:12) ≤ | ψ | .

Original: Further details can be found in SM B.4.
Revised: Further details can be found in SM B.4.

Original: Masking (or context-dependent gating , Masse et al.
Revised: Masking (or context-dependent gating , Masse et al.

Original: (2018)) applies a binary random mask per task for all hidden units of a multi-head network, and can be seen as a simple method for selecting a different subnetwork per task.
Revised: (2018)) applies a binary random mask per task for all hidden units of a multi-head network, and can be seen as a simple method for selecting a different subnetwork per task.

Original: Since catastrophic interference can occur because of the overlap between subnetworks, this method can be combined with other CL methods such as SI ( Masking+SI ).
Revised: Since catastrophic interference can occur because of the overlap between subnetworks, this method can be combined with other CL methods such as SI ( Masking+SI ).

Original: We also consider methods based on replaying input data from previous tasks, either via a sequentially trained generative model (Shin et al., 2017; van de Ven and
Revised: We also consider methods based on replaying input data from previous tasks, either via a sequentially trained generative model (Shin et al., 2017; van de Ven and

Original: Tolias, 2018), denoted Generative Replay , or by maintaining a small subset of previous training data (Rebufﬁ et al., 2017; Nguyen et al., 2018), denoted Coresets- N , where N refers to the number of samples stored for each task.
Revised: Tolias, 2018), denoted Generative Replay , or by maintaining a small subset of previous training data (Rebufﬁ et al., 2017; Nguyen et al., 2018), denoted Coresets- N , where N refers to the number of samples stored for each task.

Original: Target outputs for replayed data are obtained via a copy of the main network, stored before training on the current task (detailed baseline descriptions in SM B).
Revised: Target outputs for replayed data are obtained via a copy of the main network, stored before training on the current task (detailed baseline descriptions in SM B).

Original: Task Identity.
Revised: Task Identity.

Original: We assume that task identity is provided to the system during training and inference, either by selecting the correct output head or by feeding the correct task embedding e k into the hypernetwork, and elaborate in SM G.12 on how to overcome this limitation.
Revised: We assume that task identity is provided to the system during training and inference, either by selecting the correct output head or by feeding the correct task embedding e k into the hypernetwork, and elaborate in SM G.12 on how to overcome this limitation.

Original:  A NALYSIS OF WEIGHT - IMPORTANCE METHODS
Revised:  A NALYSIS OF WEIGHT - IMPORTANCE METHODS

Original: Weight-importance methods have widely been used in feedforward networks, but whether or not they are suited for tackling CL in RNNs has remained unclear.
Revised: Weight-importance methods have widely been used in feedforward networks, but whether or not they are suited for tackling CL in RNNs has remained unclear.

Original: Here, we investigate the particularities that weight-importance methods face when applied to RNNs.
Revised: Here, we investigate the particularities that weight-importance methods face when applied to RNNs.

Original: As opposed to feedforward networks, RNNs provide a natural way to process temporal sequences.
Revised: As opposed to feedforward networks, RNNs provide a natural way to process temporal sequences.

Original: Because their hidden states are a function of newly incoming inputs as well as their own activity in the previous timestep, RNNs are able to store and manipulate sample-speciﬁc information within their hidden activity, thus providing a form of working memory.
Revised: Because their hidden states are a function of newly incoming inputs as well as their own activity in the previous timestep, RNNs are able to store and manipulate sample-speciﬁc information within their hidden activity, thus providing a form of working memory.

Original: Importantly,the processingof input sequencesby looking at one sequence element at a time results in the reuse of recurrent weights for a number of times that is equal to the length of the input sequence.
Revised: Importantly, processing input sequences one sequence element at a time results in the reuse of recurrent weights for a number of times that is equal to the length of the input sequence.

Original: In this section, we investigate whether weight importance values are directly affected by working  memory requirements or by the length of processed sequences.
Revised: In this section, we investigate whether weight importance values are directly affected by working [Figure] memory requirements or by the length of processed sequences.

Original: For this, we develop some intuitions by studying linear RNNs, which we then test in non-linear RNNs using a synthetic dataset.
Revised: For this, we develop some intuitions by studying linear RNNs, which we then test in non-linear RNNs using a synthetic dataset.

Original: First, we obtain some theoretical insights by analysing how linear RNNs can learn to solve a set of tasks (for details refer to SM C).
Revised: First, we obtain some theoretical insights by analysing how linear RNNs can learn to solve a set of tasks (for details refer to SM C).

Original: Whenever task-speciﬁc output heads are not rich enough to model task variabilities, RNNs trained with methods whose recurrent computation is not task-conditioned (e.g. weight-importance methods) must solve all tasks simultaneously within their hidden space.
Revised: Whenever task-speciﬁc output heads are not rich enough to model task variabilities, RNNs trained with methods whose recurrent computation is not task-conditioned (e.g. weight-importance methods) must solve all tasks simultaneously within their hidden space.

Original: In an extreme scenario where tasks are so different that they cannot share any useful computation,[Figure] it becomes clear that task interferences can be prevented if the information relevant to each task resides in task-speciﬁc orthogonal subspaces (refer to SM G.10 for a discussion of scenarios with task similarity).
Revised: In an extreme scenario where tasks are so different that they cannot share any useful computation, it becomes clear that task interferences can be prevented if the information relevant to each task resides in task-speciﬁc orthogonal subspaces (refer to SM G.10 for a discussion of scenarios with task similarity).

Original: Maintaining this structure within the hidden space imposes certain constraints in the recurrent weights, Our theory shows these constraints increase with the number of tasks and with the dimensionality of the task-speciﬁc subspaces.
Revised: Maintaining this structure within the hidden space imposes certain constraints on the recurrent weights. Our theory shows these constraints increase with the number of tasks and with the dimensionality of the task-speciﬁc subspaces.

Original: Based on these theoretical insights, we hypothesize that also in nonlinear RNNs increasing working memory requirements cause high weight rigidity (as illustrated by high importance values), whereas the recurrent reuse of weightsitself is not a driving factor.
Revised: Based on these theoretical insights, we hypothesize that also in nonlinear RNNs increasing working memory requirements cause high weight rigidity (as illustrated by high importance values), whereas the recurrent reuse of weights is not a driving factor.

Original: To test this hypothesis, we explore a synthetic dataset consisting of several variations of the Copy
Revised: To test this hypothesis, we explore a synthetic dataset consisting of several variations of the Copy

Original: Task (Graves et al., 2014), in which a random binary input pattern has to be recalled by the network after a stop bit is observed (see Fig. 2, and SM E.1 for details).
Revised: Task (Graves et al., 2014), in which a random binary input pattern has to be recalled by the network after a stop bit is observed (see Fig. 2, and SM E.1 for details).

Original: For all Copy Task experiments, we use vanilla RNNs combined with orthogonal regularization (see SM G.2).
Revised: For all Copy Task experiments, we use vanilla RNNs combined with orthogonal regularization (see SM G.2).

Original: We denote the length (number of timesteps) of the binary input pattern to be copied by p , and the actual number of timesteps until the stop bit by i (examples can be found in SM Fig. S1).
Revised: We denote the length (number of timesteps) of the binary input pattern to be copied by p , and the actual number of timesteps until the stop bit by i (examples can be found in SM Fig. S1).

Original: This distinction allows us to consider two variants, the basic Copy Task where p =
Revised: This distinction allows us to consider two variants, the basic Copy Task where p =

Original: i , and the Padded Copy Task where i > p (Fig.
Revised: i , and the Padded Copy Task where i > p (Fig.

Original: 2 a and b).
Revised: 2 a and b).

Original: In this variant, we zero-pad a binary input pattern of length p for i − p timesteps until the occurrence of the stop bit, resulting in an input sequence with i timesteps.
Revised: In this variant, we zero-pad a binary input pattern of length p for i − p timesteps until the occurrence of the stop bit, resulting in an input sequence with i timesteps.

Original: Speciﬁcally, we consider a set of Copy
Revised: Speciﬁcally, we consider a set of Copy

Original: Tasks 2 with varying input lengths i and, either a ﬁxed pattern length p = 5 , or a pattern length tied to the input length ( p = i ).
Revised: Tasks 2 with varying input lengths i and, either a ﬁxed pattern length p = 5 , or a pattern length tied to the input length ( p = i ).

Original: This allows us to disentangle how sequence length and memory load affect weight importance.
Revised: This allows us to disentangle how sequence length and memory load affect weight importance.

Original: As in Online EWC , we calculate weight importance as the diagonal elements of the empirical Fisher information matrix (see SM B.5).
Revised: As in Online EWC , we calculate weight importance as the diagonal elements of the empirical Fisher information matrix (see SM B.5).

Original: To quantify memory load, we study the intrinsic dimensionality of the hidden state of the RNN using principal component analysis (PCA), once networks have been trained to achieve near optimal performance (above 99%).
Revised: To quantify memory load, we study the intrinsic dimensionality of the hidden state of the RNN using principal component analysis (PCA), once networks have been trained to achieve near optimal performance (above 99%).

Original: We deﬁne the intrinsic dimensionality as the number of principal components that are needed to explain 75% of the variance.
Revised: We deﬁne the intrinsic dimensionality as the number of principal components that are needed to explain 75% of the variance.

Original: As expected, the intrinsic dimensionality of the hidden space increases during input pattern presentation and peaks after p timesteps, i.e. at the stop bit for tasks with p
Revised: As expected, the intrinsic dimensionality of the hidden space increases during input pattern presentation and peaks after p timesteps, i.e. at the stop bit for tasks with p

Original: = i (Fig. 3a)
Revised: = i (Fig. 3a)

Original: and i − p timesteps before the stop bit if p = 5 remains ﬁxed (Fig. 3b).
Revised: and i − p timesteps before the stop bit if p = 5 remains ﬁxed (Fig. 3b).

Original: 3 Weight importance values rapidly increase with memory requirements ( p ), but notwith sequence length (increasing i for ﬁxed p ) (Fig. 3c).
Revised: 3 Weight importance values rapidly increase with memory requirements ( p ), but not sequence length (increasing i , ﬁxed p ) (Fig. 3c).

Original: The same trend is observed when computing weight-importance values according to SI (cf.
Revised: The same trend is learned independently to isolate the effects of i and p on weight importance values.

Original: SM G.3), and when using LSTMs instead of vanilla RNNs (cf.
Revised: Note, Fig. 3a shows a decreased dimensionality at t = 0 for i = 35 or i = 40 compared to i = 30 .

Original: SM G.4). We extend this analysis to a CL setting in SM G.5.
Revised: We hypothesize that this is due to a need to non-linearly encode information into the hidden state for large i , and veriﬁed that the dimensionality increases with i when using Kernel PCA (Schölkopf et al., 1997), data not shown.

Original: nu m b e r o f d i m .
Revised: nu m b e r o f d i m .

Original: t (a)  0  nu m b e r
Revised: t (a)  0  nu m b e r

Original: o
Revised: o

Original: f
Revised: f

Original: d i m .
Revised: d i m .

Original: i npu t l e ng t h t
Revised: i npu t l e ng t h t

Original: i (b)   40 input length i   m ea n F i s h e r varying fixed p p (c) 
Revised: i (b)   40 input length i   m ea n F i s h e r varying fixed p p (c) observed when computing weight-importance values according to SI (cf. SM G.3), and when using LSTMs instead of vanilla RNNs (cf. SM G.4).

Revised: We extend this analysis to a CL setting in SM G.5.

Original: Overall, this analysis reveals that weight-importance methods are affected by the processing and storage required by the task, but not by the sequential nature of the data, even if the same set of weights is reused over many timesteps.
Revised: Overall, this analysis reveals that weight-importance methods are affected by the processing and storage required by the task, but not by the sequential nature of the data, even if the same set of weights is reused over many timesteps.

Original: This could cause weight-importance methods to suffer from a saturation of importance values when working memory requirements are high, which could in turn reduce their plasticity for learning new tasks.
Revised: This could cause weight-importance methods to suffer from a saturation of importance values when working memory requirements are high, which could in turn reduce their plasticity for learning new tasks.

Original: We explore this in the next section.
Revised: We explore this in the next section.

Original:  C ONTINUAL L EARNING E XPERIMENTS
Revised:  C ONTINUAL L EARNING E XPERIMENTS

Original: To highlight strengths and weaknesses of different CL methods in various settings, we performed experiments on one synthetic and two real-world sequential datasets, using different types of RNNs.
Revised: To highlight strengths and weaknesses of different CL methods in various settings, we performed experiments on one synthetic and two real-world sequential datasets, using different types of RNNs.

Original: We distinguish between during and final accuracies.
Revised: We distinguish between during and final accuracies.

Original: The during accuracy of a CL experiment is obtained by taking the mean over the test accuracy from each task right after it has been trained on, i.e., when tasks have not yet been subject to forgetting.
Revised: The during accuracy of a CL experiment is obtained by taking the mean over the test accuracy from each task right after it has been trained on, i.e., when tasks have not yet been subject to forgetting.

Original: The final accuracy describes the mean test accuracy over all tasks obtained after the last task has been learned.
Revised: The final accuracy describes the mean test accuracy over all tasks obtained after the last task has been learned.

Original: For all reported methods, results were obtained via an extensive hyperparameter search, where the hyperparameter conﬁguration of the run with best final accuracy was selected and subsequently tested on multiple random seeds (experimental details in SM F).
Revised: For all reported methods, results were obtained via an extensive hyperparameter search, where the hyperparameter conﬁguration of the run with best final accuracy was selected and subsequently tested on multiple random seeds (experimental details in SM F).

Original: We provide additional CL experiments on multilingual NLP data in SM G.9.
Revised: We provide additional CL experiments on multilingual NLP data in SM G.9.

Original: 5. V ARIATIONS OF THE C OPY T ASK
Revised: 5. V ARIATIONS OF THE C OPY T ASK

Original: After exposing the challenges that weight-importance methods face when dynamically processing data, we explore how these manifest in a CL scenario.
Revised: After exposing the challenges that weight-importance methods face when dynamically processing data, we explore how these manifest in a CL scenario.

Original: We compare weight-importance methods against other CL approaches, with a particular focus on HNET , which can in principle bypass those challenges.
Revised: We compare weight-importance methods against other CL approaches, with a particular focus on HNET , which can in principle bypass those challenges.

Original: We transform the Copy Task into a set of CL tasks by applying for each task k
Revised: We transform the Copy Task into a set of CL tasks by applying for each task k

Original: a different random time-permutation π ( k ) .
Revised: a different random time-permutation π ( k ) .

Original: In this setting, which we denote Permuted Copy Task , each timestep t i ∈ { 1 , . . .
Revised: In this setting, which we denote Permuted Copy Task , each timestep t i ∈ { 1 , . . .

Original: , p } from the input pattern has to be recalled at output timestep t o =
Revised: , p } from the input pattern has to be recalled at output timestep t o =

Original: π
Revised: π

Original: ( k ) ( t i ) (Fig.
Revised: ( k ) ( t i ) (Fig.

Original: 2c).
Revised: 2c).

Original: We perform these experiments on vanilla RNNs.
Revised: We perform these experiments on vanilla RNNs.

Original: First, we evaluate all methods in a relatively simple scenario with ﬁve tasks using p = i = 5 (Table 1).
Revised: First, we evaluate all methods in a relatively simple scenario with ﬁve tasks using p = i = 5 (Table 1).

Original: Online EWC achieves very high performance, and HNET reaches close to 100% accuracy.
Revised: Online EWC achieves very high performance, and HNET reaches close to 100% accuracy.

Original: The random subnetworks in Masking can learn individual tasks to perfection.
Revised: The random subnetworks in Masking can learn individual tasks to perfection.

Original: However, weight changes within subnetworks, which result from random overlaps, cause severe performance drops and show the need to add stabilization mechanisms, e.g., Masking+SI .
Revised: However, weight changes within subnetworks, which result from random overlaps, cause severe performance drops and show the need to add stabilization mechanisms, e.g., Masking+SI .

Original: Since the input data distribution is relatively simple and identical across tasks, learning a generative model is feasible, which is illustrated by the performance of Generative
Revised: Since the input data distribution is relatively simple and identical across tasks, learning a generative model is feasible, which is illustrated by the performance of Gen. Replay . [Table]

Original: Replay .
Revised: [Table]

Original: In the following, we focus on a comparison between Online EWC and HNET to further investigate how these methods are affected by sequence length and working memory requirements.
Revised: In the following, we focus on a comparison between Online EWC and HNET to further investigate how these methods are affected by sequence length and working memory requirements.

Original: We ﬁrst test whether Online EWC is affected by sequence length by investigating the Permuted Copy Task at 
Revised: We ﬁrst test whether Online EWC is affected by sequence length by investigating the Permuted Copy Task at p

Original: [Table] [Table] p = 5 , i = 25 using 5 tasks.
Revised: = 5 , i = 25 using 5 tasks.

Original: As Table 2 shows [Table] the performance of both methods is not markedly affected by sequence length.
Revised: As Table 2 shows, the performance of both methods is not markedly affected by sequence length.

Original: Interestingly, the results are slightly better for longer sequences with both methods, which can be due to an increased processing time between input presentation and recall.
Revised: Interestingly, the results are slightly better for longer sequences with both methods, which can be due to an increased processing time between input presentation and recall.

Original: Next, we compare the performance of Online EWC and HNET in a set of tasks for which working memory requirements can be easily controlled.
Revised: Next, we compare the performance of Online EWC and HNET in a set of tasks for which working memory requirements can be easily controlled.

Original: In this setting, referred to as Pattern Manipulation
Revised: In this setting, referred to as Pattern Manipulation

Original: Task , difﬁculty is controlled by a set of r task-speciﬁc random permutations along the time axis (Fig.
Revised: Task , difﬁculty is controlled by a set of r task-speciﬁc random permutations along the time axis (Fig.

Original: 2d).
Revised: 2d).

Original: The output is computed from the input pattern by applying a binary XOR operation iteratively with all of its r permutations (i.e. the result of the XOR between the input and its ﬁrst permutation will then undergo a second XOR operation with the second permutation of the input, and so on).
Revised: The output is computed from the input pattern by applying a binary XOR operation iteratively with all of its r permutations (i.e. the result of the XOR between the input and its ﬁrst permutation will then undergo a second XOR operation with the second permutation of the input, and so on).

Original: Note that this variant substantially differs from previous Copy Task variations, since the processing of input patterns is now both input- and task-dependent.
Revised: Note that this variant substantially differs from previous Copy Task variations, since the processing of input patterns is now both input- and task-dependent.

Original: As shown in Table 2, Online EWC experiences a larger drop with increased task difﬁculty than HNET , conﬁrming that it is more severely affected by working memory requirements.
Revised: As shown in Table 2, Online EWC experiences a larger drop with increased task difﬁculty than HNET , conﬁrming that it is more severely affected by working memory requirements.

Original: Finally, we investigate the difference between single-head and multi-head settings, as well as task conditional processing in SM G.13.
Revised: Finally, we investigate the difference between single-head and multi-head settings, as well as task conditional processing in SM G.13.

Original: 5. S EQUENTIAL S TROKE MNIST
Revised: 5. S EQUENTIAL S TROKE MNIST

Original: To test whether the results from the synthetic Copy Task hold true for real world data we turned to a sequential digit recognition task where task difﬁculty can be directly controlled.
Revised: To test whether the results from the synthetic Copy Task hold true for real world data we turned to a sequential digit recognition task where task difﬁculty can be directly controlled.

Original: In the Stroke
Revised: In the Stroke

Original: MNIST (SMNIST) dataset (de Jong, 2016), MNIST images (LeCun et al., 1998) are represented as sequences of pen displacements, that result in the original digits when drawn in order.
Revised: MNIST (SMNIST) dataset (de Jong, 2016), MNIST images (LeCun et al., 1998) are represented as sequences of pen displacements, that result in the original digits when drawn in order.

Original: We adapt this dataset to a CL scenario by splitting it into ﬁve binary classiﬁcation problems (digits 0 vs 1, 2 vs 3, etc.), reminiscent of the popular Split-MNIST experiment commonly used to benchmark
Revised: We adapt this dataset to a CL scenario by splitting it into ﬁve binary classiﬁcation problems (digits 0 vs 1, 2 vs 3, etc.), reminiscent of the popular Split-MNIST experiment commonly used to benchmark

Original: CL methods on static data (Zenke et al., 2017).
Revised: CL methods on static data (Zenke et al., 2017).

Original: Interestingly, this dataset allows exploring how the performance of different CL methods depends on the difﬁculty of individual tasks by generalizing the notion of Split-SMNIST to sequences of m SMNIST samples (cf.
Revised: Interestingly, this dataset allows exploring how the performance of different CL methods depends on the difﬁculty of individual tasks by generalizing the notion of Split-SMNIST to sequences of m SMNIST samples (cf.

Original: Gulcehre et al., 2017), where each sequence contains only two types of digits (e.g. 2332 or 7767 for m = 4 ).
Revised: Gulcehre et al., 2017), where each sequence contains only two types of digits (e.g. 2332 or 7767 for m = 4 ).

Original: To obtain a binary decision problem, we randomly group all 2 m possible sequences within a task into two classes.
Revised: To obtain a binary decision problem, we randomly group all 2 m possible sequences within a task into two classes.

Original: This ensures that despite increasing levels of task difﬁculty, as determined by m , chance level is not affected.
Revised: This ensures that despite increasing levels of task difﬁculty, as determined by m , chance level is not affected.

Original: Crucially, an increase in m leads to an increase in the amount of information that needs to be stored and manipulated per input sequence, and therefore allows exploring the effect that increasing working memory requirements have on different CL methods in a real-world dataset.
Revised: Crucially, an increase in m leads to an increase in the amount of information that needs to be stored and manipulated per input sequence, and therefore allows exploring the effect that increasing working memory requirements have on different CL methods in a real-world dataset.

Original: We train LSTMs on ﬁve tasks for an increasing number of digits per sequence ( m = 1 , 2 , 3 , 4 ) and observe that methods are differently affected by the task difﬁculty level (see Fig. 4).
Revised: We train LSTMs on ﬁve tasks for an increasing number of digits per sequence ( m = 1 , 2 , 3 , 4 ) and observe that methods are differently affected by the task difﬁculty level (see Fig. 4).

Original: For m = 1 Online EWC , SI and HNET all achieve above 97% performance.
Revised: For m = 1 Online EWC , SI and HNET all achieve above 97% performance.

Original: However for m =the performance of Online EWC and SI drops to 73 .
Revised: However for m =the performance of Online EWC and SI drops to 73 .

Original: 16 ± 0 .
Revised: 16 ± 0 .

Original: 98% and 69 .
Revised: 98% and 69 .

Original: 58 ± 0 .
Revised: 58 ± 0 .

Original: 55% respectively, while the hypernetwork approach successfully classiﬁes 94 .
Revised: 55% respectively, while the hypernetwork approach successfully classiﬁes 94 .

Original: 42 ± 1 .
Revised: 42 ± 1 .

Original: 85% of all inputs.
Revised: 85% of all inputs.

Original: Thus weightimportance methods seem more strongly affected than HNET by an increase in task complexity and working memory requirements.
Revised: Thus weightimportance methods seem more strongly affected than HNET by an increase in task complexity and working memory requirements.

Original: Interestingly, the performance gap depends on the experimental setup as outlined in SM G.13.
Revised: Interestingly, the performance gap depends on the experimental setup as outlined in SM G.13.

Original: Coresets perform slightly worse, especially when task-complexity increases.
Revised: Coresets perform slightly worse, especially when task-complexity increases.

Original: Masking+SI , which trades-off network capacity for the ability of ﬁnding solutions in aless rigid subnetwork, emerges as the preferable method for this experiment. We additionally list during accuracies for all methods in SM Table 5 and discuss the use of replay for Split-SMNIST in SM G.7. Finally we show that, consistent with our Copy Task results, the performance of weight-
Revised: Masking+SI , which trades-off network capacity for the ability of ﬁnding solutions in a

Original:     GLJLWVSHUVHTXHQFHP   DFF X U DF \     +1(72QOLQH(:&6,0DVNLQJ6,&RUHVHWV)LQHWXQLQJ)URPVFUDWFK0XOWLWDVN
Revised:     GLJLWVSHUVHTXHQFHP   DFF X U DF \     +1(72QOLQH(:&6,0DVNLQJ6,&RUHVHWV)LQHWXQLQJ)URPVFUDWFK0XOWLWDVN

Original: [Figure]
Revised: [Figure]

Original: Online EWC
Revised: EWC

Original: HNET
Revised: HNET

Original: [Figure]
Revised: [Figure]

Original: importance methods is not signiﬁcantly affected when sequence lengths are increased without a concomitant increase in working memory (cf.
Revised: less rigid subnetwork, emerges as the preferable method for this experiment.

Original: SM.
Revised: We additionally list during accuracies for all methods in SM Table 5 and discuss the use of replay for Split-SMNIST in

Original: G.8).
Revised: SM G.7.

Original: 5. A UDIO S ET
Revised: Finally we show that, consistent with our Copy Task results, the performance of weightimportance methods is not signiﬁcantly affected when sequence lengths are increased without a concomitant increase in working memory (cf.

Original:  [Table]
Revised: SM. G.8). 5. A UDIO S ET [Table]

Original: AudioSet (Gemmeke et al., 2017) is a dataset of manually annotated audio events.
Revised: AudioSet (Gemmeke et al., 2017) is a dataset of manually annotated audio events.

Original: It consists of 10-second audio snippets which have been preprocessed by a VGG network to extract 128dimensional feature vectors at 1 Hz.
Revised: It consists of 10-second audio snippets which have been preprocessed by a VGG network to extract 128dimensional feature vectors at 1 Hz.

Original: This dataset has been previously adapted for CL by Kemker and Kanan (2018) and Kemker et al.
Revised: This dataset has been previously adapted for CL by Kemker and Kanan (2018) and Kemker et al.

Original: (2018), whose particular split has not been made public, and by Cossu et al.
Revised: (2018), whose particular split has not been made public, and by Cossu et al.

Original: (2020b), for which the test set size largely differed across classes.
Revised: (2020b), for which the test set size largely differed across classes.

Original: We therefore created a new variant, which we call
Revised: We therefore created a new variant, which we call

Original: Split-AudioSet-10 , containing 10 tasks withclasses each (see SM F.3 for details).
Revised: Split-AudioSet-10 , containing 10 tasks withclasses each (see SM F.3 for details).

Original: The results obtained in this dataset using LSTMs are listed in Table 3.
Revised: The results obtained in this dataset using LSTMs are listed in Table 3.

Original: HNET is the strongest among regularization based methods, and is only outperformed by Coresets , which rely on storing past data.
Revised: HNET is the strongest among regularization based methods, and is only outperformed by Coresets , which rely on storing past data.

Original: Masking during accuracies indicate that random subnetworks have enough capacity to learn individual tasks, but low final accuracies suggest that catastrophic forgetting occurs, presumably because of the overlap between subnetworks.
Revised: Masking during accuracies indicate that random subnetworks have enough capacity to learn individual tasks, but low final accuracies suggest that catastrophic forgetting occurs, presumably because of the overlap between subnetworks.

Original: This is partly solved in Masking+SI by introducing stabilization which, however, reduces plasticity for learning new tasks.
Revised: This is partly solved in Masking+SI by introducing stabilization which, however, reduces plasticity for learning new tasks.

Original: In contrast to the ﬁndings of Sec.
Revised: In contrast to the ﬁndings of Sec.

Original: 5.2 and SM G.9, Masking+SI performs worst among regularization approaches, indicating that trading-off capacity for complex datasets can be harmful.
Revised: 5.2 and SM G.9, Masking+SI performs worst among regularization approaches, indicating that trading-off capacity for complex datasets can be harmful.

Original: The From-scratch baseline outperforms other methods, which is explained by the fact that it trains a separate model per task, leading to 10 times more network capacity.
Revised: The From-scratch baseline outperforms other methods, which is explained by the fact that it trains a separate model per task, leading to 10 times more network capacity.

Original: Notably, we were not able to successfully train a
Revised: Notably, we were not able to successfully train a

Original: Generative Replay model on this dataset despite extensive hyperparameter search.
Revised: Generative Replay model on this dataset despite extensive hyperparameter search.

Original: Together with the results in Sec.
Revised: Together with the results in Sec.

Original: 5.1, this highlights that the performance of Generative Replay depends on the complexity of the input data distribution, and not necessarily on the CL nature of the problem.
Revised: 5.1, this highlights that the performance of Generative Replay depends on the complexity of the input data distribution, and not necessarily on the CL nature of the problem.

Original: To further investigate the stability-plasticity trade-off, we tested HNET and Online EWC across a range of difﬁculty levels in individual tasks.
Revised: To further investigate the stability-plasticity trade-off, we tested HNET and Online EWC across a range of difﬁculty levels in individual tasks.

Original: This can be controlled by the number of classes to be learned within each task, which we varied from two to ten.
Revised: This can be controlled by the number of classes to be learned within each task, which we varied from two to ten.

Original: For both methods we used the best hyperparameters found for Split-AudioSet-10.
Revised: For both methods we used the best hyperparameters found for Split-AudioSet-10.

Original: Since the performance of Online EWC strongly depends on the regularization parameter λ , we tuned this value to achieve optimal results in each setting (cf.
Revised: Since the performance of Online EWC strongly depends on the regularization parameter λ , we tuned this value to achieve optimal results in each setting (cf.

Original: Fig. S2).
Revised: Fig. S2).

Original: Fig.
Revised: Fig.

Original: 5 shows the task-averaged final accuracies for the different task-difﬁculty settings.
Revised: 5 shows the task-averaged final accuracies for the different task-difﬁculty settings.

Original: While HNET performance is primarily affected by task difﬁculty but not by the number of tasks, results for Online EWC show an interplay between task difﬁculty and the ability to retain good performance on many tasks.
Revised: While HNET performance is primarily affected by task difﬁculty but not by the number of tasks, results for Online EWC show an interplay between task difﬁculty and the ability to retain good performance on many tasks.

Original: These results provide further evidence that the hypernetwork-based approach can resolve the limitations of weight-importance CL methods for sequential data.
Revised: These results provide further evidence that the hypernetwork-based approach can resolve the limitations of weight-importance CL methods for sequential data.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: The stability-plasticity dilemma with sequential data.
Revised: The stability-plasticity dilemma with sequential data.

Original: Weight-importance methods address CL by progressively constraining a network’s weights, directly trading plasticity for stability.
Revised: Weight-importance methods address CL by progressively constraining a network’s weights, directly trading plasticity for stability.

Original: In the case of RNNs, weights are subject to additional constraints, since the same set of weights is reused across time to dynamically process an input stream of data.
Revised: In the case of RNNs, weights are subject to additional constraints, since the same set of weights is reused across time to dynamically process an input stream of data.

Original: We show that increased working memory requirements, resulting from more complex processing within individual tasks, lead to high weightimportance values and can hinder the ability to learn future tasks (cf.
Revised: We show that increased working memory requirements, resulting from more complex processing within individual tasks, lead to high weightimportance values and can hinder the ability to learn future tasks (cf.

Original: Sec. 4).
Revised: Sec. 4).

Original: On the contrary, we ﬁnd that longer sequence lengths do not impact performance for a ﬁxed level of task complexity (cf.
Revised: On the contrary, we ﬁnd that longer sequence lengths do not impact performance for a ﬁxed level of task complexity (cf.

Original: Fig.
Revised: Fig.

Original: 3b, Table 2), suggesting that weight reuse doesn’t interfere with the RNN’s ability to retain previous knowledge.
Revised: 3b, Table 2), suggesting that weight reuse doesn’t interfere with the RNN’s ability to retain previous knowledge.

Original: These observations are consistent with our theoretical analysis of linear RNNs (SM C), which predicts that more challenging processing within individual tasks leads to increased interference between tasks.
Revised: These observations are consistent with our theoretical analysis of linear RNNs (SM C), which predicts that more challenging processing within individual tasks leads to increased interference between tasks.

Original: This aggravates the stability-plasticity dilemma in weight-importance based methods, which we conﬁrm in a range of experiments.
Revised: This aggravates the stability-plasticity dilemma in weight-importance based methods, which we conﬁrm in a range of experiments.

Original: Beneﬁts of a hypernetwork-based CL approach for sequential data.
Revised: Beneﬁts of a hypernetwork-based CL approach for sequential data.

Original: We propose that a hypernetwork-based approach (von Oswald et al., 2020) can alleviate the stability-plasticity dilemma when continually learning with RNNs.
Revised: We propose that a hypernetwork-based approach (von Oswald et al., 2020) can alleviate the stability-plasticity dilemma when continually learning with RNNs.

Original: Since stability is outsourced to a regularizer that does not directly limit the plasticity of main network weights (cf.
Revised: Since stability is outsourced to a regularizer that does not directly limit the plasticity of main network weights (cf.

Original: SM Eq. 2), this approach has more ﬂexibility than weight-importance methods for ﬁnding new solutions, as shown by our experiments.
Revised: SM Eq. 2), this approach has more ﬂexibility than weight-importance methods for ﬁnding new solutions, as shown by our experiments.

Original: Although
Revised: Although

Original: Coresets and Generative Replay perform better, these approaches might not always be applicable; Coresets rely on the storage of past data (which might not always be feasible for privacy or storage reasons), and Generative Replay scales poorly to complex data.
Revised: Coresets and Generative Replay perform better, these approaches might not always be applicable; Coresets rely on the storage of past data (which might not always be feasible for privacy or storage reasons), and Generative Replay scales poorly to complex data.

Original: On the contrary, an approach based on hypernetworks has the versatility of regularization approaches and can be used in a variety of situations.
Revised: On the contrary, an approach based on hypernetworks has the versatility of regularization approaches and can be used in a variety of situations.

Original: Future avenues for CL with RNNs.
Revised: Future avenues for CL with RNNs.

Original: As discussed in SM G.13, an interesting avenue for weightimportance methods is the use of task-conditional processing in order to overcome the need of solving all learned tasks in parallel.
Revised: As discussed in SM G.13, an interesting avenue for weightimportance methods is the use of task-conditional processing in order to overcome the need of solving all learned tasks in parallel.

Original: Although hypernetworks overcome this limitation by design, they introduce additional optimization challenges, especially in conjunction with vanilla RNNs (cf.
Revised: Although hypernetworks overcome this limitation by design, they introduce additional optimization challenges, especially in conjunction with vanilla RNNs (cf.

Original: SM
Revised: SM

Original: G.2), which leaves room for future improvements.
Revised: G.2), which leaves room for future improvements.

Original: An interesting direction is the use of a recurrent hypernetwork to generate timestep-speciﬁc weights in the main RNN (Ha et al., 2017; Suarez, 2017).
Revised: An interesting direction is the use of a recurrent hypernetwork to generate timestep-speciﬁc weights in the main RNN (Ha et al., 2017; Suarez, 2017).

Original: Although a naive application of this combination for CL using SM Eq.
Revised: Although a naive application of this combination for CL using SM Eq.

Original: 2 would come at the cost of a linear increase in computation with the number of timesteps, this problem can be elegantly sidestepped by the use of a feed-forward hypernetwork that generates the weights of the recurrent hypernetwork.
Revised: 2 would come at the cost of a linear increase in computation with the number of timesteps, this problem can be elegantly sidestepped by the use of a feed-forward hypernetwork that generates the weights of the recurrent hypernetwork.

Original: SM Eq.
Revised: SM Eq.

Original: 2 can then simply be applied to the static output of this hyper-hypernetwork , protecting a set of timestep-speciﬁc weights per task without the need to increase the regularization budget.
Revised: 2 can then simply be applied to the static output of this hyper-hypernetwork , protecting a set of timestep-speciﬁc weights per task without the need to increase the regularization budget.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Our work advances the CL community in three ways.
Revised: Our work advances the CL community in three ways.

Original: First, by systematically evaluating the performance of established CL methods when applied to RNNs, we provide extensive baselines that can serve as reference for future studies on CL with sequential data.
Revised: First, by systematically evaluating the performance of established CL methods when applied to RNNs, we provide extensive baselines that can serve as reference for future studies on CL with sequential data.

Original: Second, we use theoretical arguments derived from linear RNNs to hypothesize limitations of weight-importance based CL in the context of recurrent computation, and provide empirical evidence to support these statements.
Revised: Second, we use theoretical arguments derived from linear RNNs to hypothesize limitations of weight-importance based CL in the context of recurrent computation, and provide empirical evidence to support these statements.

Original: Third, derived from these insights, we suggest that an approach based on hypernetworks mitigates the stability-plasticity dilemma, and show that it outperforms weight-importance methods on synthetic as well as real-world data.
Revised: Third, derived from these insights, we suggest that an approach based on hypernetworks mitigates the stability-plasticity dilemma, and show that it outperforms weight-importance methods on synthetic as well as real-world data.

Original: Finally, our work discusses several future improvements and directions of CL approaches for sequential data.
Revised: Finally, our work discusses several future improvements and directions of CL approaches for sequential data.