Original: U NIFYING R EGULARISATION M ETHODS FOR C ONTINUAL L EARNING
Revised: U NIFYING R EGULARISATION M ETHODS FOR C ONTINUAL L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Continual Learning addresses the challenge of learning a number of different distributions sequentially.
Revised: Continual Learning addresses the challenge of learning a number of different distributions sequentially.

Original: The goal of maintaining knowledge of earlier distributions without re-accessing them starkly conﬂicts with standard SGD training for artiﬁcial neural networks.
Revised: The goal of maintaining knowledge of earlier distributions without re-accessing them starkly conﬂicts with standard SGD training for artiﬁcial neural networks.

Original: An inﬂuential method to tackle this are so-called regularisation approaches.
Revised: An inﬂuential method to tackle this are so-called regularisation approaches.

Original: They measure the importance of each parameter for modelling a given distribution and subsequently protect important parameters from large changes.
Revised: They measure the importance of each parameter for modelling a given distribution and subsequently protect important parameters from large changes.

Original: In the literature, three ways to measure parameter importance have been put forward and they have inspired a large body of follow-up work.
Revised: In the literature, three ways to measure parameter importance have been put forward and they have inspired a large body of follow-up work.

Original: Here, we present strong theoretical and empirical evidence that these three methods, Elastic Weight
Revised: Here, we present strong theoretical and empirical evidence that these three methods, Elastic Weight

Original: Consolidation (EWC), Synaptic Intelligence (SI) and Memory Aware Synapses (MAS), all approximate the Fisher Information.
Revised: Consolidation (EWC), Synaptic Intelligence (SI) and Memory Aware Synapses (MAS), all approximate the Fisher Information.

Original: Only EWC intentionally relies on the Fisher, while the other two methods stem from rather different motivations.
Revised: Only EWC intentionally relies on the Fisher, while the other two methods stem from rather different motivations.

Original: We ﬁnd that for SI the relation to the Fisher – and in fact its performance – is due to a previously unknown bias.
Revised: We ﬁnd that for SI the relation to the Fisher – and in fact its performance – is due to a previously unknown bias.

Original: Altogether, this uniﬁes a large body of regularisation approaches.
Revised: Altogether, this uniﬁes a large body of regularisation approaches.

Original: It also provides the ﬁrst theoretical explanation for the effectiveness of
Revised: It also provides the ﬁrst theoretical explanation for the effectiveness of

Original: SI- and MAS-based algorithms and offers theoretically justiﬁed versions of these algorithms.
Revised: SI- and MAS-based algorithms and offers theoretically justiﬁed versions of these algorithms.

Original: From a practical viewpoint, our insights offer computational speed-ups and uncover conditions needed for different algorithms to work.
Revised: From a practical viewpoint, our insights offer computational speed-ups and uncover conditions needed for different algorithms to work.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Despite considerable progress, many gaps between biological and machine intelligence remain.
Revised: Despite considerable progress, many gaps between biological and machine intelligence remain.

Original: Animals, for example, ﬂexibly learn new tasks throughout their lives, while at the same time maintaining robust memories of previous knowledge.
Revised: Animals, for example, ﬂexibly learn new tasks throughout their lives, while at the same time maintaining robust memories of previous knowledge.

Original: This ability conﬂicts with traditional training procedures for artiﬁcial neural networks, which overwrite previous skills when optimizing new tasks (McCloskey & Cohen, 1989; Goodfellow et al., 2013).
Revised: This ability conﬂicts with traditional training procedures for artiﬁcial neural networks, which overwrite previous skills when optimizing new tasks (McCloskey & Cohen, 1989; Goodfellow et al., 2013).

Original: The ﬁeld of continual learning is dedicated to mitigate this crucial shortcoming of machine learning.
Revised: The ﬁeld of continual learning is dedicated to mitigate this crucial shortcoming of machine learning.

Original: It exposes neural networks to a sequence of distinct tasks.
Revised: It exposes neural networks to a sequence of distinct tasks.

Original: During training new tasks, the algorithm is not allowed to revisit old data, but should retain previous skills while remaining ﬂexible enough to also acquire new knowledge.
Revised: During training new tasks, the algorithm is not allowed to revisit old data, but should retain previous skills while remaining ﬂexible enough to also acquire new knowledge.

Original: An inﬂuential line of work to approach this challenge was introduced by Kirkpatrick et al.
Revised: An inﬂuential line of work to approach this challenge was introduced by Kirkpatrick et al.

Original: (2017), who proposed the ﬁrst regularisation-algorithm, Elastic Weight Consolidation (EWC).
Revised: (2017), who proposed the ﬁrst regularisation-algorithm, Elastic Weight Consolidation (EWC).

Original: After training a task, EWC measures the importance of each parameter and introduces an auxiliary loss penalising large changes in important parameters.
Revised: After training a task, EWC measures the importance of each parameter and introduces an auxiliary loss penalising large changes in important parameters.

Original: Naturally, this raises the question of how to measure this ‘importance’.
Revised: Naturally, this raises the question of how to measure this ‘importance’.

Original: While EWC uses the diagonal Fisher Information, two alternatives have been proposed:
Revised: While EWC uses the diagonal Fisher Information, two alternatives have been proposed:

Original: Synaptic Intelligence (SI, Zenke et al.
Revised: Synaptic Intelligence (SI, Zenke et al.

Original: (2017)) aims to attribute the decrease in loss during training to individual parameters and Memory Aware Synapses (MAS, Aljundi et al.
Revised: (2017)) aims to attribute the decrease in loss during training to individual parameters and Memory Aware Synapses (MAS, Aljundi et al.

Original: (2018)) introduces a heuristic measure of output sensitivity.
Revised: (2018)) introduces a heuristic measure of output sensitivity.

Original: Together, these three approaches have inspired many further regularisation-based approaches, including combinations of them (Chaudhry et al., 2018), reﬁnements (Huszár, 2018; Ritter et al., 2018; Chaudhry et al., 2018;
Revised: Together, these three approaches have inspired many further regularisation-based approaches, including combinations of them (Chaudhry et al., 2018), reﬁnements (Huszár, 2018; Ritter et al., 2018; Chaudhry et al., 2018;

Original: Yin et al., 2020), extensions (Schwarz et al., 2018; Liu et al., 2018; Park et al., 2019; Lee et al., 2020) and applications in different continual learning settings (Aljundi et al., 2019) as well as different domains of machine learning (Lan et al., 2019).
Revised: Yin et al., 2020), extensions (Schwarz et al., 2018; Liu et al., 2018; Park et al., 2019; Lee et al., 2020) and applications in different continual learning settings (Aljundi et al., 2019) as well as different domains of machine learning (Lan et al., 2019).

Original: Almost every new continual learning method compares to at least one of EWC, SI and MAS.
Revised: Almost every new continual learning method compares to at least one of EWC, SI and MAS.

Original: Despite their popularity and inﬂuence, basic practical and theoretical questions regarding these algorithms had previously been unanswered.
Revised: Despite their popularity and inﬂuence, basic practical and theoretical questions regarding these algorithms had previously been unanswered.

Original: Notably, it was unkown how similar these importance measures are.
Revised: Notably, it was unkown how similar these importance measures are.

Original: Additionally, for SI and MAS (and their follow-ups) there was no solid theoretical understanding of their effectiveness.
Revised: Additionally, for SI and MAS (and their follow-ups) there was no solid theoretical understanding of their effectiveness.

Original: Here, we close both these gaps through a theoretical analysis conﬁrmed by a series of carefully designed experiments on standard continual learning benchmarks.
Revised: Here, we close both these gaps through a theoretical analysis conﬁrmed by a series of carefully designed experiments on standard continual learning benchmarks.

Original: Our main ﬁndings can be summarised as follows:
Revised: Our main ﬁndings can be summarised as follows:

Original: (a) We show that MAS is almost identical to a natural variant of the Fisher Information.
Revised: (a) We show that MAS is almost identical to a natural variant of the Fisher Information.

Original: (b) We show that SI’s importance approximation is biased, that the bias is responsible for SI’s performance and that the bias, like MAS, is tightly linked to the Fisher Information.
Revised: (b) We show that SI’s importance approximation is biased, that the bias is responsible for SI’s performance and that the bias, like MAS, is tightly linked to the Fisher Information.

Original: (c) Our ﬁne-grained analysis leads to new baselines, including theoretically justiﬁed versions of MAS and SI as well as a signiﬁcantly faster version of EWC with equal performance.
Revised: (c) Our ﬁne-grained analysis leads to new baselines, including theoretically justiﬁed versions of MAS and SI as well as a signiﬁcantly faster version of EWC with equal performance.

Original: (d) Together, (a) and (b) show that all three regularisation approaches (and their follow-ups) rely on the Fisher Information.
Revised: (d) Together, (a) and (b) show that all three regularisation approaches (and their follow-ups) rely on the Fisher Information.

Original: This uniﬁes a large body of regularisation literature.
Revised: This uniﬁes a large body of regularisation literature.

Original: It also gives the ﬁrst solid theoretical justiﬁcation for the effectiveness of SI- and MAS-based algorithms.
Revised: It also gives the ﬁrst solid theoretical justiﬁcation for the effectiveness of SI- and MAS-based algorithms.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The problem of catastrophic forgetting in neural networks has been studied for many decades (McCloskey & Cohen, 1989; Ratcliff, 1990; French, 1999).
Revised: The problem of catastrophic forgetting in neural networks has been studied for many decades (McCloskey & Cohen, 1989; Ratcliff, 1990; French, 1999).

Original: In the context of deep learning, it received more attention again (Goodfellow et al., 2013; Srivastava et al., 2013).
Revised: In the context of deep learning, it received more attention again (Goodfellow et al., 2013; Srivastava et al., 2013).

Original: All previous work we are aware of proposes new or modiﬁed algorithms to tackle continual learning.
Revised: All previous work we are aware of proposes new or modiﬁed algorithms to tackle continual learning.

Original: No attention has been directed towards understanding or unifying existing methods
Revised: No attention has been directed towards understanding or unifying existing methods

Original: and we hope that our work will not remain the only effort of this kind.
Revised: and we hope that our work will not remain the only effort of this kind.

Original: We now review the broad body of continual learning algorithms.
Revised: We now review the broad body of continual learning algorithms.

Original: Following (Parisi et al., 2019), they are often categorised into regularisation-, replay- and architectural approaches.
Revised: Following (Parisi et al., 2019), they are often categorised into regularisation-, replay- and architectural approaches.

Original: Most regularisation methods will be reviewed more closely in the next section.
Revised: Most regularisation methods will be reviewed more closely in the next section.

Original: There we cover regularisation for standard neural nets, but the same idea has also been applied to bayesian neural networks (Nguyen et al., 2017; Swaroop et al., 2019).
Revised: There we cover regularisation for standard neural nets, but the same idea has also been applied to bayesian neural networks (Nguyen et al., 2017; Swaroop et al., 2019).

Original: An alternative approach (Mirzadeh et al., 2020) does not penalize deviating from important parameters, but rather modiﬁes training so that the network is robust to such deviations.
Revised: An alternative approach (Mirzadeh et al., 2020) does not penalize deviating from important parameters, but rather modiﬁes training so that the network is robust to such deviations.

Original: Replay methods refer to algorithms which either store a small sample or generate data of old distributions and use this data while training on new methods (Rebufﬁ et al., 2017; Lopez-Paz & Ranzato, 2017; Shin et al., 2017; Kemker & Kanan, 2017).
Revised: Replay methods refer to algorithms which either store a small sample or generate data of old distributions and use this data while training on new methods (Rebufﬁ et al., 2017; Lopez-Paz & Ranzato, 2017; Shin et al., 2017; Kemker & Kanan, 2017).

Original: These approaches can be seen as investigating how far standard i.i.d.-training can be relaxed towards the (highly non-i.i.d.)
Revised: These approaches can be seen as investigating how far standard i.i.d.-training can be relaxed towards the (highly non-i.i.d.)

Original: continual learning setting without losing too much performance.
Revised: continual learning setting without losing too much performance.

Original: They are interesting, but usually circumvent the original motivation of continual learning to maintain knowledge without accessing old distributions.
Revised: They are interesting, but usually circumvent the original motivation of continual learning to maintain knowledge without accessing old distributions.

Original: Intriguingly, the most effective way to use old data appears to be simply replaying it, i.e. mimicking training with i.i.d.
Revised: Intriguingly, the most effective way to use old data appears to be simply replaying it, i.e. mimicking training with i.i.d.

Original: batches sampled from all tasks simultaneously (Chaudhry et al., 2019).
Revised: batches sampled from all tasks simultaneously (Chaudhry et al., 2019).

Original: Architectural methods extend the network as new tasks arrive (Fernando et al., 2017; Li et al., 2019;
Revised: Architectural methods extend the network as new tasks arrive (Fernando et al., 2017; Li et al., 2019;

Original: Schwarz et al., 2018; Golkar et al., 2019; von Oswald et al., 2019).
Revised: Schwarz et al., 2018; Golkar et al., 2019; von Oswald et al., 2019).

Original: This can be seen as a study of how old parts of the network can be effectively used to solve new tasks and touches upon transfer learning.
Revised: This can be seen as a study of how old parts of the network can be effectively used to solve new tasks and touches upon transfer learning.

Original: Typically, it avoids the challenge of integrating new knowledge into an existing networks.
Revised: Typically, it avoids the challenge of integrating new knowledge into an existing networks.

Original: Finally, van de Ven & Tolias (2019); Hsu et al.
Revised: Finally, van de Ven & Tolias (2019); Hsu et al.

Original: Farquhar & Gal (2018) point out that different continual learning scenarios and assumptions with varying difﬁculty were used across the literature.
Revised: Farquhar & Gal (2018) point out that different continual learning scenarios and assumptions with varying difﬁculty were used across the literature.

Original:  R EVIEW OF E XISTING R EGULARISATION M ETHODS 3. F ORMAL D ESCRIPTION OF C ONTINUAL L EARNING
Revised:  R EVIEW OF E XISTING R EGULARISATION M ETHODS 3. F ORMAL D ESCRIPTION OF C ONTINUAL L EARNING

Original: In continual learning we are given K datasets D 1 , . . .
Revised: In continual learning we are given K datasets D 1 , . . .

Original: , D K sequentially.
Revised: , D K sequentially.

Original: When training a neural net with N parameters θ ∈ R N on dataset D k , we have no access to the previously seen datasets D 1: k − 1 .
Revised: When training a neural net with N parameters θ ∈ R N on dataset D k , we have no access to the previously seen datasets D 1: k − 1 .

Original: However, at test time the algorithms is tested on all K tasks, usually the average accuracy is taken as measure of the algorithms performance, but see also Chaudhry et al.
Revised: However, at test time the algorithms is tested on all K tasks, usually the average accuracy is taken as measure of the algorithms performance, but see also Chaudhry et al.

Original: 3. C OMMON F RAMEWORK FOR R EGULARISATION M ETHODS
Revised: 3. C OMMON F RAMEWORK FOR R EGULARISATION M ETHODS

Original: Regularisation based approaches introduced in Kirkpatrick et al.
Revised: Regularisation based approaches introduced in Kirkpatrick et al.

Original: Huszár (2018) protect previous memories by modifying the loss function L k related to dataset D k .
Revised: Huszár (2018) protect previous memories by modifying the loss function L k related to dataset D k .

Original: Let us denote the parameters obtained after ﬁnishing training on task k by θ ( k ) and let ω ( k ) ∈
Revised: Let us denote the parameters obtained after ﬁnishing training on task k by θ ( k ) and let ω ( k ) ∈

Original: R N be the parameters’ importances .
Revised: R N be the parameters’ importances .

Original: When training on task k , regularisation methods use the loss
Revised: When training on task k , regularisation methods use the loss

Original: [Equation]
Revised: [Equation]

Original: where c > 0 is a hyperparameter.
Revised: where c > 0 is a hyperparameter.

Original: The ﬁrst term L k is the standard (e.g. cross entropy) loss of task k .
Revised: The ﬁrst term L k is the standard (e.g. cross entropy) loss of task k .

Original: The second term ensures that the parameters do not move away too far from their previous values.
Revised: The second term ensures that the parameters do not move away too far from their previous values.

Original: Usually, ω ( k )
Revised: Usually, ω ( k )

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: , where ω i is the importance for the most recently learned task k .
Revised: , where ω i is the importance for the most recently learned task k .

Original: 3. R EVIEW OF EWC, MAS, SI AND FOLLOW - UPS
Revised: 3. R EVIEW OF EWC, MAS, SI AND FOLLOW - UPS

Original: Elastic Weight Consolidation (Kirkpatrick et al., 2017) uses the diagonal of the Fisher Information as importance measure.
Revised: Elastic Weight Consolidation (Kirkpatrick et al., 2017) uses the diagonal of the Fisher Information as importance measure.

Original: It is evaluated at the end of training a given task.
Revised: It is evaluated at the end of training a given task.

Original: To deﬁne the Fisher
Revised: To deﬁne the Fisher

Original: Information Matrix F , we omit the task index k and simply consider a dataset X .
Revised: Information Matrix F , we omit the task index k and simply consider a dataset X .

Original: For each datapoint
Revised: For each datapoint

Original: X ∈ X the network predicts a probability distribution q X over the set of labels L , where we suppress dependence of q X on θ for simplicity of notation.
Revised: X ∈ X the network predicts a probability distribution q X over the set of labels L , where we suppress dependence of q X on θ for simplicity of notation.

Original: We denote the predicted probability of class y by q X ( y ) for each y ∈ L .
Revised: We denote the predicted probability of class y by q X ( y ) for each y ∈ L .

Original: Let us assume that we minimize the negative log-likelihood log q X ( y ) and write g ( X, y )
Revised: Let us assume that we minimize the negative log-likelihood log q X ( y ) and write g ( X, y )

Original: = − ∂ log q X ( y )
Revised: = − ∂ log q X ( y )

Original: ∂θ for its gradient.
Revised: ∂θ for its gradient.

Original: Then the Fisher Information F is given by
Revised: Then the Fisher Information F is given by

Original: [Equation]
Revised: [Equation]

Original: Taking only the diagonal Fisher means ω
Revised: Taking only the diagonal Fisher means ω

Original: [Equation]
Revised: [Equation]

Original: The Empirical Fisher Information replaces the expectation over y ∼ q X by the (deterministic) label y given by the labels of the dataset.
Revised: The Empirical Fisher Information replaces the expectation over y ∼ q X by the (deterministic) label y given by the labels of the dataset.

Original: We also deﬁne the Predicted Fisher Information as taking the maximum-likelihood predicted label, i.e. the argmax of q X ( · ) .
Revised: We also deﬁne the Predicted Fisher Information as taking the maximum-likelihood predicted label, i.e. the argmax of q X ( · ) .

Original: When the network classiﬁes images correctly and conﬁdently, these three versions clearly become increasingly similar.
Revised: When the network classiﬁes images correctly and conﬁdently, these three versions clearly become increasingly similar.

Original: Under the assumption that the learned label distribution q X is the real label distribution the Fisher equals the Hessian of the loss (Martens, 2014; Pascanu & Bengio, 2013).
Revised: Under the assumption that the learned label distribution q X is the real label distribution the Fisher equals the Hessian of the loss (Martens, 2014; Pascanu & Bengio, 2013).

Original: Its use for continual learning has a clean, bayesian, theoretical interpretation (Kirkpatrick et al., 2017; Huszár, 2018).
Revised: Its use for continual learning has a clean, bayesian, theoretical interpretation (Kirkpatrick et al., 2017; Huszár, 2018).

Original: Memory Aware Synapses (Aljundi et al., 2018) heuristically argues that the sensitivity of the ﬁnal layer with respect to a parameter should be used as the parameter’s importance.
Revised: Memory Aware Synapses (Aljundi et al., 2018) heuristically argues that the sensitivity of the ﬁnal layer with respect to a parameter should be used as the parameter’s importance.

Original: This sensitivity is evaluated after training a given task.
Revised: This sensitivity is evaluated after training a given task.

Original: Denoting, as before, the ﬁnal layer of learned probabilities by q X , this is implemented as (using a euclidean norm)
Revised: Denoting, as before, the ﬁnal layer of learned probabilities by q X , this is implemented as (using a euclidean norm)

Original: [Equation]
Revised: [Equation]

Original: Synaptic Intelligence (Zenke et al., 2017) approximates the contribution of each parameter to the decrease in loss and uses this contribution as importance.
Revised: Synaptic Intelligence (Zenke et al., 2017) approximates the contribution of each parameter to the decrease in loss and uses this contribution as importance.

Original: To formalise the ‘contribution of a parameter’, let us denote the parameters at time t by θ ( t ) and the loss by L ( t ) .
Revised: To formalise the ‘contribution of a parameter’, let us denote the parameters at time t by θ ( t ) and the loss by L ( t ) .

Original: If the parameters follow a smooth trajectory in parameter space, we can write the decrease in loss from time 0 to T as
Revised: If the parameters follow a smooth trajectory in parameter space, we can write the decrease in loss from time 0 to T as

Original: [Equation]
Revised: [Equation]

Original: The i -th summand in (2) can be interpreted as the contribution of parameter θ i to the decrease in loss.
Revised: The i -th summand in (2) can be interpreted as the contribution of parameter θ i to the decrease in loss.

Original: While we cannot evaluate the integral precisely, we can use a ﬁrst order approximation to obtain the importances.
Revised: While we cannot evaluate the integral precisely, we can use a ﬁrst order approximation to obtain the importances.

Original: To do so, we write ∆ i ( t )
Revised: To do so, we write ∆ i ( t )

Original: =
Revised: =

Original: ( θ i ( t + 1) − θ i ( t )) for an approximation of θ (cid:48)
Revised: ( θ i ( t + 1) − θ i ( t )) for an approximation of θ (cid:48)

Original: i ( t ) dt and get
Revised: i ( t ) dt and get

Original: [Equation]
Revised: [Equation]

Original: Thus, we have (cid:80) i ˜ ω
Revised: Thus, we have (cid:80) i ˜ ω

Original: i ( SI ) ≈
Revised: i ( SI ) ≈

Original: L (0) − L ( T ) .
Revised: L (0) − L ( T ) .

Original: In addition, SI rescales its importances as follows
Revised: In addition, SI rescales its importances as follows

Original: [Equation]
Revised: [Equation]

Original: Zenke et al.
Revised: Zenke et al.

Original: (2017) use additional assumptions to justify this importance.
Revised: (2017) use additional assumptions to justify this importance.

Original: One of the them – using full batch gradient descent – is violated in practice and we will show that this has an important effect.
Revised: One of the them – using full batch gradient descent – is violated in practice and we will show that this has an important effect.

Original: Follow up work.
Revised: Follow up work.

Original: We presented a version of EWC due to Huszár (2018) and tested in (Chaudhry et al., 2018; Schwarz et al., 2018).
Revised: We presented a version of EWC due to Huszár (2018) and tested in (Chaudhry et al., 2018; Schwarz et al., 2018).

Original: It is theoretically more sound and was shown to perform better.
Revised: It is theoretically more sound and was shown to perform better.

Original: Chaudhry et al.
Revised: Chaudhry et al.

Original: (2018) combine EWC with a ‘KL-rescaled’-SI.
Revised: (2018) combine EWC with a ‘KL-rescaled’-SI.

Original: Ritter et al.
Revised: Ritter et al.

Original: (2018) use a block-diagonal (rather than diagonal) approximation of the Fisher; Yin et al.
Revised: (2018) use a block-diagonal (rather than diagonal) approximation of the Fisher; Yin et al.

Original: (2020) use the full Hessian matrix for small networks.
Revised: (2020) use the full Hessian matrix for small networks.

Original: Liu et al.
Revised: Liu et al.

Original: (2018) rotate the network to diagonalise the most recent Fisher Matrix, Park et al.
Revised: (2018) rotate the network to diagonalise the most recent Fisher Matrix, Park et al.

Original: (2019) modify the loss of SI and Lee et al.
Revised: (2019) modify the loss of SI and Lee et al.

Original: (2020) aim to account for batch-normalisation.
Revised: (2020) aim to account for batch-normalisation.

Original:  M EMORY A WARE S YNAPSES (MAS) APPROXIMATES A BSOLUTE F ISHER
Revised:  M EMORY A WARE S YNAPSES (MAS) APPROXIMATES A BSOLUTE F ISHER

Original: Here, we explain why – despite its different motivation – MAS is almost linearly dependent of the diagonal Absolute Fisher , a variant of the diagonal Fisher taking absolute values of gradients rather than squares.
Revised: Here, we explain why – despite its different motivation – MAS is almost linearly dependent of the diagonal Absolute Fisher , a variant of the diagonal Fisher taking absolute values of gradients rather than squares.

Original: We ﬁrst present the theory and then carry out careful empirical conﬁrmations of the underlying assumptions.
Revised: We ﬁrst present the theory and then carry out careful empirical conﬁrmations of the underlying assumptions.

Original: For a tabular summary of algorithms and baselines see Appendix B.1.
Revised: For a tabular summary of algorithms and baselines see Appendix B.1.

Original: 4. T HEORETICAL R ELATION OF MAS AND A BSOLUTE F ISHER
Revised: 4. T HEORETICAL R ELATION OF MAS AND A BSOLUTE F ISHER

Original: We take a closer look at the deﬁnition of the importance of MAS.
Revised: We take a closer look at the deﬁnition of the importance of MAS.

Original: We apply linearity of derivatives, the chain rule and write y 

Revised: We apply linearity of derivatives, the chain rule and write y

Original: = arg max q X to obtain (omitting expectation over X ∼ X for brevity)
Revised: = arg max q X to obtain (omitting expectation over X ∼ X for brevity)

Original: MAS:
Revised: MAS:

Original: [Equation]
Revised: [Equation]

Original: Here we made the assumption that the sum is dominated by its maximum-likelihood label y 0 , which should be the case if the network classiﬁes its images conﬁdently, i.e. q ( y 0 )
Revised: Here we made the assumption that the sum is dominated by its maximum-likelihood label y 0 , which should be the case if the network classiﬁes its images conﬁdently, i.e. q ( y 0 )

Original: (cid:29) q ( y ) for y (cid:54) =
Revised: (cid:29) q ( y ) for y (cid:54) =

Original: y 0 .
Revised: y 0 .

Original: Recall that the importance is measured at the end of training, so that our assumption is justiﬁed if the task has been learned successfully.
Revised: Recall that the importance is measured at the end of training, so that our assumption is justiﬁed if the task has been learned successfully.

Original: Using the same assumption for the Absolute Fisher we obtain
Revised: Using the same assumption for the Absolute Fisher we obtain

Original: Abs.
Revised: Abs.

Original: Fisher
Revised: Fisher

Original: [Equation]
Revised: [Equation]

Original: This shows the similarity between MAS and the Absolute Fisher.
Revised: This shows the similarity between MAS and the Absolute Fisher.

Original: The only difference is a factor of 2 q X ( y 0 ) , which we can assume to be approximately constant as the model learned to classify training images conﬁdently.
Revised: The only difference is a factor of 2 q X ( y 0 ) , which we can assume to be approximately constant as the model learned to classify training images conﬁdently.

Original: Even with a pessimistic guess that q X ( y 0 ) is in a range of 0.5 (rather inconﬁdent) and 1.0 (absolutely conﬁdent), the two measures would be highly correlated.
Revised: Even with a pessimistic guess that q X ( y 0 ) is in a range of 0.5 (rather inconﬁdent) and 1.0 (absolutely conﬁdent), the two measures would be highly correlated.

Original: 4. E MPIRICAL R ELATION OF MAS AND A BSOLUTE F ISHER
Revised: 4. E MPIRICAL R ELATION OF MAS AND A BSOLUTE F ISHER

Original: Our theoretical derivation used two approximations, both of which we now explicitly check empirically on standard benchmarks; see Section 6 for a summary of our experimental setup.
Revised: Our theoretical derivation used two approximations, both of which we now explicitly check empirically on standard benchmarks; see Section 6 for a summary of our experimental setup.

Original: Max-Likeilhood Approximation First, in equation 5 we assumed that sum over the labels is dominated by its maximum likelihood label y 0 .
Revised: Max-Likeilhood Approximation First, in equation 5 we assumed that sum over the labels is dominated by its maximum likelihood label y 0 .

Original: To assess the validity of this, we directly compare the entire sum (MAS) with the part only using the max-likelihood label (MAS-MAX).
Revised: To assess the validity of this, we directly compare the entire sum (MAS) with the part only using the max-likelihood label (MAS-MAX).

Original: The results in
Revised: The results in

Original: Figure 1 show scatter plots and a summary of the pearson correlations between MAS and MAS-MAX on each task.
Revised: Figure 1 show scatter plots and a summary of the pearson correlations between MAS and MAS-MAX on each task.

Original: On P-MNIST the correlations range from 0.95 to 0.99, on CIFAR they are always above 0.99.
Revised: On P-MNIST the correlations range from 0.95 to 0.99, on CIFAR they are always above 0.99.

Original: This shows that MAS and MAS-MAX are nearly equal as claimed.
Revised: This shows that MAS and MAS-MAX are nearly equal as claimed.

Original: Absolute Fisher Approximation Second, we assumed that the factor 2 q X ( y 0 ) distinguishing MASMAX (RHS of equation 5) and Absolute Fisher is roughly constant.
Revised: Absolute Fisher Approximation Second, we assumed that the factor 2 q X ( y 0 ) distinguishing MASMAX (RHS of equation 5) and Absolute Fisher is roughly constant.

Original: Recall that even if q X ( y 0 ) varied between 0.5 and 1.0, the two measures would have large correlations.
Revised: Recall that even if q X ( y 0 ) varied between 0.5 and 1.0, the two measures would have large correlations.

Original: The data in Figureshows that we are far above this pessimistic regime with correlations between MAS-MAX and AF always above 0.98, showing that MAS-MAX indeed approximates AF.
Revised: The data in Figureshows that we are far above this pessimistic regime with correlations between MAS-MAX and AF always above 0.98, showing that MAS-MAX indeed approximates AF.

Original: The data also shows that the resulting correlation between MAS and AF is large (between 0.9-0.97 on MNIST, >0.99 on CIFAR), conﬁrming that MAS is an approximation of AF.
Revised: The data also shows that the resulting correlation between MAS and AF is large (between 0.9-0.97 on MNIST, >0.99 on CIFAR), conﬁrming that MAS is an approximation of AF.

Original: Continual Learning Performance Experiment
Revised: Continual Learning Performance Experiment

Original: As an additional check of our hypothesis, we implemented continual learning baselines based on MAS, MAS-MAX, AF and compared their performances.
Revised: As an additional check of our hypothesis, we implemented continual learning baselines based on MAS, MAS-MAX, AF and compared their performances.

Original: The results in Table 1(1) show that all baselines have the same performance, further supporting the claim that MAS is effective because it approximates the Absolute Fisher.
Revised: The results in Table 1(1) show that all baselines have the same performance, further supporting the claim that MAS is effective because it approximates the Absolute Fisher.

Original:  S YNAPTIC I NTELLIGENCE APPROXIMATES (O NLINE ) A BSOLUTE F ISHER
Revised:  S YNAPTIC I NTELLIGENCE APPROXIMATES (O NLINE ) A BSOLUTE F ISHER

Original: Here, we explain why the importance of SI is related to the Absolute Fisher, despite the apparent contrast between SI’s path integral and the Fisher Information.
Revised: Here, we explain why the importance of SI is related to the Absolute Fisher, despite the apparent contrast between SI’s path integral and the Fisher Information.

Original: The theoretical part ﬁrst identiﬁes the bias of SI when approximating the path integral.
Revised: The theoretical part ﬁrst identiﬁes the bias of SI when approximating the path integral.

Original: It then relates the bias to the Absolute Fisher.
Revised: It then relates the bias to the Absolute Fisher.

Original: As before, we carefully design baselines to validate each assumption of our analysis empirically.
Revised: As before, we carefully design baselines to validate each assumption of our analysis empirically.

Original: 5. B IAS OF S YNAPTIC I NTELLIGENCE
Revised: 5. B IAS OF S YNAPTIC I NTELLIGENCE

Original: To calculate ω (SI) (equation 3), we need to calculate the product p
Revised: To calculate ω (SI) (equation 3), we need to calculate the product p

Original: = ∂L ( t ) ∂θ · ∆( t ) at each step t .
Revised: = ∂L ( t ) ∂θ · ∆( t ) at each step t .

Original: Since evaluating the full gradient ∂L ( t ) ∂θ is prohibitively expensive, SI uses a stochastic mini-batch gradient.
Revised: Since evaluating the full gradient ∂L ( t ) ∂θ is prohibitively expensive, SI uses a stochastic mini-batch gradient.

Original: The resulting estimate of p is biased since the same mini-batch is used for the parameter update ∆( t ) and the estimate of ∂L ( t )
Revised: The resulting estimate of p is biased since the same mini-batch is used for the parameter update ∆( t ) and the estimate of ∂L ( t )

Original: ∂θ .
Revised: ∂θ .

Original: We now give the calculations detailing the argument above.
Revised: We now give the calculations detailing the argument above.

Original: For ease of exposition, let us assume that the network is optimized using vanilla SGD with learning rate 1 .
Revised: For ease of exposition, let us assume that the network is optimized using vanilla SGD with learning rate 1 .

Original: Given a mini-batch, denote its gradient estimate by g + σ , where g = ∂L ( t )
Revised: Given a mini-batch, denote its gradient estimate by g + σ , where g = ∂L ( t )

Original: ∂θ denotes the real gradient and σ the mini-batch noise.
Revised: ∂θ denotes the real gradient and σ the mini-batch noise.

Original: The parameter update equals ∆( t ) =
Revised: The parameter update equals ∆( t ) =

Original: g + σ .
Revised: g + σ .

Original: Thus, our product p should be p = g · ( g + σ ) .
Revised: Thus, our product p should be p = g · ( g + σ ) .

Original: However, using g + σ , which was used for the parameter update, to estimate ∂L ( t ) ∂θ i results in p biased = ( g + σ ) 2 .
Revised: However, using g + σ , which was used for the parameter update, to estimate ∂L ( t ) ∂θ i results in p biased = ( g + σ ) 2 .

Original: Thus, the gradient noise introduces a bias of E [ σ 2 + σg ] =
Revised: Thus, the gradient noise introduces a bias of E [ σ 2 + σg ] =

Original: E [ σ 2 ] .
Revised: E [ σ 2 ] .

Original: Unbiased Synaptic Intelligence.
Revised: Unbiased Synaptic Intelligence.

Original: Having understood the bias, we can design an unbiased estimate by using two independent mini-batches to calculate the parameter update and to estimate g .
Revised: Having understood the bias, we can design an unbiased estimate by using two independent mini-batches to calculate the parameter update and to estimate g .

Original: This way we get ∆( t ) =
Revised: This way we get ∆( t ) =

Original: g + σ and an estimate g + σ (cid:48) for g from an independent mini-batch with independent noise σ (cid:48) .
Revised: g + σ and an estimate g + σ (cid:48) for g from an independent mini-batch with independent noise σ (cid:48) .

Original: We obtain p unbiased =
Revised: We obtain p unbiased =

Original: ( g + σ (cid:48) )
Revised: ( g + σ (cid:48) )

Original: · ( g + σ ) which in expectation equals p = g · ( g + σ ) .
Revised: · ( g + σ ) which in expectation equals p = g · ( g + σ ) .

Original: Based on this we deﬁne an unbiased importance measure
Revised: Based on this we deﬁne an unbiased importance measure

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Bias-Only version of SI.
Revised: Bias-Only version of SI.

Original: To isolate the bias, we can take the difference between biased and unbiased estimate.
Revised: To isolate the bias, we can take the difference between biased and unbiased estimate.

Original: Concretely, this gives an importance which only measures the bias of SI
Revised: Concretely, this gives an importance which only measures the bias of SI

Original: [Equation]
Revised: [Equation]

Original: Observe that this estimate multiplies the parameter-update ∆( t ) with nothing but stochastic gradient noise.
Revised: Observe that this estimate multiplies the parameter-update ∆( t ) with nothing but stochastic gradient noise.

Original: From the perspective of the SI path-integral, this should be meaningless and perform poorly.
Revised: From the perspective of the SI path-integral, this should be meaningless and perform poorly.

Original: 5. R ELATION OF B IAS OF SI TO A BSOLUTE F ISHER
Revised: 5. R ELATION OF B IAS OF SI TO A BSOLUTE F ISHER

Original: The bias of SI depends on the optimizer used.
Revised: The bias of SI depends on the optimizer used.

Original: The original SI-paper (and we) uses Adam (Kingma & Ba, 2014) and we now analyse the inﬂuence of this choice in detail.
Revised: The original SI-paper (and we) uses Adam (Kingma & Ba, 2014) and we now analyse the inﬂuence of this choice in detail.

Original: Recall that ω ( SI ) is a sum over terms ∂L ( t ) ∂θ · ∆( t ) , where ∆( t )
Revised: Recall that ω ( SI ) is a sum over terms ∂L ( t ) ∂θ · ∆( t ) , where ∆( t )

Original: = θ ( t + 1) − θ ( t ) is the parameter update at time t .
Revised: = θ ( t + 1) − θ ( t ) is the parameter update at time t .

Original: Both terms, ∂L ( t )
Revised: Both terms, ∂L ( t )

Original: ∂θ as well as ∆( t ) , are computed using the same mini-batch.
Revised: ∂θ as well as ∆( t ) , are computed using the same mini-batch.

Original: Given a stochastic gradient g t + σ t ,
Revised: Given a stochastic gradient g t + σ t ,

Original: Adam keeps an exponential average of the gradient m t =
Revised: Adam keeps an exponential average of the gradient m t =

Original: (1 − β 1 )( g t + σ t ) + β 1 m t − 1 as well as the squared gradient v t =
Revised: (1 − β 1 )( g t + σ t ) + β 1 m t − 1 as well as the squared gradient v t =

Original: (1 − β 2 )( g t + σ t ) 2 + β 2 v t − 1 .
Revised: (1 − β 2 )( g t + σ t ) 2 + β 2 v t − 1 .

Original: Ignoring minor normalisations and the learning rate, the parameter update is ∆( t )
Revised: Ignoring minor normalisations and the learning rate, the parameter update is ∆( t )

Original: = m
Revised: = m

Original: t / ( √ v t + (cid:15) ) , with β 1 = 0 .
Revised: t / ( √ v t + (cid:15) ) , with β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: Thus, ∂L
Revised: Thus, ∂L

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here, we made Assumption 1 that the gradient noise is larger than the gradient, or more precisely: (1 − β 1 ) σ 2 t (cid:29) β 1 m t − 1 g t
Revised: Here, we made Assumption 1 that the gradient noise is larger than the gradient, or more precisely: (1 − β 1 ) σ 2 t (cid:29) β 1 m t − 1 g t

Original: (we ignore σ t m t − 1 since E [ σ t m t − 1 ] = 

Revised: (we ignore σ t m t − 1 since E [ σ t m t − 1 ] =

Original: and since we average SI over many time steps).
Revised: and since we average SI over many time steps).

Original: Assumption 1 is equivalent to the bias of SI being larger than its unbiased part as detailed in Appendix D.1.
Revised: Assumption 1 is equivalent to the bias of SI being larger than its unbiased part as detailed in Appendix D.1.

Original: Experiments in Section 5.3 provide strong support for this assumption.
Revised: Experiments in Section 5.3 provide strong support for this assumption.

Original: Next, consider v t .
Revised: Next, consider v t .

Original: It is a slowly moving exponential average of ( g t + σ t ) 2 and is thus approximately E [( g t + σ t ) 2 ] .
Revised: It is a slowly moving exponential average of ( g t + σ t ) 2 and is thus approximately E [( g t + σ t ) 2 ] .

Original: It is reasonable to expect that √ v t is roughly equal to | g t + σ t | .
Revised: It is reasonable to expect that √ v t is roughly equal to | g t + σ t | .

Original: This clearly does not hold for all possible distributions of g t + σ t , but is unlikely to be violated by non adversarially chosen, realistic noise distribution.
Revised: This clearly does not hold for all possible distributions of g t + σ t , but is unlikely to be violated by non adversarially chosen, realistic noise distribution.

Original: With equation 7 and Assumption 2 that √ v t
Revised: With equation 7 and Assumption 2 that √ v t

Original: ≈ | g t + σ t | , we get
Revised: ≈ | g t + σ t | , we get

Original: [Equation]
Revised: [Equation]

Original: An alternative way to arrive at equation 8 is following the analysis of Balles & Hennig (2018), who show that the update of Adam often depends on the sign of the moment m t rather than its magnitude.
Revised: An alternative way to arrive at equation 8 is following the analysis of Balles & Hennig (2018), who show that the update of Adam often depends on the sign of the moment m t rather than its magnitude.

Original: Summary.
Revised: Summary.

Original: Equation 8 shows that SI approximates the(Empirical) Online Absolute Fisher , which averages the (empirical) Absolute Fisher ‘online’ along the training trajectory of the parameters rather than evaluating it only at the ﬁnal parameter-point.
Revised: Equation 8 shows that SI approximates the Online Absolute Fisher , which averages the (empirical) Absolute Fisher ‘online’ along the training trajectory of the parameters rather than evaluating it only at the ﬁnal parameter-point.

Original: We point out one subtle difference between equation 8 and AF, namely that equation 8 takes absolute values of mini-batch gradients rather than single-image gradients (like AF).
Revised: We point out one subtle difference between equationand AF, namely that equation 8 takes absolute values of mini-batch gradients rather than single-image gradients (like AF).

Original: In Appendix D.2 we show why this is negligible under our Assumption 1 of large gradient noise.
Revised: In Appendix D.2 we show why this is negligible under our Assumption 1 of large gradient noise.

Original: (In fact, we use this insight to signiﬁcantly speed-up EWC.)
Revised: (In fact, we use this insight to signiﬁcantly speed-up EWC.)

Original: Relation of OnAF to AF.
Revised: Relation of OnAF to AF.

Original: To see if SI, like MAS, is related to the Absolute Fisher (AF), it remains to explore the relation of OnAF and AF.
Revised: To see if SI, like MAS, is related to the Absolute Fisher (AF), it remains to explore the relation of OnAF and AF.

Original: If the latter two are similar, it is clear from the above that
Revised: If the latter two are similar, it is clear from the above that

Original: SI is similar to AF.
Revised: SI is similar to AF.

Original: Theoretically, in the limit of small learning rates or small Hessian (of the loss),
Revised: Theoretically, in the limit of small learning rates or small Hessian (of the loss),

Original: OnAF and AF become equal.
Revised: OnAF and AF become equal.

Original: Moreover, we point out that the fact that Adam’s second moment estimate usually works well with a very slowly moving average ( β 2 = 0 . 999 ) indicates that the squared gradient changes slowly in many settings, suggesting that the Absolute Fisher also changes slowly, so that OnAF and AF are similar.
Revised: Moreover, we point out that the fact that Adam’s second moment estimate usually works well with a very slowly moving average ( β 2 = 0 . 999 ) indicates that the squared gradient changes slowly in many settings, suggesting that the Absolute Fisher also changes slowly, so that OnAF and AF are similar.

Original: Therefore, we hypothesise that OnAF and AF are similar and empirically conﬁrm this claim in Section 5.3.
Revised: Therefore, we hypothesise that OnAF and AF are similar and empirically conﬁrm this claim in Section 5.3.

Original: Inﬂuence of Regularisation The derivation above assumes that the parameter update ∆( t ) is given by the gradients of the current task, ignoring the auxiliary regularisation loss.
Revised: Inﬂuence of Regularisation The derivation above assumes that the parameter update ∆( t ) is given by the gradients of the current task, ignoring the auxiliary regularisation loss.

Original: The latter will change update direction and momentum of the optimizer.
Revised: The latter will change update direction and momentum of the optimizer.

Original: Thus, strong regularisation will make the relation between SI and OnAF noisy, but not abolish it.
Revised: Thus, strong regularisation will make the relation between SI and OnAF noisy, but not abolish it.

Original: This will be conﬁrmed empirically in Section 5.3.
Revised: This will be conﬁrmed empirically in Section 5.3.

Original: 5. E MPRICAL I NVESTIGATION OF SI, ITS BIAS AND A BSOLUTE F ISHER
Revised: 5. E MPRICAL I NVESTIGATION OF SI, ITS BIAS AND A BSOLUTE F ISHER

Original: Bias dominates SI & checking Assumption 1.
Revised: Bias dominates SI & checking Assumption 1.

Original: According to the motivation of SI the sum of importances (over parameters) (cid:80) i ˜ ω
Revised: According to the motivation of SI the sum of importances (over parameters) (cid:80) i ˜ ω

Original: i (SI) should track the decrease in loss L (0) − L ( T ) , see equation 2.
Revised: i (SI) should track the decrease in loss L (0) − L ( T ) , see equation 2.

Original: Therefore, we investigated the summed importances for SI, its unbiased version SIU and for an approximation of the path integral based on the full training set gradient (rather than approximating it).
Revised: Therefore, we investigated the summed importances for SI, its unbiased version SIU and for an approximation of the path integral based on the full training set gradient (rather than approximating it).

Original: The results, Figure 2 (left) show: (1) the bias is 5-6 times larger than the unbiased part; (2) using an unbiased gradient estimate and the full gradient gives almost identical sums, supporting the validity of the unbiased estimator.
Revised: The results, Figure 2 (left) show: (1) the bias is 5-6 times larger than the unbiased part; (2) using an unbiased gradient estimate and the full gradient gives almost identical sums, supporting the validity of the unbiased estimator.

Original: Note that even the unbiased ﬁrst order approximation of the path integral overestimates the decrease in loss.
Revised: Note that even the unbiased ﬁrst order approximation of the path integral overestimates the decrease in loss.

Original: This is consistent with previous studies showing that the loss has positive curvature (Jastrzebski et al., 2018; Lan et al., 2019).
Revised: This is consistent with previous studies showing that the loss has positive curvature (Jastrzebski et al., 2018; Lan et al., 2019).

Original: We emphasise that the bias (difference between SI and SIU) in Figure 2 (left) is due to the term (1 − β 1 ) σ 2 t and that this bias is considerably larger than the unbiased part.
Revised: We emphasise that the bias (difference between SI and SIU) in Figure 2 (left) is due to the term (1 − β 1 ) σ 2 t and that this bias is considerably larger than the unbiased part.

Original: This is direct, strong evidence of Assumption 1 (see
Revised: This is direct, strong evidence of Assumption 1 (see

Original: Appendix D.1 for full calculation).
Revised: Appendix D.1 for full calculation).

Original: Additionally, a detailed analysis of the gradient noise in Appendix
Revised: Additionally, a detailed analysis of the gradient noise in Appendix

Original: E gives further support for Assumption 1.
Revised: E gives further support for Assumption 1.

Original: Results on CIFAR are analogous (Appendix Figure G.6 ).
Revised: Results on CIFAR are analogous (Appendix Figure G.6 ).

Original: Bias explains SI’s performance.
Revised: Bias explains SI’s performance.

Original: We saw that the bias is much larger than the unbiased part.
Revised: We saw that the bias is much larger than the unbiased part.

Original: But how does it inﬂuence SI’s performance?
Revised: But how does it inﬂuence SI’s performance?

Original: To quantify this, we compared SI to its unbiased version
Revised: To quantify this, we compared SI to its unbiased version

Original: SIU and the bias-only version SIB.
Revised: SIU and the bias-only version SIB.

Original: Note that SIB is completely independent of the path integral motivating SI, only measures gradient noise and therefore should perform poorly according to SI’s original motivation.
Revised: Note that SIB is completely independent of the path integral motivating SI, only measures gradient noise and therefore should perform poorly according to SI’s original motivation.

Original: However, the results in Table 1(2), reveal the opposite: Removing the bias reduces performance of SI (SIU is worse), whereas isolating the bias does not affect or slightly improve performance (difference between SIB and SI on CIFAR is statistically signiﬁcant, p < 

Revised: However, the results in Table 1(2), reveal the opposite: Removing the bias reduces performance of SI (SIU is worse), whereas isolating the bias does not affect or slightly improve performance (difference between SIB and SI on CIFAR is statistically signiﬁcant, p <

Original: .two-sided t-test).
Revised: .two-sided t-test).

Original: This demonstrates that SI relies on its bias for its continual learning performance.
Revised: This demonstrates that SI relies on its bias for its continual learning performance.

Original: Bias and Fisher & Checking Assumption 2.
Revised: Bias and Fisher & Checking Assumption 2.

Original: As additional experiment, we checked how much the relation of SI to the Fisher depends on the bias.
Revised: As additional experiment, we checked how much the relation of SI to the Fisher depends on the bias.

Original: Our theory predicts that the relation between SI and
Revised: Our theory predicts that the relation between SI and

Original: OnAF is mostly based on the bias.
Revised: OnAF is mostly based on the bias.

Original: Figure 2 (mid) conﬁrms this, showing a strong correlation between
Revised: Figure 2 (mid) conﬁrms this, showing a strong correlation between

Original: SI, SIB and OnAF and a weaker correlation between SIU and OnAF.
Revised: SI, SIB and OnAF and a weaker correlation between SIU and OnAF.

Original: Note that the correlation of
Revised: Note that the correlation of

Original: OnAF and SI clearly supports Assumption 2; results on CIFAR are similar (c.f. Figure G.1), but with a stronger inﬂuence of regularisation as seen in Figure 2 (right) and full text below.
Revised: OnAF and SI clearly supports Assumption 2; results on CIFAR are similar (c.f. Figure G.1), but with a stronger inﬂuence of regularisation as seen in Figure 2 (right) and full text below.

Original: Relation of SI, OnAF to AF.
Revised: Relation of SI, OnAF to AF.

Original: Next, we investigated the relation between OnAF and AF, ﬁnding strong correlations: between 0.73-0.87 on MNIST and 0.86-0.95 on CIFAR, see Figure G.3 in the appendix.
Revised: Next, we investigated the relation between OnAF and AF, ﬁnding strong correlations: between 0.73-0.87 on MNIST and 0.86-0.95 on CIFAR, see Figure G.3 in the appendix.

Original: Figure 2 (mid panel for MNIST, right panel for CIFAR, blue) shows that this carries over to relating SI and AF.
Revised: Figure 2 (mid panel for MNIST, right panel for CIFAR, blue) shows that this carries over to relating SI and AF.

Original: An additional control in Appendix G.2.1 shows that this observation is not an artefact of long training times.
Revised: An additional control in Appendix G.2.1 shows that this observation is not an artefact of long training times.

Original: These results conﬁrm that SI is closely related to AF.
Revised: These results conﬁrm that SI is closely related to AF.

Original: Effect of Regularisation on SI.
Revised: Effect of Regularisation on SI.

Original: Lastly, we assessed our derivation that large regularisation will diminish the correlation between SI and OnAF.
Revised: Lastly, we assessed our derivation that large regularisation will diminish the correlation between SI and OnAF.

Original: To this end, we compared standard SI to two variants of it which have less strong regularisation.
Revised: To this end, we compared standard SI to two variants of it which have less strong regularisation.

Original: The ﬁrst control simply sets regularisation strength to c = 0 .
Revised: The ﬁrst control simply sets regularisation strength to c = 0 .

Original: The second control refrains from re-initialising the network weights after each task (exactly as inthe original SI paper, albeit with slightly worse validation performance than our version).
Revised: The second control refrains from re-initialising the network weights after each task (exactly as in original SI, albeit with slightly worse validation performance than the version with re-initialisation).

Original: In the second setting the current parameters θ never move too far from their old value θ ( k − 1) , implying smaller gradients from the quadratic regularisation loss, and also meaning that a smaller value of c = 0 .
Revised: In the second setting the current parameters θ never move too far from their old value θ ( k − 1) , implying smaller gradients from the quadratic regularisation loss, and also meaning that a smaller value of c = 0 .

Original: 5 is optimal.
Revised: 5 is optimal.

Original: We see that for both controls with weak regularisation the relation to OnAF is larger than for standard-SI with strong regularisation (Figure 2 (right)), verifying our theory.Appendix Table
Revised: We see that for both controls with weak regularisation the relation to OnAF is larger than for standard-SI with strong regularisation (Figure 2 (right)), consistent with our theory.

Original: Efﬁcient version of EWC.
Revised: Efﬁcient version of EWC.

Original: Our analysis of SI shows that the Fisher can be approximated by the Empirical Fisher and that the gradient computation for the Empirical Fisher can be averaged across mini-batches (before it is squared) rather than carried out for each image individually (see
Revised: Our analysis of SI shows that the Fisher can be approximated by the Empirical Fisher and that the gradient computation for the Empirical Fisher can be averaged across mini-batches (before it is squared) rather than carried out for each image individually (see

Original: Appendix D.2 for theory).
Revised: Appendix D.2 for theory).

Original: This is important as it offers large computational savings.
Revised: This is important as it offers large computational savings.

Original: First, we checked that the approximations described above do not harm performance.
Revised: First, we checked that the approximations described above do not harm performance.

Original: To this end we compared EWC (using the ‘real’ Fisher) to the Empirical Fisher and a variant calculating the Empirical Fisher in mini-batches (Batch-EF), c.f.  C.1.
Revised: To this end we compared EWC (using the ‘real’ Fisher) to the Empirical Fisher and a variant calculating the Empirical Fisher in mini-batches (Batch-EF), c.f. Appendix C.1.

Original: The results (Table 1 (4)) show that the approximations have the same performance as EWC.
Revised: The results (Table 1 (4)) show that the approximations have the same performance as EWC.

Original: At the same time, our ﬁnal algorithm Batch-EF is more than 800x faster than EWC at approximating the Fisher (Figure 3); onSplit
Revised: At the same time, our ﬁnal algorithm Batch-EF is more than 800x faster than EWC at approximating the Fisher (Figure 3); on

Original:  CIFAR the speed up is >300x.
Revised: Split CIFAR the speed up is >300x.

Original: Note also that EWC is dominated by the time it needs to approximate the Fisher, while for our sped-up Batch-EF this time has become negligible compared to the training time.
Revised: Note also that EWC is dominated by the time it needs to approximate the Fisher, while for our sped-up Batch-EF this time has become negligible compared to the training time.

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Original: Above, we carried out experimental validations of our assumptions.
Revised: Above, we carried out experimental validations of our assumptions.

Original: Here, we outline our setup, see
Revised: Here, we outline our setup, see

Original: Appendix C for full details.
Revised: Appendix C for full details.

Original: We closely follow SI’s setting (Zenke et al., 2017):
Revised: We closely follow SI’s setting (Zenke et al., 2017):

Original: In domain-incremental
Revised: In domain-incremental

Original: Permuted MNIST (Goodfellow et al., 2013) each of 10 tasks consists of a random (but ﬁxed) pixelpermutation of MNIST and a fully connected ReLU network is used.
Revised: Permuted MNIST (Goodfellow et al., 2013) each of 10 tasks consists of a random (but ﬁxed) pixelpermutation of MNIST and a fully connected ReLU network is used.

Original: The task-incremental Split
Revised: The task-incremental Split

Original: CIFAR 10/100 benchmark (Zenke et al., 2017) consists of six 10-way classiﬁcation tasks, the ﬁrst is CIFAR 10, and the other ones are extracted from CIFAR 100.
Revised: CIFAR 10/100 benchmark (Zenke et al., 2017) consists of six 10-way classiﬁcation tasks, the ﬁrst is CIFAR 10, and the other ones are extracted from CIFAR 100.

Original: The keras default CIFARconvolutional architecture (with dropout and max-pooling) is used keras.
Revised: The keras default CIFARconvolutional architecture (with dropout and max-pooling) is used keras.

Original: The only difference to the setup of Zenke et al.
Revised: The only difference to the setup of Zenke et al.

Original: (2017) is that like Swaroop et al.
Revised: (2017) is that like Swaroop et al.

Original: (2019), we usually re-initialise the network weights after each task, observing performance improvements.
Revised: (2019), we usually re-initialise the network weights after each task, observing performance improvements.

Original: Code is available on github.
Revised: Code is available on github.

Original: For brevity, we sometimes only showed plots based on one benchmark.
Revised: For brevity, we sometimes only showed plots based on one benchmark.

Original: The analogous plot based on the other benchmark can then be found in the appendix.
Revised: The analogous plot based on the other benchmark can then be found in the appendix.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have investigated regularisation approaches for continual learning, which are the method of choice for continual learning without replaying old data or expanding the model.
Revised: We have investigated regularisation approaches for continual learning, which are the method of choice for continual learning without replaying old data or expanding the model.

Original: We have provided strong theoretical and experimental evidence that the three most inﬂuential methods in this domain are surprisingly similar.
Revised: We have provided strong theoretical and experimental evidence that the three most inﬂuential methods in this domain are surprisingly similar.

Original: This does not only unify these three methods, but also their many follow ups.
Revised: This does not only unify these three methods, but also their many follow ups.

Original: Additionally, relating SI and MAS to the Fisher Information has the beneﬁt of putting these algorithms on a solid theoretical underpinning.
Revised: Additionally, relating SI and MAS to the Fisher Information has the beneﬁt of putting these algorithms on a solid theoretical underpinning.

Original: Kirkpatrick et al.
Revised: Kirkpatrick et al.

Original: Huszár (2018) give a theoretical, bayesian explanation of why the Fisher Information with a quadratic weight penalty is a principled approach to continual learning and recently, Yin et al.
Revised: Huszár (2018) give a theoretical, bayesian explanation of why the Fisher Information with a quadratic weight penalty is a principled approach to continual learning and recently, Yin et al.

Original: (2020) provided theoretical guarantees as well as a frequentist interpretation of this approach.
Revised: (2020) provided theoretical guarantees as well as a frequentist interpretation of this approach.

Original: Our results extend these insights from EWC to all MAS- and SI-based algorithms providing them with a solid theoretical basis.
Revised: Our results extend these insights from EWC to all MAS- and SI-based algorithms providing them with a solid theoretical basis.

Original: Our thorough analysis and carefully designed baselines yielded several  insights.
Revised: Our thorough analysis and carefully designed baselines yielded several additional insights.

Original: Among other things, they showed that: our baseline OnAF is a theoretically more sound version of SI; our(predictive) AF algorithm is more sound than MAS; EWC can be replaced by our Batch-EF offering large computational speed-ups.
Revised: Among other things, they showed that: our baseline OnAF is a theoretically more sound version of SI; our AF algorithm is more sound than MAS; EWC can be replaced by our Batch-EF offering large computational speed-ups.

Original: Moreover, we point out that our algorithm SIU to approximate SI’s path integral opens up the possibility to apply the (non continual learning related) algorithm LCA
Revised: Moreover, we point out that our algorithm SIU to approximate SI’s path can used for the (non continual learning related) algorithm LCA

Original: (Lan et al., 2019) to larger datasets and architectures, since it drastically reduces its computational costs.
Revised: (Lan et al., 2019), which relies on a computationally expensive approximation of the same integral.

Revised: This opens up new opportunities to apply LCA to larger models and datasets.

Original: Finally, we hope that more future work will focus on unifying and explaining the ever growing body of continual learning literature, advancing our understanding of these methods and allowing to discover new, efﬁcient algorithms.
Revised: Finally, we hope that more future work will focus on unifying and explaining the ever growing body of continual learning literature, advancing our understanding of these methods and allowing to discover new, efﬁcient algorithms.

Original: A APPENDIX
Revised: A APPENDIX

Original: This appendix has several components:
Revised: This appendix has several components:

Original: We provide a tabular overview over algorithms and baselines described in the paper in B 2.
Revised: We provide a tabular overview over algorithms and baselines described in the paper in B 2.

Original: We give full experimental details including hyperparameter seraches and values, as well as full details about the variants and speed-ups of EWC in Appendix C. 3.
Revised: We give full experimental details including hyperparameter seraches and values, as well as full details about the variants and speed-ups of EWC in Appendix C. 3.

Original: We give details for two calculations omitted in the main paper in Appendix D. 4.
Revised: We give details for two calculations omitted in the main paper in Appendix D. 4.

Original: We empirically investigate the gradient noise in Appendix E.
Revised: We empirically investigate the gradient noise in Appendix E.

Original: This experiment is not speciﬁc to continual learning.
Revised: This experiment is not speciﬁc to continual learning.

Original: We critically review experimental claims from previous work about regularisation methods in Appendix F. 6.
Revised: We critically review experimental claims from previous work about regularisation methods in Appendix F. 6.

Original: We include additional experiments and plots in Appendix G.
Revised: We include additional experiments and plots in Appendix G.

Original: B O VERVIEW OVER A LGORITHMS AND B ASELINES
Revised: B O VERVIEW OVER A LGORITHMS AND B ASELINES

Original: [Table]
Revised: [Table]

Original: C E XPERIMENTAL DETAILS C. D ETAILS , V ARIANTS AND S PEED -U PS OF EWC. D ETAILS OF MAS AND AF.
Revised: C E XPERIMENTAL DETAILS C. D ETAILS , V ARIANTS AND S PEED -U PS OF EWC. D ETAILS OF MAS AND AF.

Original: For EWC we calculate the ‘real’ Fisher Information as deﬁned in the main article.
Revised: For EWC we calculate the ‘real’ Fisher Information as deﬁned in the main article.

Original: 3 For P-MNIST, we randomly sample 2000 training images (rather than iterating through the entire training set, which is prohibitively expensive).
Revised: 3 For P-MNIST, we randomly sample 2000 training images (rather than iterating through the entire training set, which is prohibitively expensive).

Original: For Split CIFAR we use 500 random training images.
Revised: For Split CIFAR we use 500 random training images.

Original: Preliminary experiments with more than 2000 (500) images gave no performance improvement.
Revised: Preliminary experiments with more than 2000 (500) images gave no performance improvement.

Original: Similarly, the Empirical Fisher is based on 2000 (500) samples.
Revised: Similarly, the Empirical Fisher is based on 2000 (500) samples.

Original: For Batch-EF, we also use 2000 (resp 500) samples and split them into 20 (resp 5) mini-batches of size 100.
Revised: For Batch-EF, we also use 2000 (resp 500) samples and split them into 20 (resp 5) mini-batches of size 100.

Original: We compute the average gradient for each mini-batch, square it and average the squares over the mini-batches.
Revised: We compute the average gradient for each mini-batch, square it and average the squares over the mini-batches.

Original: We also multiply the result by the mini-batch size (100) for normalisation.
Revised: We also multiply the result by the mini-batch size (100) for normalisation.

Original: For a theoretical justiﬁcation see D.2.
Revised: For a theoretical justiﬁcation see D.2.

Original: Note that this requires 100 times less forward passes than calculating the Empirical Fisher (albeit with larger batch-size) and makes more efﬁcient use of the deep-learning libraries optimised for parallel-batch computation.
Revised: Note that this requires 100 times less forward passes than calculating the Empirical Fisher (albeit with larger batch-size) and makes more efﬁcient use of the deep-learning libraries optimised for parallel-batch computation.

Original: MAS, MAS-X and AF algorithms are based on 1000 resp 500 random samples.
Revised: MAS, MAS-X and AF algorithms are based on 1000 resp 500 random samples.

Original: When comparing these algorithms, we use the same set of samples for each algorithm.
Revised: When comparing these algorithms, we use the same set of samples for each algorithm.

Original: When calculating the corelation between SI and AF, we use 30000 (MNIST) resp 500 (CIFAR) samples to evaluate AF.
Revised: When calculating the corelation between SI and AF, we use 30000 (MNIST) resp 500 (CIFAR) samples to evaluate AF.

Original: This is because we observed a higher variance on MNIST for AF (but as pointed out above taking more samples did not improve performance of AF).
Revised: This is because we observed a higher variance on MNIST for AF (but as pointed out above taking more samples did not improve performance of AF).

Original: C. B ENCHMARKS
Revised: C. B ENCHMARKS

Original: In Permuted MNIST (Goodfellow et al., 2013) each task consists of predicting the label of a random (but ﬁxed) pixel-permutation of MNIST.
Revised: In Permuted MNIST (Goodfellow et al., 2013) each task consists of predicting the label of a random (but ﬁxed) pixel-permutation of MNIST.

Original: We use a total of 10 tasks.
Revised: We use a total of 10 tasks.

Original: As noted, we use a domain incremental setting, i.e. a single output head shared by all tasks.
Revised: As noted, we use a domain incremental setting, i.e. a single output head shared by all tasks.

Original: In Split CIFAR10/100 the network ﬁrst has to classify CIFAR10 and then is successively given groups of 10 (consecutive) classes of CIFAR100.
Revised: In Split CIFAR10/100 the network ﬁrst has to classify CIFAR10 and then is successively given groups of 10 (consecutive) classes of CIFAR100.

Original: As in (Zenke et al., 2017), we use 6 tasks in total.
Revised: As in (Zenke et al., 2017), we use 6 tasks in total.

Original: Preliminary experiments on the maximum of 11 tasks showed very little difference.
Revised: Preliminary experiments on the maximum of 11 tasks showed very little difference.

Original: We use a task-incremental setting, i.e. each task has its output head and task-identity is known during training and testing.
Revised: We use a task-incremental setting, i.e. each task has its output head and task-identity is known during training and testing.

Original: C. P RE - PROCESSING
Revised: C. P RE - PROCESSING

Original: Following Zenke et al.
Revised: Following Zenke et al.

Original: (2017), we normalise all pixel values to be in the interval [0 , 1] (i.e. we divide by 255) for MNIST and CIFAR datasets and use no data augmentation.
Revised: (2017), we normalise all pixel values to be in the interval [0 , 1] (i.e. we divide by 255) for MNIST and CIFAR datasets and use no data augmentation.

Original: We point out that pre-processing can affect performance, e.g. differences in pre-processing are part of the reason why results for SI on Permuted-MNIST in (Hsu et al., 2018) are considerably worse than reported in (Zenke et al., 2017)
Revised: We point out that pre-processing can affect performance, e.g. differences in pre-processing are part of the reason why results for SI on Permuted-MNIST in (Hsu et al., 2018) are considerably worse than reported in (Zenke et al., 2017)

Original: (the other two reasons being an unusually small learning rate of 10 − 4 for Adam in (Hsu et al., 2018) and a slightly smaller architecture).
Revised: (the other two reasons being an unusually small learning rate of 10 − 4 for Adam in (Hsu et al., 2018) and a slightly smaller architecture).

Original: C. A RCHITECTURES AND I NITIALIZATION
Revised: C. A RCHITECTURES AND I NITIALIZATION

Original: For P-MNIST we use a fully connected net with ReLu activations and two hidden layers ofunits each.
Revised: For P-MNIST we use a fully connected net with ReLu activations and two hidden layers ofunits each.

Original: For Split CIFAR10/100 we use the default Keras CNN for CIFAR 10 with 4 convolutional layers and two fully connected layers, dropout (Srivastava et al., 2014) and maxpooling, see Table
Revised: For Split CIFAR10/100 we use the default Keras CNN for CIFAR 10 with 4 convolutional layers and two fully connected layers, dropout (Srivastava et al., 2014) and maxpooling, see Table

Original: C.1.
Revised: C.1.

Original: We use Glorot-uniform (Glorot & Bengio, 2010) initialization for both architectures.
Revised: We use Glorot-uniform (Glorot & Bengio, 2010) initialization for both architectures.

Original: C. O PTIMIZATION
Revised: C. O PTIMIZATION

Original: We use the Adam Optimizer (Kingma & Ba, 2014) with tensorﬂow (Abadi et al., 2016) default settings ( lr = 0 .
Revised: We use the Adam Optimizer (Kingma & Ba, 2014) with tensorﬂow (Abadi et al., 2016) default settings ( lr = 0 .

Original: 001 , β 1 = 0 .
Revised: 001 , β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 , (cid:15) =
Revised: 999 , (cid:15) =

Original: 10 − 8 ) and batchsize 256 for both tasks like Zenke et al.
Revised: 10 − 8 ) and batchsize 256 for both tasks like Zenke et al.

Original: On Permuted-MNIST we train each task for 20 epochs, on Split CIFAR we train each task for 60 epochs, also following Zenke et al.
Revised: On Permuted-MNIST we train each task for 20 epochs, on Split CIFAR we train each task for 60 epochs, also following Zenke et al.

Original: We reset the optimizer variables (momentum and such) after each task.
Revised: We reset the optimizer variables (momentum and such) after each task.

Original: C. SI & O N AF DETAILS
Revised: C. SI & O N AF DETAILS

Original: Recall that we applied the operation max(0 , · ) (i.e. a ReLU activation) to the importance measure of each individual task (equation 4 of main paper), before adding it to the overall importance.
Revised: Recall that we applied the operation max(0 , · ) (i.e. a ReLU activation) to the importance measure of each individual task (equation 4 of main paper), before adding it to the overall importance.

Original: In the original SI implementation, this seems to be replaced by applying the same operation to the overall importances (after adding potentially negative values from a given task).
Revised: In the original SI implementation, this seems to be replaced by applying the same operation to the overall importances (after adding potentially negative values from a given task).

Original: No description of either of these operations is given in the SI-paper.
Revised: No description of either of these operations is given in the SI-paper.

Original: In light of our ﬁndings, our version seems more justiﬁed.
Revised: In light of our ﬁndings, our version seems more justiﬁed.

Original: Somewhat naturally, the gradient ∂L∂θ usually refers to the cross-entropy loss of the current task and not the total loss including regularisation.
Revised: Somewhat naturally, the gradient ∂L∂θ usually refers to the cross-entropy loss of the current task and not the total loss including regularisation.

Original: For CIFAR we evaluated this gradient without dropout (but the parameter update was of course evaluated with dropout).
Revised: For CIFAR we evaluated this gradient without dropout (but the parameter update was of course evaluated with dropout).

Original: For OnAF, similarly to SI, we used the gradient evaluated without dropout on the cross-entropy loss of the current task for our importance measure.
Revised: For OnAF, similarly to SI, we used the gradient evaluated without dropout on the cross-entropy loss of the current task for our importance measure.

Original: When evaluated on benchmarks, SI, SIU, SIB and OnAF are all rescaled according to equationto make the comparison as fair as possible.
Revised: When evaluated on benchmarks, SI, SIU, SIB and OnAF are all rescaled according to equationto make the comparison as fair as possible.

Original: Scatter plots and correlations use values before this rescaling.
Revised: Scatter plots and correlations use values before this rescaling.

Original: [Table]
Revised: [Table]

Original: C
Revised: C

Original: [Table]
Revised: [Table]

Original: H YPERPARAMETERS
Revised: H YPERPARAMETERS

Original: For all methods and benchmarks, we performed grid searches for the hyperparameter c and over the choice whether or not to re-initialise model variables after each task.
Revised: For all methods and benchmarks, we performed grid searches for the hyperparameter c and over the choice whether or not to re-initialise model variables after each task.

Original: The grid of c included values a · 10 i where a ∈ { 1 , 2 , 5 } and i was chosen in a suitable range (if a hyperparameter close to a boundary of our grid performed best, we extended the grid).
Revised: The grid of c included values a · 10 i where a ∈ { 1 , 2 , 5 } and i was chosen in a suitable range (if a hyperparameter close to a boundary of our grid performed best, we extended the grid).

Original: For CIFAR, we measured validation set performance based on at least three repetitions for good hyperparameters.
Revised: For CIFAR, we measured validation set performance based on at least three repetitions for good hyperparameters.

Original: We then picked the best HP and ran 10 repetitions on the test set.
Revised: We then picked the best HP and ran 10 repetitions on the test set.

Original: For MNIST, we measured HP-quality on the test set based on at least 3 runs for good HPs.
Revised: For MNIST, we measured HP-quality on the test set based on at least 3 runs for good HPs.

Original: Additionally, SI and consequently SIU, SIB as well as OnAF have rescaled importance (c.f. equation equation 4 from main paper).
Revised: Additionally, SI and consequently SIU, SIB as well as OnAF have rescaled importance (c.f. equation equation 4 from main paper).

Original: The damping term ξ in this rescaling was set to 0 .
Revised: The damping term ξ in this rescaling was set to 0 .

Original: 1 for MNIST and to 0 .
Revised: 1 for MNIST and to 0 .

Original: 001 for CIFAR following Zenke et al.
Revised: 001 for CIFAR following Zenke et al.

Original: (2017) without further HP search.
Revised: (2017) without further HP search.

Original: All results shown are based on the same hyperparameters obtained – individually for each method – as described above.
Revised: All results shown are based on the same hyperparameters obtained – individually for each method – as described above.

Original: They can be found in Table C.2.
Revised: They can be found in Table C.2.

Original: We note that the difference between the HPs for MAS and MASX might seem to contradict our claims that the two measures are almost identical (they should require the same c in this case), but this is most likely due to similar performance for different HPs and random ﬂuctuations.
Revised: We note that the difference between the HPs for MAS and MASX might seem to contradict our claims that the two measures are almost identical (they should require the same c in this case), but this is most likely due to similar performance for different HPs and random ﬂuctuations.

Original: For example on CIFAR, MAS had almost identical validation performance for c = 200 (best for MAS) and c = 1000 (best for MASX) ( 74 . 2 ± 0 .
Revised: For example on CIFAR, MAS had almost identical validation performance for c = 200 (best for MAS) and c = 1000 (best for MASX) ( 74 . 2 ± 0 .

Original: 7 vs 74 .
Revised: 7 vs 74 .

Original: 1 ± 0 . 5 ).
Revised: 1 ± 0 . 5 ).

Original: Also for the other methods, we observed that usually there were two or more HP-conﬁgurations which performed very similarly.
Revised: Also for the other methods, we observed that usually there were two or more HP-conﬁgurations which performed very similarly.

Original: The precise ‘best’ values as found by the HP search and reported in
Revised: The precise ‘best’ values as found by the HP search and reported in

Original: Table C.2 are therefore subject to random ﬂuctuations in the grid search.
Revised: Table C.2 are therefore subject to random ﬂuctuations in the grid search.

Original: [Table]
Revised: [Table]

Original: C. S CATTER P LOT D ATA C OLLECTION AND I NTENSITY P LOTS
Revised: C. S CATTER P LOT D ATA C OLLECTION AND I NTENSITY P LOTS

Original: The scatter plots are based on a run of SI, where we in parallel to evaluating the SI importance measure also evaluated all other importance measures.
Revised: The scatter plots are based on a run of SI, where we in parallel to evaluating the SI importance measure also evaluated all other importance measures.

Original: Figures 1 and 2 show data based on 4 repetitions of this experiment and conﬁrm that the correlations observed in the scatter plots are representative, it also shows the effect of sample sizes on AF.
Revised: Figures 1 and 2 show data based on 4 repetitions of this experiment and conﬁrm that the correlations observed in the scatter plots are representative, it also shows the effect of sample sizes on AF.

Original: Intensity Plots (as used for some ﬁgures in the appendix) Show same kind of data as scatter plots, but each weight is binned into one of 80x50 equispaced bins.
Revised: Intensity Plots (as used for some ﬁgures in the appendix) Show same kind of data as scatter plots, but each weight is binned into one of 80x50 equispaced bins.

Original: The number of weights per bin is divided by the maximum number of weights per bin in that column and the resulting value is shown on a gray scale.
Revised: The number of weights per bin is divided by the maximum number of weights per bin in that column and the resulting value is shown on a gray scale.

Original: Normalisation per column was performed since for ‘global normalisation’ (divide by total number of weights) only weights in the bottom left are visible (as most weights have small importances); the number of bins was chosen to match aspect ratio of plots.
Revised: Normalisation per column was performed since for ‘global normalisation’ (divide by total number of weights) only weights in the bottom left are visible (as most weights have small importances); the number of bins was chosen to match aspect ratio of plots.

Original: C. C OMPUTING I NFRASTRUCTURE
Revised: C. C OMPUTING I NFRASTRUCTURE

Original: We ran experiments on one NVIDIA GeForce GTX 980 and on one NVIDIA GeForce GTX 

Revised: We ran experiments on one NVIDIA GeForce GTX 980 and on one NVIDIA GeForce GTX

Original: Ti
Revised: Ti

Original: GPU.
Revised: GPU.

Original: All wall-clock time experiments were run on the latter GPU.
Revised: All wall-clock time experiments were run on the latter GPU.

Original: We used tensorﬂow 1.4 for all experiments.
Revised: We used tensorﬂow 1.4 for all experiments.

Original: D D ETAILED C ALCULATIONS FOR THE BIAS OF SI, AND FOR CALCULATING F ISHER IN B ATCHES
Revised: D D ETAILED C ALCULATIONS FOR THE BIAS OF SI, AND FOR CALCULATING F ISHER IN B ATCHES

Original: Here we give the details for two calculations omitted in the main text.
Revised: Here we give the details for two calculations omitted in the main text.

Original: D. F ORMULA OF B IAS OF SI WITH A DAM
Revised: D. F ORMULA OF B IAS OF SI WITH A DAM

Original: Here, we show that the bias, i.e. the expected difference between SI and SIU, is equal to (1 − β 1 ) σ 2 t / ( √ v t + (cid:15) ) and that Assumption 1 is equivalent to the bias being larger than the unbiased part of SI.
Revised: Here, we show that the bias, i.e. the expected difference between SI and SIU, is equal to (1 − β 1 ) σ 2 t / ( √ v t + (cid:15) ) and that Assumption 1 is equivalent to the bias being larger than the unbiased part of SI.

Original: Recall that for SI, we approximate ∂L ( t )
Revised: Recall that for SI, we approximate ∂L ( t )

Original: ∂θ by g t + σ t , which is the same gradient estimate given to
Revised: ∂θ by g t + σ t , which is the same gradient estimate given to

Original: Adam.
Revised: Adam.

Original: So we get
Revised: So we get

Original: [Equation]
Revised: [Equation]

Original: For SIU, we use an independent mini-batch estimate g t + σ (cid:48) t for ∂L ( t ) ∂θ and therefore obtain
Revised: For SIU, we use an independent mini-batch estimate g t + σ (cid:48) t for ∂L ( t ) ∂θ and therefore obtain

Original: SIU:
Revised: SIU:

Original: [Equation]
Revised: [Equation]

Original: Taking the difference between these two and ignoring all terms which have expectation zero (note that E [ σ t ]
Revised: Taking the difference between these two and ignoring all terms which have expectation zero (note that E [ σ t ]

Original: =
Revised: =

Original: E [ σ (cid:48) t ] = 0 and that σ t , σ (cid:48) t are independent of m t − 1 and g t ) gives
Revised: E [ σ (cid:48) t ] = 0 and that σ t , σ (cid:48) t are independent of m t − 1 and g t ) gives

Original: [Equation]
Revised: [Equation]

Original: as claimed.
Revised: as claimed.

Original: To see that Assumption 1 is equivalent to the bias being larger than the unbiased part, note that in expectation SIU equals
Revised: To see that Assumption 1 is equivalent to the bias being larger than the unbiased part, note that in expectation SIU equals

Original: [Equation]
Revised: [Equation]

Original: The last approximation here is valid because β 1 m t − 1 (cid:29) (1 − β 1 ) g t which holds since (1) β 1 (cid:29) (1 − β 1 ) and (2) E [ | m t − 1 | ] ≥ E [ | g t | ] .
Revised: The last approximation here is valid because β 1 m t − 1 (cid:29) (1 − β 1 ) g t which holds since (1) β 1 (cid:29) (1 − β 1 ) and (2) E [ | m t − 1 | ] ≥ E [ | g t | ] .

Original: (1) follows from β 1 = 0 . 9 , and (2) holds since E [ m t − 1 ]
Revised: (1) follows from β 1 = 0 . 9 , and (2) holds since E [ m t − 1 ]

Original: ≈ g t and since m t − 1 has a noise component (due to gradient noise), while g t is a deterministic quantity, implying E
Revised: ≈ g t and since m t − 1 has a noise component (due to gradient noise), while g t is a deterministic quantity, implying E

Original: [ | m t − 1 | ] ≥ E [ | g t | ] by elementary considerations.
Revised: [ | m t − 1 | ] ≥ E [ | g t | ] by elementary considerations.

Original: In particular, the two calculations for the bias and the expectation of SIU above imply that Assumption 1, which is (1 − β 1 ) σ 2 t (cid:29) β 1 m t − 1 g t , is equivalent to the bias of SI being larger than its unbiased part.
Revised: In particular, the two calculations for the bias and the expectation of SIU above imply that Assumption 1, which is (1 − β 1 ) σ 2 t (cid:29) β 1 m t − 1 g t , is equivalent to the bias of SI being larger than its unbiased part.

Original: D. E FFECT OF C ALCULATING F ISHER IN M INI -B ATCHES : A NALYSIS OF B ATCH -EF
Revised: D. E FFECT OF C ALCULATING F ISHER IN M INI -B ATCHES : A NALYSIS OF B ATCH -EF

Original: For this subsection, let us slightly change notation and denote the images by X 1 , . . .
Revised: For this subsection, let us slightly change notation and denote the images by X 1 , . . .

Original: , X D and the gradients (with respect to their labels and the cross entropy loss) by g + σ 1 , . . .
Revised: , X D and the gradients (with respect to their labels and the cross entropy loss) by g + σ 1 , . . .

Original: g + σ D .
Revised: g + σ D .

Original: Here, again g is the overall training set gradient and σ i is the noise (i.e. (cid:80) Di =1 σ i = 0 ).
Revised: Here, again g is the overall training set gradient and σ i is the noise (i.e. (cid:80) Di =1 σ i = 0 ).

Original: Then the Empirical Fisher is given by
Revised: Then the Empirical Fisher is given by

Original: [Equation]
Revised: [Equation]

Original: We want to compare this to evaluating the squared gradient over a batch.
Revised: We want to compare this to evaluating the squared gradient over a batch.

Original: Let i 1 , . .
Revised: Let i 1 , . .

Original: , i b denote uniformly random, independent indices from { 1 , . . .
Revised: , i b denote uniformly random, independent indices from { 1 , . . .

Original: , D } , so
Revised: , D } , so

Original: that X i 1 , . . .
Revised: that X i 1 , . . .

Original: , X i b is a random minibatch of size b .
Revised: , X i b is a random minibatch of size b .

Original: Let g + σ be the gradient on this mini-batch.
Revised: Let g + σ be the gradient on this mini-batch.

Original: We then have, taking expectations over the random indices,
Revised: We then have, taking expectations over the random indices,

Original: [Equation]
Revised: [Equation]

Original: This conclusion may seem surprising, but recall that it is only based on one assumption, namely that the gradient noise is considerably bigger than the gradient itself - an assumption for which we have collected ample evidence.
Revised: This conclusion may seem surprising, but recall that it is only based on one assumption, namely that the gradient noise is considerably bigger than the gradient itself - an assumption for which we have collected ample evidence.

Original: Concretely, we assume E [( g + σ i ) 2 ] (cid:29) ( b − 1) g 2 .
Revised: Concretely, we assume E [( g + σ i ) 2 ] (cid:29) ( b − 1) g 2 .

Original: Note that unlike in other sections of this manuscript, here σ i refers to the gradient noise of individual images (rather than mini-batch noise).
Revised: Note that unlike in other sections of this manuscript, here σ i refers to the gradient noise of individual images (rather than mini-batch noise).

Original: We have seen that even with a batch-size of 256 (i.e. when we reduce then noise by a factor of 256) the noise is still much bigger than the gradient (recall Figures G.5, G.6, E.1, E.2), so that our assumption is true with room to spare.
Revised: We have seen that even with a batch-size of 256 (i.e. when we reduce then noise by a factor of 256) the noise is still much bigger than the gradient (recall Figures G.5, G.6, E.1, E.2), so that our assumption is true with room to spare.

Original: E G RADIENT N OISE
Revised: E G RADIENT N OISE

Original: Here, we quantitatively assess the noise magnitude outside the continual learning context.
Revised: Here, we quantitatively assess the noise magnitude outside the continual learning context.

Original: Recall that Figure 2 (left) from the main paper, as well as Figures G.5 and G.6 already show that the noise
Revised: Recall that Figure 2 (left) from the main paper, as well as Figures G.5 and G.6 already show that the noise

Original: Data obtained by training a ReLu network with 2 hidden layers of 2000 hidden units for 20 epochs with default Adam settings.
Revised: Data obtained by training a ReLu network with 2 hidden layers of 2000 hidden units for 20 epochs with default Adam settings.

Original: Only every 20-th datapoint shown for better visualisation.
Revised: Only every 20-th datapoint shown for better visualisation.

Original: dominates the SI importance measure, which indicates that the noise is considerably larger than the gradient itself.
Revised: dominates the SI importance measure, which indicates that the noise is considerably larger than the gradient itself.

Original: To obtain an assessment independent of the SI continual learning importance measure, we trained our network on MNIST as described before, i.e. a ReLu network with 2 hidden layers of 2000 units each, trained for 20 epochs with batch size 256 and default Adam settings.
Revised: To obtain an assessment independent of the SI continual learning importance measure, we trained our network on MNIST as described before, i.e. a ReLu network with 2 hidden layers of 2000 units each, trained for 20 epochs with batch size 256 and default Adam settings.

Original: At each training iteration, on top of calculating the stochastic mini-batch gradient used for optimization, we also computed the full gradient on the entire training set and computed the noise – which refers to the squared (cid:96) 2 distance between the stochastic mini-batch gradient and the full gradient – as well as the ratio between noise and gradient, measured as the ratio of squared (cid:96) 2 norms.
Revised: At each training iteration, on top of calculating the stochastic mini-batch gradient used for optimization, we also computed the full gradient on the entire training set and computed the noise – which refers to the squared (cid:96) 2 distance between the stochastic mini-batch gradient and the full gradient – as well as the ratio between noise and gradient, measured as the ratio of squared (cid:96) 2 norms.

Original: The results are shown in Figure E.1.
Revised: The results are shown in Figure E.1.

Original: In addition, we computed the fraction of iterations in which the ratio between noise and squared gradient norm is above a certain threshold, see Figure E.2.
Revised: In addition, we computed the fraction of iterations in which the ratio between noise and squared gradient norm is above a certain threshold, see Figure E.2.

Original: F R ELATIVE P ERFORMANCE OF R EGULARISATION M ETHODS
Revised: F R ELATIVE P ERFORMANCE OF R EGULARISATION M ETHODS

Original: Several papers (van de Ven & Tolias, 2019; Hsu et al., 2018; Farquhar & Gal, 2018) played an important role in recognising that different continual learning algorithms had been evaluated in settings with varying difﬁculties.
Revised: Several papers (van de Ven & Tolias, 2019; Hsu et al., 2018; Farquhar & Gal, 2018) played an important role in recognising that different continual learning algorithms had been evaluated in settings with varying difﬁculties.

Original: These papers also greatly clariﬁed these setting, making an important contribution for future work.
Revised: These papers also greatly clariﬁed these setting, making an important contribution for future work.

Original: Here, we want to point out that some of the results (and thus conclusions) of the experiments in (Hsu et al., 2018)
Revised: Here, we want to point out that some of the results (and thus conclusions) of the experiments in (Hsu et al., 2018)

Original: (these were the only ones we investigated more closely) are speciﬁc to suboptimal settings and hyperparameters used there.
Revised: (these were the only ones we investigated more closely) are speciﬁc to suboptimal settings and hyperparameters used there.

Original: For example, it is found there that EWC, MAS, SI perform worse than an importance measure which assigns the same importance to all parameters (called ‘L2 regularisation’).
Revised: For example, it is found there that EWC, MAS, SI perform worse than an importance measure which assigns the same importance to all parameters (called ‘L2 regularisation’).

Original: This is in contrast to results reported in the EWC paper (Kirkpatrick et al., 2017), who found the L2 baseline to perform worse than EWC.
Revised: This is in contrast to results reported in the EWC paper (Kirkpatrick et al., 2017), who found the L2 baseline to perform worse than EWC.

Original: We also attempted to reproduce the L2 results and tried two versions of this approach, one with constant importance for all tasks and one with importance increasing linearly with the number of tasks.
Revised: We also attempted to reproduce the L2 results and tried two versions of this approach, one with constant importance for all tasks and one with importance increasing linearly with the number of tasks.

Original: The latter approach worked better, but still considerably underperformed EWC, MAS, SI on Permuted-MNIST (88% vs 97%).
Revised: The latter approach worked better, but still considerably underperformed EWC, MAS, SI on Permuted-MNIST (88% vs 97%).

Original: We didn’t run these experiments on CIFAR.
Revised: We didn’t run these experiments on CIFAR.

Original: An alternative importance measure based on the weights’ magnitudes after training was better, but still could not compete with SI, EWC, MAS (<95% vs >97%) showing that these importance measure are more useand meaningful than naive baselines.
Revised: An alternative importance measure based on the weights’ magnitudes after training was better, but still could not compete with SI, EWC, MAS (<95% vs >97%) showing that these importance measure are more useand meaningful than naive baselines.

Original: Additionally, in other work we found that for domain incremental settings (again, these were the only ones we investigated), results of baselines EWC and SI, but also of ﬁne-tune (a baseline which takes no measures to prevent catastrophic forgetting) can be improved, often by more than 10% compared to previously mentioned reports (Hsu et al., 2018; van de Ven & Tolias, 2019).
Revised: Additionally, in other work we found that for domain incremental settings (again, these were the only ones we investigated), results of baselines EWC and SI, but also of ﬁne-tune (a baseline which takes no measures to prevent catastrophic forgetting) can be improved, often by more than 10% compared to previously mentioned reports (Hsu et al., 2018; van de Ven & Tolias, 2019).

Original: This also implies that the difference between well set-up regularisation approaches and for example repaly-methods is not nearly as big as previously thought.
Revised: This also implies that the difference between well set-up regularisation approaches and for example repaly-methods is not nearly as big as previously thought.

Original: The differences between these results are explained by different factors (we explicitly tested eeach factor): hyperparameters (for example using Adam with the standard learning rate 0.001 improves performance over the learning rate 0.0001 used previously), initialisation (Glorot Uniform Initialisation works considerably better for SI than pytorch standard initialisation for linear layers, which reveals unexpected defaults upon close investigation), training duration (regularisation approaches usually notably beneﬁt from longer training, this is reported in Swaroop et al.
Revised: The differences between these results are explained by different factors (we explicitly tested eeach factor): hyperparameters (for example using Adam with the standard learning rate 0.001 improves performance over the learning rate 0.0001 used previously), initialisation (Glorot Uniform Initialisation works considerably better for SI than pytorch standard initialisation for linear layers, which reveals unexpected defaults upon close investigation), training duration (regularisation approaches usually notably beneﬁt from longer training, this is reported in Swaroop et al.

Original: and we also found that performance on P-MNIST can be improved to 98% by training for 200 rather than 20 epochs; Hsu et al.
Revised: and we also found that performance on P-MNIST can be improved to 98% by training for 200 rather than 20 epochs; Hsu et al.

Original: (2018) use very short training times), data-preprocessing (depending on the setting different normalisations have different, non-negligible impacts).
Revised: (2018) use very short training times), data-preprocessing (depending on the setting different normalisations have different, non-negligible impacts).

Original: G M ORE E XPERIMENTS AND P LOTS G. P LOTS , D ATA ANALOGOUS TO F IGURES /T ABLES IN MAIN PAPER
Revised: G M ORE E XPERIMENTS AND P LOTS G. P LOTS , D ATA ANALOGOUS TO F IGURES /T ABLES IN MAIN PAPER

Original: Here, we show data for experiments already partly reported in the main paper, including both benchmarks and each task of each benchmark.
Revised: Here, we show data for experiments already partly reported in the main paper, including both benchmarks and each task of each benchmark.

Original: We start with the magnitude of the bias of SI (Figure 2, left panel): The observation that most of the SI importance is due to its bias is consistent across datasets and tasks, see Figures G.5 and G.6.
Revised: We start with the magnitude of the bias of SI (Figure 2, left panel): The observation that most of the SI importance is due to its bias is consistent across datasets and tasks, see Figures G.5 and G.6.

Original: Intriguingly, on CIFAR we ﬁnd that the unbiased approximation of SI slightly underestimates the decrease in loss in the last task, suggesting that strong regularisation pushes the parameters in places, where the cross-entropy of the current task has negative curvature.
Revised: Intriguingly, on CIFAR we ﬁnd that the unbiased approximation of SI slightly underestimates the decrease in loss in the last task, suggesting that strong regularisation pushes the parameters in places, where the cross-entropy of the current task has negative curvature.

Original: We show correlations between SI, SIB, SIU and AF, OnAF on CIFAR (analogous to Figure 2, middle) in Figure G.1.
Revised: We show correlations between SI, SIB, SIU and AF, OnAF on CIFAR (analogous to Figure 2, middle) in Figure G.1.

Original: The ﬁndings are in accordance with the ones on MNIST reported in the main paper.
Revised: The ﬁndings are in accordance with the ones on MNIST reported in the main paper.

Original: Note that only the ﬁrst task of CIFAR (i.e. CIFAR10) shows slightly different behaviour, as SIU and OnAF are more similar than on other tasks: correlation with OnAF is 0 .
Revised: Note that only the ﬁrst task of CIFAR (i.e. CIFAR10) shows slightly different behaviour, as SIU and OnAF are more similar than on other tasks: correlation with OnAF is 0 .

Original: 994 ± 0 .
Revised: 994 ± 0 .

Original: 001 for SI; 0 .
Revised: 001 for SI; 0 .

Original: 991 ± 0 .
Revised: 991 ± 0 .

Original: 001 for SIU; 0 .
Revised: 001 for SIU; 0 .

Original: 979 ± 0 .
Revised: 979 ± 0 .

Original: 003 for SIB.
Revised: 003 for SIB.

Original: The similarity between SIU and OnAF in this situation is explained by the weights with largest OnAF importance: If we remove the 5% of weights which have highest OnAF importance, correlation between OnAF, SIU drops to 0 .
Revised: The similarity between SIU and OnAF in this situation is explained by the weights with largest OnAF importance: If we remove the 5% of weights which have highest OnAF importance, correlation between OnAF, SIU drops to 0 .

Original: 599 ± 0 . 027 , but for SI resp.
Revised: 599 ± 0 . 027 , but for SI resp.

Original: SIB the same procedure yields 0 .
Revised: SIB the same procedure yields 0 .

Original: 977 ± 0 .
Revised: 977 ± 0 .

Original: 002 resp. 0 .
Revised: 002 resp. 0 .

Original: 965 ± 0 .
Revised: 965 ± 0 .

Original: This indicates that for the ﬁrst task of CIFAR there is a small fraction of weights with large OnAF importances, which are dominated by the gradient rather than the noise.
Revised: This indicates that for the ﬁrst task of CIFAR there is a small fraction of weights with large OnAF importances, which are dominated by the gradient rather than the noise.

Original: The remainder of weights is in accordance with our previous observations and dominated by noise, recall also Figure G.6.
Revised: The remainder of weights is in accordance with our previous observations and dominated by noise, recall also Figure G.6.

Original: We did not conduct the analogue of experiment corresponding to Figure 2, right panel, on MNIST since there the inﬂuence of regularisation was already weak there.
Revised: We did not conduct the analogue of experiment corresponding to Figure 2, right panel, on MNIST since there the inﬂuence of regularisation was already weak there.

Original: We also compare wall-clock time between EWC and our proposed speed-ups, see Figure G.2 for an analogue of Figure 3 on CIFAR rather than MNIST and see Table G.1 for raw data.
Revised: We also compare wall-clock time between EWC and our proposed speed-ups, see Figure G.2 for an analogue of Figure 3 on CIFAR rather than MNIST and see Table G.1 for raw data.

Original: Additional Details: ‘Time to approx.
Revised: Additional Details: ‘Time to approx.

Original: Fisher’ refers to the accumulated time used to calculate the
Revised: Fisher’ refers to the accumulated time used to calculate the

Original: Fisher on each task (10 resp 6 tasks on MNIST/CIFAR).
Revised: Fisher on each task (10 resp 6 tasks on MNIST/CIFAR).

Original: ‘Total Time’ refers to the accumulated time the algorithm needs to both train on each task and evaluate the Fisher on each task.
Revised: ‘Total Time’ refers to the accumulated time the algorithm needs to both train on each task and evaluate the Fisher on each task.

Original: It excludes
Revised: It excludes

Original: [Table]
Revised: [Table]

Original: model-evaluation (which is part of the code provided for the other experiments).
Revised: model-evaluation (which is part of the code provided for the other experiments).

Original: Note that for EWC on CIFAR (probably due to some peculiarities of tensorﬂow and our speciﬁc implementation), the forward pass to calculate the predicted label distribution for the Fisher was more than an order of magnitude slower than comparable forward passes.
Revised: Note that for EWC on CIFAR (probably due to some peculiarities of tensorﬂow and our speciﬁc implementation), the forward pass to calculate the predicted label distribution for the Fisher was more than an order of magnitude slower than comparable forward passes.

Original: Since this model-evaluation is not part of Emp.
Revised: Since this model-evaluation is not part of Emp.

Original: Fisher and Batch-EF (and since we are conﬁdent that more tf-compatible implementations exists)
Revised: Fisher and Batch-EF (and since we are conﬁdent that more tf-compatible implementations exists)

Original: [Table]
Revised: [Table]

Original: we excluded it from the time-measurements of EWC.
Revised: we excluded it from the time-measurements of EWC.

Original: Observe that this modiﬁcation only makes EWC faster, and without it the speed-up of our algorithms compared to EWC would be even more drastic.
Revised: Observe that this modiﬁcation only makes EWC faster, and without it the speed-up of our algorithms compared to EWC would be even more drastic.

Original: G. R ELATION OF O N AF AND AF
Revised: G. R ELATION OF O N AF AND AF

Original: We show summaries of the correlations between OnAF and AF in Figure G.3.
Revised: We show summaries of the correlations between OnAF and AF in Figure G.3.

Original: This includes comparisons of AF based on different sample sizes.
Revised: This includes comparisons of AF based on different sample sizes.

Original: We also investigated the inﬂuence of training duration on AF, OnAF, see next section.
Revised: We also investigated the inﬂuence of training duration on AF, OnAF, see next section.

Original: G
Revised: G

Original:  I NFLUENCE OF T RAINING D URATION ON AF, O N AF
Revised:  I NFLUENCE OF T RAINING D URATION ON AF, O N AF

Original: It seems plausible that the correlation between AF and OnAF we observed is due to training on tasks for a long time – the model might converge quickly so that the sum of OnAF is dominated by summands which approximate AF close to the ﬁnal point in training.
Revised: It seems plausible that the correlation between AF and OnAF we observed is due to training on tasks for a long time – the model might converge quickly so that the sum of OnAF is dominated by summands which approximate AF close to the ﬁnal point in training.

Original: To test this, we trained our networks for varying numbers of epochs on the ﬁrst task of P-MNIST and Split CIFAR10/ and measured the correlation between OnAF and AF.
Revised: To test this, we trained our networks for varying numbers of epochs on the ﬁrst task of P-MNIST and Split CIFAR10/ and measured the correlation between OnAF and AF.

Original: The results in Tables G.2, G.3 show that the correlation between AF, OnAF does not rely on long training duration.
Revised: The results in Tables G.2, G.3 show that the correlation between AF, OnAF does not rely on long training duration.

Original: On MNIST, we even see decreasing correlation with longer training time, this may be due to either an actual decrease in correlation or due to higher variance when estimating AF using 10000 samples (or both).
Revised: On MNIST, we even see decreasing correlation with longer training time, this may be due to either an actual decrease in correlation or due to higher variance when estimating AF using 10000 samples (or both).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: G. A DDITIONAL S CATTER P LOTS AND I NTENSITY P LOTS
Revised: G. A DDITIONAL S CATTER P LOTS AND I NTENSITY P LOTS

Original: We show additional scatter and intensity plots in Figure G.7.
Revised: We show additional scatter and intensity plots in Figure G.7.

Original: Note that for both architectures there are a few million weights.
Revised: Note that for both architectures there are a few million weights.

Original: The scatter plots are overcrowded and show the whole range of dependencies between two measure that can possibly occur.
Revised: The scatter plots are overcrowded and show the whole range of dependencies between two measure that can possibly occur.

Original: Intensity plots show the dependencies that the majority of weights adheres to.
Revised: Intensity plots show the dependencies that the majority of weights adheres to.

Original: G. A LTERNATIVE C OMPARISON OF SI, SIU, SIB TO AF, O N AF
Revised: G. A LTERNATIVE C OMPARISON OF SI, SIU, SIB TO AF, O N AF

Original: Finally, we point out that our comparisons between SI, SIU, SIB and AF, OnAF are usually obtained after applying a max(0 , · ) (c.f. equation equation 4 of main paper) to the SI (and SIU, SIB) importances as the algorithms diverge without this operation.
Revised: Finally, we point out that our comparisons between SI, SIU, SIB and AF, OnAF are usually obtained after applying a max(0 , · ) (c.f. equation equation 4 of main paper) to the SI (and SIU, SIB) importances as the algorithms diverge without this operation.

Original: We present results without this max operation in G.4, further showing that (1) correlation between OnAF and SI is weakened by strong regularisation and (2) the correlation is due to the bias of SI.
Revised: We present results without this max operation in G.4, further showing that (1) correlation between OnAF and SI is weakened by strong regularisation and (2) the correlation is due to the bias of SI.