Original: V ARIATIONAL C ONTINUAL L EARNING
Revised: V ARIATIONAL C ONTINUAL L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper develops variational continual learning (VCL), a simple but general framework for continual learning that fuses online variational inference (VI) and recent advances in Monte Carlo VI for neural networks.
Revised: This paper develops variational continual learning (VCL), a simple but general framework for continual learning that fuses online variational inference (VI) and recent advances in Monte Carlo VI for neural networks.

Original: The framework can successfully train both deep discriminative models and deep generative models in complex continual learning settings where existing tasks evolve over time and entirely new tasks emerge.
Revised: The framework can successfully train both deep discriminative models and deep generative models in complex continual learning settings where existing tasks evolve over time and entirely new tasks emerge.

Original: Experimental results show that variational continual learning outperforms state-of-the-art continual learning methods on a variety of tasks, avoiding catastrophic forgetting in a fully automatic way.
Revised: Experimental results show that VCL outperforms stateof-the-art continual learning methods on a variety of tasks, avoiding catastrophic forgetting in a fully automatic way.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Continual learning (also called life-long learning and incremental learning) is a very general form of online learning in which data continuously arrive in a possibly non i.i.d. way, tasks may change over time (e.g. new classes may be discovered), and entirely new tasks can emerge (Schlimmer & Fisher, 1986; Sutton & Whitehead, 1993; Ring, 1997).
Revised: Continual learning (also called life-long learning and incremental learning) is a very general form of online learning in which data continuously arrive in a possibly non i.i d. way, tasks may change over time (e g. new classes may be discovered) and entirely new tasks can emerge (Schlimmer & Fisher, 1986; Sutton & Whitehead, 1993; Ring, 1997).

Original: What is more, continual learning systems must adapt to perform well on the entire set of tasks in an online way that avoids revisiting all previous data at each stage.
Revised: What is more, continual learning systems must adapt to perform well on the entire set of tasks in an incremental way that avoids revisiting all previous data at each stage.

Original: This is a key problem in machine learning since real world tasks continually evolve over time (e.g. they suffer from covariate and dataset shift) and the size of datasets often prohibits frequent batch updating.
Revised: This is a key problem in machine learning since real world tasks continually evolve over time (e g. they suffer from covariate and dataset shift) and the size of datasets often prohibits frequent batch updating.

Original: Moreover, practitioners are often interested in solving a set of related tasks that beneﬁt from being handled jointly in order to leverage multi-task transfer.
Revised: Moreover, practitioners are often interested in solving a set of related tasks that beneﬁt from being handled jointly in order to leverage multi-task transfer.

Original: Continual learning is also of interest to cognitive science, being an intrinsic human ability.
Revised: Continual learning is also of interest to cognitive science, being an intrinsic human ability.

Original: The ubiquity of deep learning means that it is important to develop deep continual learning methods.
Revised: The ubiquity of deep learning means that it is important to develop deep continual learning methods.

Original: However, it is challenging to strike a balance between adapting to recent data and retaining knowledge from old data.
Revised: However, it is challenging to strike a balance between adapting to recent data and retaining knowledge from old data.

Original: Too much plasticity leads to the infamous catastrophic forgetting problem (McCloskey & Cohen, 1989; Ratcliff, 1990; Goodfellow et al., 2014a) and too much stability leads to an inability to adapt.
Revised: Too much plasticity leads to the infamous catastrophic forgetting problem (McCloskey & Cohen, 1989; Ratcliff, 1990; Goodfellow et al., 2014a) and too much stability leads to an inability to adapt.

Original: Recently there has been a resurgence of interest in this area.
Revised: Recently there has been a resurgence of interest in this area.

Original: One approach trains individual models on each task and then carries out a second stage of training to combine them (Lee et al., 2017).
Revised: One approach trains individual models on each task and then carries out a second stage of training to combine them (Lee et al., 2017).

Original: A more elegant and more ﬂexible approach maintains a single model and uses a single type of regularized training that prevents drastic changes in the parameters which have a large inﬂuence on prediction, but allows other parameters to change more freely (Li & Hoiem, 2016; Kirkpatrick et al., 2017; Zenke et al., 2017).
Revised: A more elegant and more ﬂexible approach maintains a single model and uses a single type of regularized training that prevents drastic changes in the parameters which have a large inﬂuence on prediction, but allows other parameters to change more freely (Li & Hoiem, 2016; Kirkpatrick et al., 2017; Zenke et al., 2017).

Original: The approach developed here follows this venerable work, but is arguably more principled, extensible and automatic.
Revised: The approach developed here follows this venerable work, but is arguably more principled, extensible and automatic.

Original: This paper is built on the observation that there already exists an extremely general framework for continual learning: Bayesian inference.
Revised: This paper is built on the observation that there already exists an extremely general framework for continual learning: Bayesian inference.

Original: Critically, Bayesian inference retains a distribution over model parameters that indicates the plausibility of any setting given the observed data.
Revised: Critically, Bayesian inference retains a distribution over model parameters that indicates the plausibility of any setting given the observed data.

Original: When new data arrive, we combine what previous data have told us about the model parameters (the previous posterior) with what the current data are telling us (the likelihood).
Revised: When new data arrive, we combine what previous data have told us about the model parameters (the previous posterior) with what the current data are telling us (the likelihood).

Original: Multiplying and renormalizing yields the new posterior, from which point we can recurse.
Revised: Multiplying and renormalizing yields the new posterior, from which point we can recurse.

Original: Critically, the previous posterior constrains parameters that strongly inﬂuence prediction, preventing them from changing drastically, but it allows other parameters to change.
Revised: Critically, the previous posterior constrains parameters that strongly inﬂuence prediction, preventing them from changing drastically, but it allows other parameters to change.

Original: The wrinkle is that exact Bayesian inference is typically intractable and so approximations are required.
Revised: The wrinkle is that exact Bayesian inference is typically intractable and so approximations are required.

Original: Fortunately, there is an extensive literature on approximate inference for neural networks.
Revised: Fortunately, there is an extensive literature on approximate inference for neural networks.

Original: We merge online variational inference (VI) (Ghahramani &
Revised: We merge online variational inference (VI) (Ghahramani &

Original: Attias, 2000; Sato, 2001; Broderick et al., 2013) with Monte Carlo VI for neural networks (Blundell et al., 2015) to yield variational continual learning (VCL).
Revised: Attias, 2000; Sato, 2001; Broderick et al., 2013) with Monte Carlo VI for neural networks (Blundell et al., 2015) to yield variational continual learning (VCL).

Original: In addition, we extend VCL to include a small episodic memory by combining VI with the coreset data summarization method (Bachem et al., 2015; Huggins et al., 2016).
Revised: In addition, we extend VCL to include a small episodic memory by combining VI with the coreset data summarization method (Bachem et al., 2015; Huggins et al., 2016).

Original: We demonstrate that the framework is general, applicable to both deep discriminative models and deep generative models, and that it yields excellent performance.
Revised: We demonstrate that the framework is general, applicable to both deep discriminative models and deep generative models, and that it yields excellent performance.

Original:  C ONTINUAL L EARNING BY A PPROXIMATE B AYESIAN I NFERENCE
Revised:  C ONTINUAL L EARNING BY A PPROXIMATE B AYESIAN I NFERENCE

Original: Consider a discriminative model that returns a probability distribution over an output y given an input x and parameters θ , that is p ( y | θ , x ) .
Revised: Consider a discriminative model that returns a probability distribution over an output y given an input x and parameters θ , that is p ( y | θ , x ) .

Original: Below we consider the speciﬁc case of a softmax distribution returned by a neural network with weight and bias parameters, but we keep the development general for now.
Revised: Below we consider the speciﬁc case of a softmax distribution returned by a neural network with weight and bias parameters, but we keep the development general for now.

Original: In the continual learning setting, the goal is to learn the parameters of the model from a set of sequentially arriving datasets { x ( n ) t , y ( n ) t } N t n =1 where, in principle, each might contain a single datum, N t = 1 .
Revised: In the continual learning setting, the goal is to learn the parameters of the model from a set of sequentially arriving datasets { x ( n ) t , y ( n ) t } N t n =1 where, in principle, each might contain a single datum, N t = 1 .

Original: Following a Bayesian approach, a prior distribution p ( θ ) is placed over θ .
Revised: Following a Bayesian approach, a prior distribution p ( θ ) is placed over θ .

Original: The posterior distribution after seeing T datasets is recovered by applying Bayes’ rule:
Revised: The posterior distribution after seeing T datasets is recovered by applying Bayes’ rule:

Original: [Equation]
Revised: [Equation]

Original: Here the input dependence has been suppressed on the right hand side to lighten notation.
Revised: Here the input dependence has been suppressed on the right hand side to lighten notation.

Original: We have used the shorthand D t = { y ( n ) t } N t n =1 .
Revised: We have used the shorthand D t = { y ( n ) t } N t n =1 .

Original: Importantly, a recursion has been identiﬁed whereby the posterior after seeing the T -th dataset is produced by taking the posterior after seeing the ( T − 1) -th dataset, multiplying by the likelihood and renormalizing.
Revised: Importantly, a recursion has been identiﬁed whereby the posterior after seeing the T -th dataset is produced by taking the posterior after seeing the ( T − 1) -th dataset, multiplying by the likelihood and renormalizing.

Original: In other words, online updating emerges naturally from Bayes’ rule.
Revised: In other words, online updating emerges naturally from Bayes’ rule.

Original: In most cases the posterior distribution is intractable and approximation is required, even when forming the ﬁrst posteriorp ( θ |D 1 )
Revised: In most cases the posterior distribution is intractable and approximation is required, even when forming the ﬁrst posterior

Original: ≈ q 1 ( θ ) =
Revised: [Equation]

Original: proj( p ( θ ) p ( D 1 | θ )) .
Revised: ( θ ) p ( D 1 | θ )) .

Original: Here q ( θ ) =
Revised: Here q ( θ ) =

Original: proj( p ∗ ( θ )) denotes a projection operation that takes the intractable un-normalized distribution p ∗ ( θ ) and returns a tractable normalized approximation q ( θ ) .
Revised: proj( p ∗ ( θ )) denotes a projection operation that takes the intractable un-normalized distribution p ∗ ( θ ) and returns a tractable normalized approximation q ( θ ) .

Original: The ﬁeld of approximate inference provides several choices for the projection operation including i) Laplace approximation, ii) variational KL minimization, iii) moment matching, and iv) importance sampling.
Revised: The ﬁeld of approximate inference provides several choices for the projection operation including i) Laplace’s approximation, ii) variational KL minimization, iii) moment matching, and iv) importance sampling.

Original: Having approximated the ﬁrst posterior distribution, subsequent approximations can be produced recursively by combining the approximate posterior distribution with the likelihood and projecting, that is p ( θ |D 1: T )
Revised: Having approximated the ﬁrst posterior distribution, subsequent approximations can be produced recursively by combining the approximate posterior distribution with the likelihood and projecting, that is p ( θ |D 1: T )

Original: ≈ q T ( θ ) = proj( q T − 1 ( θ ) p ( D T | θ )) .
Revised: ≈ q T ( θ ) = proj( q T − 1 ( θ ) p ( D T | θ )) .

Original: In this way online updating is supported.
Revised: In this way online updating is supported.

Original: This general approach leads, for the four projection operators previously identiﬁed, to i) Laplace propagation (Smola et al., 2004), ii) online VI (Ghahramani & Attias, 2000; Sato, 2001) also known as streaming VB (Broderick et al., 2013), iii) assumed density ﬁltering (Maybeck, 1982) and iv) sequential Monte Carlo (Liu & Chen, 1998).
Revised: This general approach leads, for the four projection operators previously identiﬁed, to i) Laplace propagation (Smola et al., 2004), ii) online VI (Ghahramani & Attias, 2000; Sato, 2001) also known as streaming variational Bayes (Broderick et al., 2013), iii) assumed density ﬁltering (Maybeck, 1982) and iv) sequential Monte Carlo (Liu & Chen, 1998).

Original: In this paper the online VI approach is used as it typically outperforms the other methods for complex models in the static setting (Bui et al., 2016) and yet it has not been applied to continual learning of neural networks.
Revised: In this paper the online VI approach is used as it typically outperforms the other methods for complex models in the static setting (Bui et al., 2016) and yet it has not been applied to continual learning of neural networks.

Original: 2. V ARIATIONAL C ONTINUAL L EARNING (VCL) AND E PISODIC M EMORY E NHANCEMENT
Revised: 2. V ARIATIONAL C ONTINUAL L EARNING (VCL) AND E PISODIC M EMORY E NHANCEMENT

Original: Variational continual learning employs a projection operator deﬁned through a KL divergence minimization over the set of allowed approximate posteriors Q ,
Revised: Variational continual learning employs a projection operator deﬁned through a KL divergence minimization over the set of allowed approximate posteriors Q ,

Original: [Equation]
Revised: [Equation]

Original: The zeroth approximate distribution is deﬁned to be the prior, q 0 ( θ ) =
Revised: The zeroth approximate distribution is deﬁned to be the prior, q 0 ( θ ) =

Original: p ( θ ) .
Revised: p ( θ ) .

Original: Z t is the intractable normalizing constant of p ∗  ( θ ) =
Revised: Z t is the intractable normalizing constant of p ∗ t ( θ ) =

Original: q t − 1 ( θ ) p
Revised: q t − 1 ( θ ) p

Original: ( D t | θ ) and is not required to compute the optimum.
Revised: ( D t | θ ) and is not required to compute the optimum.

Original: VCL will perform exact Bayesian inference if the true posterior is a member of the approximating family, p ( θ |D 1 , D 2 , . . .
Revised: VCL will perform exact Bayesian inference if the true posterior is a member of the approximating family, p ( θ |D 1 , D 2 , . . .

Original: , D t ) ∈ Q at every step t .
Revised: , D t ) ∈ Q at every step t .

Original: Typically this will not be the case and we might worry that performing repeated approximations may accumulate errors causing the algorithm to forget old tasks.
Revised: Typically this will not be the case and we might worry that performing repeated approximations may accumulate errors causing the algorithm to forget old tasks, for example.

Original: Furthermore, the minimization at each step is also  approximate (e.g. due to employing an additional Monte Carlo approximation) and so additional information may be lost.
Revised: Furthermore, the minimization at each step may also be approximate (e.g. due to employing an additional Monte Carlo approximation) and so additional information may be lost.

Original: In order to mitigate this potential problem, we extend VCL to include a small representative set of data from previously observed tasks that we call the coreset.
Revised: In order to mitigate this potential problem, we extend VCL to include a small representative set of data from previously observed tasks that we call the coreset.

Original: The coreset is analogous to an episodic memory that retains key information (in our case, important training data points) from previous tasks which the algorithm can revisit in order to refresh its memory of them.
Revised: The coreset is analogous to an episodic memory that retains key information (in our case, important training data points) from previous tasks which the algorithm can revisit in order to refresh its memory of them.

Original: This general idea ofusing an episodic memoryhas been explored for continual learning  by Lopez-Paz & Ranzato (2017). 
Revised: The use of an episodic memory for continual learning has also been explored by Lopez-Paz & Ranzato (2017). [Equation] [Equation] [Equation]

Revised: [Equation]

Revised: [Equation]

Original: Algorithm 1 describes coreset VCL.
Revised: Algorithm 1 describes coreset VCL.

Original: The algorithm maintains a pair ( C t , ˜ q t ) comprising the coreset and  the variational distribution for non-coreset data  at each time step t . 
Revised: For each task, the new coreset C t is produced by selecting new data points for the current task and a selection from the old coreset C t − 1 . Any heuristic can be used to make these selections, e.g. K data points can be selected at random from D t and added to C t −to form an unbiased new coreset C t . Alternatively, the greedy K -center algorithm (Gonzalez, 1985) can be used to return K data points that are guaranteed to be spread throughout the input space. Next, a variational recursion is developed.

Revised: Bayes’ rule can be used to decompose the true posterior taking care to break out contributions from coreset,

Revised: [Equation]

Revised: [Table]

Original: The distribution ˜ q t is continuously updated using eq.
Revised: [Equation] q [Equation] p

Revised: [Equation]

Revised: |

Original: When performing prediction, ˜ q t and C t are combined toform the ﬁnal
Revised: [Equation] Here the variational distribution ˜ q t ( θ ) approximates the contribution to the posterior from the noncoreset data points. A recursion is identiﬁed by noting [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: variational distribution q t using eq.
Revised: Hence propagation is performed via ˜ q t ( θ ) =

Original: This step ensures recent exposure to data from each task and helps mitigate any residual forgetting. From a more general perspective, coreset VCL is equivalent to a message-passing implementation of VI in which the coreset data point updates are scheduled after updating the other data.
Revised: proj (˜ q t − 1 ( θ ) p

Original: The coreset VCL algorithm adds new data points to the coreset after observing each new dataset D t (it is simple to remove them too if T is large).
Revised: ( D t ∪ C t − 1 \ C t | θ )) with VCL employing the variational KL projection. A further projection step is needed before performing prediction q t ( θ ) =

Original: Any method can be used to update the coreset. For example, K data points can be selected at random from D t and added to C t − 1 to form an unbiased new coreset C t  . 
Revised: proj (˜ q t ( θ ) p ( C t | θ )) . In this way the coreset is incorporated into the approximate posterior directly before prediction which helps mitigate any residual forgetting.

Original: Alternatively, the greedy K -center algorithm (Gonzalez, 1985) can be used to return K data points that are guaranteed to be spread throughout the input space.
Revised: From a more general perspective, coreset VCL is equivalent to a message-passing implementation of VI in which the coreset data point updates are scheduled after updating the other data.

Original:  V ARIATIONAL C ONTINUAL L EARNING IN D EEP D ISCRIMINATIVE M ODELS
Revised:  V ARIATIONAL C ONTINUAL L EARNING IN D EEP D ISCRIMINATIVE M ODELS

Original: The VCL framework is general and can be applied to many discriminative probabilistic models.
Revised: The VCL framework is general and can be applied to many discriminative probabilistic models.

Original: Here we apply it to continual learning of deep fully-connected neural network classiﬁers.
Revised: Here we apply it to continual learning of deep fully-connected neural network classiﬁers.

Original: Before turning to the application of VCL, we ﬁrst consider the architecture of neural networks suitable for performing continual learning.
Revised: Before turning to the application of VCL, we ﬁrst consider the architecture of neural networks suitable for performing continual learning.

Original: In simple instances of discriminative continual learning, where data are arriving in an i.i.d.
Revised: In simple instances of discriminative continual learning, where data are arriving in an i.i.d.

Original: way or where only the input distribution p ( x 1: T ) changes over time, a standard single-head discriminative neural network sufﬁces.
Revised: way or where only the input distribution p ( x 1: T ) changes over time, a standard single-head discriminative neural network sufﬁces.

Original: In many cases, the tasks, although related, might involve different output variables.
Revised: In many cases the tasks, although related, might involve different output variables.

Original: Standard practice in multi-task learning (Bakker &
Revised: Standard practice in multi-task learning (Bakker &

Original: Heskes, 2003) uses networks that share parameters close to the inputs but with separate heads for each output, hence multi-head networks.
Revised: Heskes, 2003) uses networks that share parameters close to the inputs but with separate heads for each output, hence multi-head networks.

Original: Graphical models depicting the network architecture for deep discriminative and deep generative models are shown in ﬁg. 1.
Revised: Graphical models depicting the network architecture for deep discriminative and deep generative models are shown in ﬁg. 1.

Revised: Recent work has explored more advanced structures for continual learning (Rusu et al., 2016) and multi-task learning more generally (Swietojanski & Renals, 2014; Rebufﬁ et al., 2017).

Revised: These architectural advances are complementary to the new learning schemes developed here and a synthesis of the two would be potentially more powerful.

Original: A general solution to continual learning  would perform automatic  model building that adds new  structure to the existing model as new tasks are encountered (Rusu et al., 2016).
Revised: Moreover, a general solution to continual learning [Figure] would perform automatic continual model building adding new bespoke structure to the existing model as new tasks are encountered.

Original: Although this is an interesting research direction, here we make the simplifying assumption that the model structure is known a priori .
Revised: Although this is a very interesting research direction, here we make the simplifying assumption that the model structure is known a priori .

Original: VCL requires speciﬁcation of q ( θ ) where θ in the current case is a D dimensional vector formed by stacking the network’s biases and weights.
Revised: VCL requires speciﬁcation of q ( θ ) where θ in the current case is a D dimensional vector formed by stacking the network’s biases and weights.

Original: For simplicity we use a Gaussian mean-ﬁeld approximate posterior q t ( θ ) =
Revised: For simplicity we use a Gaussian mean-ﬁeld approximate posterior q t ( θ ) =

Original: (cid:81)
Revised: (cid:81)

Original: Dd =1 N ( θ t,d ; µ t,d , σ 2 t,d ) .
Revised: Dd =1 N ( θ t,d ; µ t,d , σ 2 t,d ) .

Original: Taking the most general case of a multi-head network, before task k is encountered the posterior distribution over the associated head parameters will remain at the prior and so q ( θ Hk )
Revised: Taking the most general case of a multi-head network, before task k is encountered the posterior distribution over the associated head parameters will remain at the prior and so q ( θ Hk )

Original: =
Revised: =

Original: p ( θ Hk ) .
Revised: p ( θ Hk ) .

Original: This is convenient as it means the variational approximation can be grown incrementally, starting from the prior, as each task emerges.
Revised: This is convenient as it means the variational approximation can be grown incrementally, starting from the prior, as each task emerges.

Original: Moreover, only tasks present in the current dataset D t need to have their posterior distributions over head parameters updated.
Revised: Moreover, only tasks present in the current dataset D t need to have their posterior distributions over head parameters updated.

Original: The shared parameters, on the other hand, will be constantly updated.
Revised: The shared parameters, on the other hand, will be constantly updated.

Original: Training the network using the VFE approach in eq.
Revised: Training the network using the VFE approach in eq.

Original: (1) is equivalent to maximizing the negative online variational free energy or the variational lower bound to the online marginal likelihood:
Revised: (1) is equivalent to maximizing the negative online variational free energy or the variational lower bound to the online marginal likelihood

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Figure]
Revised: with respect to the variational parameters { µ t,d , σ t,d } D d =1 .

Original: Whilst the KL-divergence
Revised: Whilst the KL-divergence

Original: KL( q t ( θ ) ||
Revised: KL( q t ( θ ) ||

Original: q t − 1 ( θ )) can be computed in closed-form, the expected log-likelihood requires further approximation.
Revised: q t − 1 ( θ )) can be computed in closed-form, the expected log-likelihood requires further approximation.

Original: Here we take the usual approach of employing simple Monte Carlo and use the  reparameterization trick to compute the gradients (Salimans & Knowles, 2013; Kingma & Welling, 2014).
Revised: Here we take the usual approach of employing simple Monte Carlo and use the local reparameterization trick to compute the gradients (Salimans & Knowles, 2013; Kingma & Welling, 2014; Kingma et al., 2015).

Original: At the  time step, the prior distribution, and therefore q 0 ( θ ) is chosen to be a multivariate Gaussian distribution (see e.g. Graves, 2011; Blundell et al., 2015).
Revised: At the ﬁrst time step, the prior distribution, and therefore q 0 ( θ ) is chosen to be a multivariate Gaussian distribution (see e.g. Graves, 2011; Blundell et al., 2015).

Original:  V ARIATIONAL C ONTINUAL L EARNING IN D EEP G ENERATIVE M ODELS
Revised:  V ARIATIONAL C ONTINUAL L EARNING IN D EEP G ENERATIVE M ODELS

Original: Deep generative models (DGMs) have garnered much recent attention.
Revised: Deep generative models (DGMs) have garnered much recent attention.

Original: By passing a simple noise variable (e.g. Gaussian noise) through a deep neural network, these models have been shown to be able to generate realistic images, sounds and videos sequences (Chung et al., 2015; Kingma et al., 2016; Vondrick et al., 2016).
Revised: By passing a simple noise variable (e.g. Gaussian noise) through a deep neural network, these models have been shown to be able to generate realistic images, sounds and videos sequences (Chung et al., 2015; Kingma et al., 2016; Vondrick et al., 2016).

Original: Standard approaches for learning DGMs have focused onthe batch learning, i.e. the observed instances are assumed to be i.i.d.
Revised: Standard approaches for learning DGMs have focused on batch learning, i.e. the observed instances are assumed to be i.i.d.

Original: and are all available at the same time.
Revised: and are all available at the same time.

Original: In this section we extend the VCL framework to encompass variational auto-encoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014), a form of DGMs.
Revised: In this section we extend the VCL framework to encompass variational auto-encoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014), a form of DGM.

Original: The approach could be extended to generative adversarial networks (GAN) (Goodfellow et al., 2014b) for which continual learning is an open problem, but see Seff et al.
Revised: The approach could be extended to generative adversarial networks (GANs) (Goodfellow et al., 2014b) for which continual learning is an open problem (see Seff et al.

Original: for an initial attempt.
Revised: for an initial attempt).

Original: Consider a model p ( x | z , θ ) p ( z ) , for observed data x and latent variables z .
Revised: Consider a model p ( x | z , θ ) p ( z ) , for observed data x and latent variables z .

Original: The prior over latent variables p ( z ) is typically Gaussian, and the distributional parameters of p ( x | z , θ ) are deﬁned by a deep neural network.
Revised: The prior over latent variables p ( z ) is typically Gaussian, and the distributional parameters of p ( x | z , θ ) are deﬁned by a deep neural network.

Original: For example, if Bernoulli likelihood is used, then p ( x | z , θ ) =
Revised: For example, if Bernoulli likelihood is used, then p ( x | z , θ ) =

Original: Bern(  f θ ( z )) , where f θ denotes the deep neural network transform and θ collects all the weight matrices and bias vectors.
Revised: Bern( x ; f θ ( z )) , where f θ denotes the deep neural network transform and θ collects all the weight matrices and bias vectors.

Original: In the batch setting, given a dataset D = { x  n  } Nn =1 , the standard VAE approach learns the parameters θ by approximate maximum likelihood estimation (MLE).
Revised: In the batch setting, given a dataset D = { x ( n ) } Nn =1 , the standard VAE approach learns the parameters θ by approximate maximum likelihood estimation (MLE).

Original: This proceeds by maximizing the variational lower bound with respect to θ and φ :
Revised: This proceeds by maximizing the variational lower bound with respect to θ and φ :

Original: [Equation]
Revised: [Equation]

Original: where φ denotes the variational parameters for the “encoder” or the approximate posterior of z given an observation x n .
Revised: where φ are the variational parameters of the approximate posterior or “encoder”.

Original: The approximate MLE approach is unsuitable for the continual learning setting as it does not return parameter uncertainty estimates that are critical for weighting the information learned from old data.
Revised: The approximate MLE approach is unsuitable for the continual learning setting as it does not return parameter uncertainty estimates that are critical for weighting the information learned from old data.

Original: So, instead the VCL approach will approximate the full posterior distribution over parameters, q t ( θ )
Revised: So, instead the VCL approach will approximate the full posterior distribution over parameters, q t ( θ )

Original: ≈ p ( θ |D 1: t ) , after observing the t -th dataset.
Revised: ≈ p ( θ |D 1: t ) , after observing the t -th dataset.

Original: Speciﬁcally, the approximate posterior q t is obtained by  the  variational lower bound with respect to  and φ :
Revised: Speciﬁcally, the approximate posterior q t is obtained by maximizing the full variational lower bound with respect to q t and φ :

Revised: [Equation]

Revised: n ) t ) (cid:35)(cid:41) − KL(

Original: [Equation]
Revised: [Equation]

Original:  where the encoder network q φ ( z  | x ( n ) t ) is parameterized by φ which is task-speciﬁc.
Revised: ) , where the encoder network q φ ( z ( n ) t | x ( n ) t ) is parameterized by φ which is task-speciﬁc.

Original: It is likely to be beneﬁcial to share (parts of) these encoder networks, but this is not investigated in this paper.
Revised: It is likely to be beneﬁcial to share (parts of) these encoder networks, but this is not investigated in this paper.

Original: As was the case for multi-head discriminative models, we can split the generative model into shared and task-speciﬁc parts.
Revised: As was the case for multi-head discriminative models, we can split the generative model into shared and task-speciﬁc parts.

Original: There are two options: (i) the generative models share across tasks the network that generates observations x from the intermediate-level representations h , but have their own “head network” for generating h from the latent variables z (see ﬁg. 1(b)), and (ii) the other way around.
Revised: There are two options: (i) the generative models share across tasks the network that generates observations x from the intermediate-level representations h , but have private “head networks” for generating h from the latent variables z (see ﬁg. 1(b)), and (ii) the other way around.

Original: In the latter architecture, if the task-speciﬁc lower-level network is reasonably deep, the information of  the current task can be encoded efﬁciently inthis part of the network, thus effectively ignoring the shared high-level network. Due to this reason, we focus onthe former architecture  in the experiments where the network is forced to share information between tasks.
Revised: Architecture (i) is arguably more appropriate when data are composed of a common set of structural primitives (such as strokes in handwritten digits) that are selected by high level variables (character identities). Moreover, initial experiments on architecture (ii) indicated that information about the current task tended to be encoded entirely in the task-speciﬁc lower-level network negating multi-task transfer. For these reasons, we focus on architecture (i) in the experiments.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Continual Learning for Deep Discriminative Models:
Revised: Continual Learning for Deep Discriminative Models:

Original: Many neural network continual learning approaches employ regularized maximum likelihood estimation, optimizing objectives of the form:
Revised: Many neural network continual learning approaches employ regularized maximum likelihood estimation, optimizing objectives of the form:

Original: [Equation]
Revised: [Equation]

Original: Here the regularization biases the new parameter estimates towards those estimated at the previous step θ t − 1 .
Revised: Here the regularization biases the new parameter estimates towards those estimated at the previous step θ t − 1 .

Original: λ t is a user-selected hyper-parameter that controls the overall contribution from previous data and Σ t − 1 is a matrix (normally diagonal in form) that encodes the relative strength of the regularization on each element of θ .
Revised: λ t is a user-selected hyper-parameter that controls the overall contribution from previous data and Σ t − 1 is a matrix (normally diagonal in form) that encodes the relative strength of the regularization on each element of θ .

Original: We now discuss speciﬁc instances of this scheme:
Revised: We now discuss speciﬁc instances of this scheme:

Original: • Maximum-likelihood estimation and MAP estimation : maximum likelihood estimation is recovered when there is no regularization ( λ t = 0 ).
Revised: • Maximum-likelihood estimation and MAP estimation : maximum likelihood estimation is recovered when there is no regularization ( λ t = 0 ).

Original: More generally, the regularization term can be interpreted as a Gaussian prior, q ( θ |D 1: t − 1 ) =
Revised: More generally, the regularization term can be interpreted as a Gaussian prior, q ( θ |D 1: t − 1 ) =

Original: N ( θ ; θ t − 1 , Σ t − 1 /λ t ) .
Revised: N ( θ ; θ t − 1 , Σ t − 1 /λ t ) .

Original: The optimization returns the maximum a posteriori estimate of the parameters, but this does not directly providea Σ t for the next stage.
Revised: The optimization returns the maximum a posteriori estimate of the parameters, but this does not directly provide Σ t for the next stage.

Original: A simple ﬁx is to set Σ t
Revised: A simple ﬁx is to set Σ t

Original: =
Revised: =

Original: I and use cross-validation to ﬁnd λ t , but this approximation is often coarse and leads to catastrophic forgetting (Goodfellow et al., 2014a; Kirkpatrick et al., 2017).
Revised: I and use cross-validation to ﬁnd λ t , but this approximation is often coarse and leads to catastrophic forgetting (Goodfellow et al., 2014a; Kirkpatrick et al., 2017).

Original: • Laplace Propagation  (Smola et al., 2004): applying Laplace approximation at each step leads to a recursion for Σ − 1 t , which is initialized using the covariance of the Gaussian prior, Σ− 1 t
Revised: • Laplace Propagation (LP) (Smola et al., 2004): applying Laplace’s approximation at each step leads to a recursion for Σ − 1 t , which is initialized using the covariance of the Gaussian prior, Σ

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: =
Revised: λ

Original: [Equation]
Revised: [Equation]

Original:  To avoid computing the full Hessian of the likelihood, diagonal-Laplace propagation retains only the diagonal terms of Σ − 1 t .
Revised: = 1 . To avoid computing the full Hessian of the likelihood, diagonal-Laplace propagation retains only the diagonal terms of Σ − 1 t .

Original: • Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) builds on diagonal-Laplace propagation by approximating the average Hessian of the likelihoods using well-known identities for the Fisher information:
Revised: • Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) builds on diagonal-Laplace propagation by approximating the average Hessian of the likelihoods using well-known identities for the Fisher information:

Original: [Equation]
Revised: [Equation]

Original: θ = θ t (cid:19) .

Original: EWC also modiﬁes the Laplace regularization,12 ( θ − θ t − 1 ) (cid:124)
Revised: EWC also modiﬁes the Laplace regularization,

Original: (Σ 0 + (cid:80) t − 1 t (cid:48) = 1 Φ t (cid:48) )
Revised: [Equation]

Original: ( θ − θ t − 1 ) , introducing hyper-parameters, removing the prior and regularizing to intermediate parameter estimators, rather than just that derived from the last task,12 (cid:80) t − 1 t (cid:48)
Revised: , introducing hyper-parameters, removing the prior and regularizing to intermediate parameter estimates, rather than just those derived from the last task,

Original: =1 λ t (cid:48) ( θ − θ t (cid:48) − 1 ) (cid:124)
Revised: [Equation]

Original: Φ t (cid:48) ( θ − θ  t(cid:48) − 1 ) . This ﬁnal change is likely to be unnecessary and deleterious in some cases (Huszar, 2017).
Revised: These changes may be unnecessary (Husz´ar, 2017; 2018) and require storing θ 1: t − 1 , but may slightly improve performance (see Kirkpatrick et al. (2018) and our experiments).

Original: • Synaptic Intelligence (SI) (Zenke et al., 2017): SI computes Σ − 1 t using a measure of the importance of each parameter to each task.
Revised: • Synaptic Intelligence (SI) (Zenke et al., 2017): SI computes Σ − 1 t using a measure of the importance of each parameter to each task.

Original: Practically, this is achieved by comparing the changing rate of the gradients of the objective and the changing rate of the parameters.
Revised: Practically, this is achieved by comparing the changing rate of the gradients of the objective and the changing rate of the parameters.

Original: VCL differs from the above methods in several ways.
Revised: VCL differs from the above methods in several ways.

Original: First, unlike MAP, EWC and SI approaches above, it does not have free parameters that need to be tuned on a validation set.
Revised: First, unlike MAP, EWC and SI, it does not have free parameters that need to be tuned on a validation set.

Original: This can be especially awkward in the online setting.
Revised: This can be especially awkward in the online setting.

Original: Second, although the KL regularization penalizes the mean of the approximate posterior through a quadratic cost, a full distribution is retained andneeds to be averaged over at training time and at test time.
Revised: Second, although the KL regularization penalizes the mean of the approximate posterior through a quadratic cost, a full distribution is retained and averaged over at training time and at test time

Original: Third,and more generally, VI is generally thought to return better uncertainty estimates than approaches like Laplace method and MAP estimation, and we have argued this is critical for continual learning.
Revised: Third, VI is generally thought to return better uncertainty estimates than approaches like Laplace’s method and MAP estimation, and we have argued this is critical for continual learning.

Original: There is a long history of research on approximate Bayesianapproaches to training  neural networks, including extended Kalman ﬁltering (Singhal & Wu, 1989), Laplace approximation (MacKay, 1992), variational inference (Hinton & Van Camp, 1993; Barber & Bishop, 1998; Graves, 2011; Blundell et al., 2015; Gal & Ghahramani, 2016), sequential Monte Carlo (de Freitas et al., 2000), EP (Hern´andez-Lobato & Adams, 2015), and approximate Power EP (Hern´andez-Lobato et al., 2016).
Revised: There is a long history of research on approximate Bayesian training of neural networks, including extended Kalman ﬁltering (Singhal & Wu, 1989), Laplace’s approximation (MacKay, 1992), variational inference (Hinton & Van Camp, 1993; Barber & Bishop, 1998; Graves, 2011; Blundell et al., 2015; Gal & Ghahramani, 2016), sequential Monte Carlo (de Freitas et al., 2000), expectation propagation (EP) (Hern ´ andez-Lobato & Adams, 2015), and approximate power EP (Hern ´ andez-Lobato et al., 2016).

Original: These approaches havebeen focused on batch learning, but the framework described in sectionalso enables them to be applied to continual learning.
Revised: These approaches have focused on batch learning, but the framework described in section 2 enables them to be applied to continual learning.

Original: On the other hand,the online variational inference has been previously explored (Ghahramani & Attias, 2000; Broderick et al., 2013; Bui et al., 2017), but not for neural networks or in the context of  complex tasks.
Revised: On the other hand, online variational inference has been previously explored (Ghahramani & Attias, 2000; Broderick et al., 2013; Bui et al., 2017), but not for neural networks or in the context of sets of related complex tasks.

Original: Continual Learning for Deep Generative Models: A na¨ıve continual learning approach for deep generative models would directly apply the VAE algorithm to the new dataset D t with the model parameters initialized at the previous parameter values θ t − 1 .
Revised: Continual Learning for Deep Generative Models: A na ¨ ıve continual learning approach for deep generative models would directly apply the VAE algorithm to the new dataset D t with the model parameters initialized at the previous parameter values θ t − 1 .

Original: Our experiments show that this approach leads to catastrophic forgetting, in the sense that the generator can only generate instances that are similar to the data points from the most recently observed dataset.
Revised: The experiments show that this approach leads to catastrophic forgetting, in the sense that the generator can only generate instances that are similar to the data points from the most recently observed task.

Original: Alternatively, EWC regularization can be added to the VAE objective:
Revised: Alternatively, EWC regularization can be added to the VAE objective:

Original: [Equation]
Revised: [Equation]

Original: However computing Φ t requires the gradient  ∇ θ log p ( x | θ ) which is intractable.
Revised: However computing Φ t requires the gradient of the intractable marginal likelihood ∇ θ log p ( x | θ ) .

Original: Instead, we can approximate the marginal likelihood by the variational lower bound,i.e.
Revised: Instead, we can approximate the marginal likelihood by the variational lower bound,

Original: [Equation]
Revised: [Equation]

Revised: Similar variational lower-bound approximations apply when computing the Hessian matrices for LP and Σ − 1 t for SI.

Original: An importance sampling estimate could also be used using Burda et al.
Revised: An importance sampling estimate could also be used (Burda et al., 2016).  E XPERIMENTS

Original: In our experiments with DGMs, this version of EWC is used as a baseline to compare with VCL.
Revised: The experiments evaluate the performance and ﬂexibility of VCL through three discriminative tasks and two generative tasks. Standard continual learning benchmarks are used where possible. Comparisons are made to EWC, diagonal LP and SI that employ tuned hyper-parameters λ whereas VCL’s objective is hyper-parameter free.

Revised: More details of the experiment settings and an additional experiment are available in the appendix.

Original: E XPERIMENTS 6. E XPERIMENTS WITH D EEP D ISCRIMINATIVE M ODELS
Revised: [Figure] [Figure] 6. E XPERIMENTS WITH D EEP D ISCRIMINATIVE M ODELS

Original: We consider  three continual learning experiments for deep discriminative models: the permuted
Revised: We consider the following three continual learning experiments for deep discriminative models.

Original: MNIST, split MNIST, and split notMNIST experiments.

Original: More details of the experiment settings and an additional experiment are available in the appendix.

Original: Permuted MNIST:
Revised: Permuted MNIST:

Original: This is a popular continual learning benchmark (Goodfellow et al., 2014a; Kirkpatrick et al., 2017; Zenke et al., 2017).
Revised: This is a popular continual learning benchmark (Goodfellow et al., 2014a; Kirkpatrick et al., 2017; Zenke et al., 2017).

Original: The dataset received at each time step D t consists of labeled
Revised: The dataset received at each time step D t consists of labeled

Original: MNIST images whose pixels have undergone a ﬁxed random permutation.
Revised: MNIST images whose pixels have undergone a ﬁxed random permutation.

Original: We compare VCL to
Revised: We compare VCL to

Original: EWC and SI.
Revised: EWC, SI, and diagonal LP.

Original: For all algorithms, we use fully connected single-head networks with two hidden layers, where each layer contains 100 hidden units with ReLU activations.
Revised: For all algorithms, we use fully connected single-head networks with two hidden layers, where each layer contains 100 hidden units with ReLU activations.

Original: We evaluate three versions of VCL: VCL with no coreset, VCL with a random coreset, and VCL with a coreset selected by the K-center method.
Revised: We evaluate three versions of VCL: VCL with no coreset, VCL with a random coreset, and VCL with a coreset selected by the K-center method.

Original: For the coresets, we select 200 data points from each task.
Revised: For the coresets, we select 200 data points from each task.

Original: Figure 2 compares the average test set accuracy on all observed tasks.
Revised: Figure 2 compares the average test set accuracy on all observed tasks.

Original: From this ﬁgure, VCL outperforms the EWC and SI by large margins, even though they beneﬁted from an extensive hyperparameter search for λ . 
Revised: From this ﬁgure, VCL outperforms EWC, SI, and LP by large margins, even though they beneﬁted from an extensive hyperparameter search for λ . Diagonal LP performs slightly worse than EWC both when λ

Revised: = 1 and when the values of λ are tuned.

Original: After 10 tasks, VCL achieves 90% average accuracy, while EWC and SI only achieve 84% and 86% respectively.
Revised: After 10 tasks, VCL achieves 90% average accuracy, while EWC, SI, and LP only achieve 84%, 86%, and 82% respectively.

Original: The results also show thatusing the coresets alone is not useful for this task, but combining them with VCL leads to a modest improvement over VCL: both random coresets and K-center coresets achieve 93% accuracy.
Revised: The results also show that the coresets perform poorly by themselves, but combining them with VCL leads to a modest improvement: both random coresets and K-center coresets achieve 93% accuracy.

Original: We also investigate the effect of the coreset sizes in this experiment.
Revised: We also investigate the effect of the coreset size.

Original: In ﬁg. 3, we plot the average test set accuracy of VCL with random coresets of different sizes.
Revised: In ﬁg. 3, we plot the average test set accuracy of VCL with random coresets of different sizes.

Original: At the coreset size of 5,000 examples per task, VCL achieves 95.5% accuracy after 10 tasks, which is signiﬁcantly better than the 90% of vanilla VCL.
Revised: At the coreset size of 5,000 examples per task, VCL achieves 95.5% accuracy after 10 tasks, which is signiﬁcantly better than the 90% of vanilla VCL.

Original: From the ﬁgure, larger coreset sizes are better for VCL although the improvement, of course, asymptotes as the coreset size grows.

Original: Furthermore, if a sufﬁciently large coreset is employed,[Figure] [Figure] it will be fully representative of the task and thus training on the coreset alone can achieve a good performance.
Revised: Performance improves with the coreset size although it asymptotes for large coresets as expected: if a sufﬁciently large coreset is employed, it will be fully representative of the task and thus training on the coreset alone can achieve a good performance.

Original: However, the experiments show that the combination of online learning and coresets is advantageous even for large coresets. 
Revised: However, the experiments show that the combination of VCL and coresets is advantageous even for large coresets. [Figure]

Original: Split MNIST:
Revised: Split MNIST:

Original: This experiment was used by Zenke et al.
Revised: This experiment was used by Zenke et al.

Original: (2017) to assess the SI method.
Revised: (2017) to assess the SI method.

Original: Five binary classiﬁcation tasks from the MNIST dataset arrive in sequence: 0/1, 2/3, 4/5, 6/7, and 8/9.
Revised: Five binary classiﬁcation tasks from the MNIST dataset arrive in sequence: 0/1, 2/3, 4/5, 6/7, and 8/9.

Original: We use fully connected multi-head networks with two hidden layers comprising 256 hidden units with
Revised: We use fully connected multi-head networks with two hidden layers comprising 256 hidden units with

Original: ReLU activations.
Revised: ReLU activations.

Original: We compare VCL (with and without coresets) to EWC and SI.
Revised: We compare VCL (with and without coresets) to EWC, SI, and diagonal LP.

Original: For the coresets, 40 data points from each task are selected through random sampling or the K-center method.
Revised: For the coresets, 40 data points from each task are selected through random sampling or the K-center method.

Original: Figure 4 compares the test set accuracy on individual tasks (averaged over 10 runs) as well as the accumulated accuracy averaged over tasks (right most column).
Revised: Figure 4 compares the test set accuracy on individual tasks (averaged over 10 runs) as well as the accumulated accuracy averaged over tasks (right).

Original: As an upper bound on the algorithms’ performance, we compare to batch VI trained on the full dataset.
Revised: As an upper bound on the algorithms’ performance, we compare to batch VI trained on the full dataset.

Original: From this ﬁgure, VCL signiﬁcantly outperforms EWC  although it is slightly worse than SI.
Revised: From this ﬁgure, VCL signiﬁcantly outperforms EWC and LP although it is slightly worse than SI.

Original: Again,  EWC and SI beneﬁted from a hyper-parameter search for λ , but a value close to 1 performs well in both cases.
Revised: Again, unlike VCL, EWC and SI beneﬁted from a hyper-parameter search for λ , but a value close to 1 performs well in both cases.

Original: After 5 tasks,
Revised: After 5 tasks,

Original: VCL achieves 97.0% average accuracy on all tasks, while EWC and SI attain 63.1% and 98.9% respectively.
Revised: VCL achieves 97.0% average accuracy on all tasks, while EWC, SI, and LP attain 63.1%, 98.9%, and 61.2% respectively.

Original: Adding the coreset improvesthe performance of VCL to around 98.4% accuracy.
Revised: Adding the coreset improves VCL to around 98.4% accuracy.

Original: Split notMNIST: This experiment is similar to the previous experiment, although we use the more challenging notMNIST dataset and deeper networks.
Revised: Split notMNIST: This experiment is similar to the previous one, but it uses the more challenging notMNIST dataset and deeper networks.

Original: The notMNIST dataset 1 here contains 400, images of the characters from A to J with different font styles.
Revised: The notMNIST dataset 2 here contains 400,000 images of the characters from A to J with different font styles.

Original: This dataset is much larger and more noisy than the MNIST dataset.

Original: In this experiment, we consider ﬁve binary classiﬁcation tasks:
Revised: We consider ﬁve binary classiﬁcation tasks:

Original: A/F, B/G, C/H, D/I, and E/J. We use deeper networks consisting of four hidden layers, each ofwhich contains 150 hidden units with ReLU activations.
Revised: A/F, B/G, C/H, D/I, and E/J using deeper networks comprising four hidden layers of 150 hidden units with ReLU activations.

Original: The other settings are kept the same as the previous experiment.
Revised: The other settings are kept the same as the previous experiment.

Original: [Figure] [Figure] From the results in ﬁg. 5, VCL is competitive with SI and signiﬁcantly outperforms EWC, although these baselines beneﬁted from a hyper-parameter search.
Revised: VCL is competitive with SI and signiﬁcantly outperforms EWC and LP (see ﬁg. 5), although the SI and EWC baselines beneﬁted from a hyper-parameter search.

Original: VCL achieves 92.0% average accuracy after 5 tasks, while EWC and SI attain 71% and 94% respectively.
Revised: VCL achieves 92.0% average accuracy after 5 tasks, while EWC, SI, and LP attain 71%, 94%, and 63% respectively.

Original: Adding the random coreset improves the performance of VCL to 96% accuracy.
Revised: Adding the random coreset improves the performance of VCL to 96% accuracy.

Original: 6. E XPERIMENTS WITH D EEP G ENERATIVE M ODELS
Revised: 6. E XPERIMENTS WITH D EEP G ENERATIVE M ODELS

Original: We consider two continual learning experiments for deep generative models: MNIST digit generation and notMNIST (small) character generation.
Revised: We consider two continual learning experiments for deep generative models: MNIST digit generation and notMNIST (small) character generation.

Original: In both cases, ten datasets are received in sequence.
Revised: In both cases, ten datasets are received in sequence.

Original: For MNIST, the ﬁrst dataset comprises exclusively of images of the digit zero, the second dataset ones and so on.
Revised: For MNIST, the ﬁrst dataset comprises exclusively of images of the digit zero, the second dataset ones and so on.

Original: For notMNIST (small), the datasets contain the characters A to J in sequence.
Revised: For notMNIST, the datasets contain the characters A to J in sequence.

Original: The generative model consists of shared and task-speciﬁc components, each represented by a one hidden layer neural network with 500 hidden units (see ﬁg. 1(b)).
Revised: The generative model consists of shared and task-speciﬁc components, each represented by a one hidden layer neural network with 500 hidden units (see ﬁg. 1(b)).

Original: The dimensionality of the latent variable z and the intermediate representation h are 50 and 500, respectively.
Revised: The dimensionality of the latent variable z and the intermediate representation h are 50 and 500, respectively.

Original: We use task-speciﬁc encoders that are neural networks with symmetric architectures to the generator.
Revised: We use task-speciﬁc encoders that are neural networks with symmetric architectures to the generator.

Original: We compare VCL to na¨ıve online learning using the standard VAE objective and EWC  (with hyperparameters λ = 1 , 10 , 100 ).
Revised: We compare VCL to na¨ıve online learning using the standard VAE objective, LP, EWC and SI (with hyper-parameters λ = 1 , 10 , 100 ).

Original: For full details of the experimental settings, see appendix E. Samples from the generative models attained at different time steps are shown in ﬁg. 6.
Revised: For full details of the experimental settings see Appendix E. Samples from the generative models attained at different time steps are shown in ﬁg. 6.

Original: The na ¨ ıve online learning method fails catastrophically and sowe have not included numerical results in the main text.
Revised: The na¨ıve [Figure] online learning method fails catastrophically and so numerical results are omitted.

Original: EWC and VCLmanage to rememberthe previous tasks.
Revised: LP, EWC, SI and VCL remember previous tasks, with SI and VCL achieving the best visual quality on both datasets.

Original: EWC and VCL are quantitatively evaluated using two metrics in ﬁg. 7: an importance sampling estimate of the test log-likelihood (test-LL) using 5 , 000 samples and a measure of quality we term “classiﬁer uncertainty”.
Revised: The algorithms are quantitatively evaluated using two metrics in ﬁg. 7: an importance sampling estimate of the test log-likelihood (test-LL) using 5 , 000 samples and a measure of quality we term “classiﬁer uncertainty”.

Original: For the latter, we train a discriminative classiﬁer for the digits/alphabets.
Revised: For the latter, we train a discriminative classiﬁer for the digits/alphabets to achieve high accuracy.

Original: [Figure] The quality of generated samples can then be assessed bycomputing the entropy of the output classiﬁcation probability vector.
Revised: The quality of generated samples can then be assessed by the KL-divergence from the one-hot vector indicating the task, to the output classiﬁcation probability vector computed on the generated images.

Original: A well-trained generator will produce images that are conﬁdently classiﬁed  resulting in zero entropy.
Revised: A well-trained generator will produce images that are correctly classiﬁed in high conﬁdence resulting in zero KL.

Original: Initially, VCL performs slightly worse in terms of test-LL on the  mostrecent task when compared to EWC. This is likely due to the fact that VCL optimizes a distribution over random weights θ while EWC computes point estimates, and  the latter strategy often produces higher test-LL.
Revised: We only report the best performance for LP, EWC and SI. We observe that LP and EWC perform similarly, most likely due to the fact that both LP and EWC use the same Σ t matrices. EWC achieves signiﬁcantly worse performance than SI.

Original: However, importantly VCL has a superior long-term memory of previous tasks which leads to better overall performance on both metrics. Again, we note that EWC beneﬁted here from carefully tuning the hyper-parameter.
Revised: VCL is on par with or slightly better than SI. VCL has a superior long-term memory of previous tasks which leads to better overall performance on both metrics even though it does not have tuned hyper-parameters in its objective function.

Original: This again reveals a key advantage of theVCL approach: its objective function is hyper-parameter free. Very interestingly, all EWC methods suffer from performance deteriorations when moving from task “digit 0” to “digit 1”, probably because these two digits are very different.
Revised: For MNIST, the performance of LP and EWC deteriorate markedly when moving from task “digit 0” to “digit 1” possibly due to the large task differences. Also for all experimental settings we tried, SI fails to produce high test-LL results after task “digit 7”.

Original: Future work will investigate continual learning on a sequence of tasks that follows “adversarial ordering”, i.e. the ordering that makes the next task maximally different from the current task.
Revised: Future work will investigate continual learning on a sequence of tasks that follows “adversarial ordering”, i.e. the ordering that makes the next task maximally different from the current task.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Approximate Bayesian inference provides a natural framework for continual learning.
Revised: Approximate Bayesian inference provides a natural framework for continual learning.

Original: VCL, developed in this paper, is an approach in this vein that extends online variational inference to handle more general continual learning tasks and complex neural network models.
Revised: Variational Continual Learning (VCL), developed in this paper, is an approach in this vein that extends online variational inference to handle more general continual learning tasks and complex neural network models.

Original: VCL can be enhanced by including a small episodic memory that leverages coreset algorithms from statistics.
Revised: VCL can be enhanced by including a small episodic memory that leverages coreset algorithms from statistics and connects to message-scheduling in variational message passing.

Original: We demonstrated how the VCL framework can be applied to both discriminative and generative models.
Revised: We demonstrated how the VCL framework can be applied to both discriminative and generative models.

Original: Experimental results showed state-of-the-art performance when compared to previous continual learning approaches, even though VCL has no free parameters in its objective function.
Revised: Experimental results showed state-of-the-art performance when compared to previous continual learning approaches, even though VCL has no free parameters in its objective function.

Original: Future work should explore alternative approximate inference methods using the same framework and also develop more sophisticated episodic memories.
Revised: Future work should explore alternative approximate inference methods using the same framework and also develop more sophisticated episodic memories.

Original: Finally, we note that VCL is ideally suited for efﬁcient model reﬁnement in sequential decision making problems, such as RL and active learning.
Revised: Finally, we note that VCL is ideally suited for efﬁcient model reﬁnement in sequential decision making problems, such as reinforcement learning and active learning.

Original: [Figure]
Revised: The authors would like to thank Brian Trippe, Siddharth Swaroop, and Matej Balog for insightful comments and discussion. Cuong V. Nguyen is supported by EPSRC grant EP/M0269571. Yingzhen Li is supported by the Schlumberger FFTF Fellowship.

Original: R
Revised: Thang D. Bui is supported by the Google European Doctoral Fellowship. Richard E. Turner is supported by Google as well as EPSRC grants EP/M0269571 and EP/L000776/1. [Figure]

Original: A PPENDIX A E XPERIMENT S ETTINGS FORTHE P ERMUTED MNIST E XPERIMENT
Revised: A PPENDIX A F URTHER D ETAILS FOR P ERMUTED MNIST E XPERIMENT

Original: In this experiment we use fully connected single-head networks with two hidden layers where each layer contains 100 hidden units with ReLU activations.
Revised: In this experiment, we use fully connected single-head networks with two hidden layers, where each layer contains 100 hidden units with ReLU activations.

Original: The metric used for comparison is the test set accuracy on all observed tasks.
Revised: The metric used for comparison is the test set accuracy on all observed tasks.

Original: We train all the models using the Adam optimizer (Kingma &
Revised: We train all the models using the Adam optimizer (Kingma &

Original: Ba, 2015) with learning rate 10 − 3 since we found that it works best for all models.
Revised: Ba, 2015) with learning rate 10 − 3 since we found that it works best for all models.

Original: All the VCL algorithms are trained with batch size 256 and 100 epochs.
Revised: All the VCL algorithms are trained with batch size 256 and 100 epochs.

Original: For all the algorithms with coresets, we choose 200 examples from each task to include into the coresets.
Revised: For all the algorithms with coresets, we choose 200 examples from each task to include into the coresets.

Original: The algorithms that use only the coresets are trained using the VFE method with batch size equal to the coreset size and 100 epochs.
Revised: The algorithms that use only the coresets are trained using the VFE method with batch size equal to the coreset size and 100 epochs.

Revised: We use the prior N ( 0 , I ) and initialize our optimizer for the ﬁrst task at the mean of the maximum likelihood model and a very small initial variance ( 10 − 6 ).

Original: We compare the performance ofthe SI method that uses c = 0 .
Revised: We compare the performance of SI with hyper-parameters λ = 0 .

Original: 5 , 1 , 2 , where c is the regularization strength hyper-parameter (see ﬁg. 8).
Revised: 5 , 1 , 2 and select the best one ( λ = 0 . 5 ) as our baseline (see ﬁg. 8).

Original: Following the settings in (Zenke et al., 2017), we train these models with batch size 256 and 20 epochs.
Revised: Following Zenke et al. (2017), we train these models with batch size 256 and 20 epochs.

Original: The best value of c for this method is 0 . 5 , so we use this value as the baseline for our experiment. We also compare the performance ofthe EWC method that uses λ = 1 , 10 , 10 2 , 10 3 , 10 4 , where λ is the regularization strength hyper-parameter (see ﬁg. 9).
Revised: We also compare the performance of EWC with λ = 1 , 10 , 10 2 , 10 3 , 10 4 and select the best value λ = 10 2 as our baseline (see ﬁg. 9).

Original: The models are trained without drop-out and with batch size 200 and 20 epochs.
Revised: The models are trained without dropout and with batch size 200 and 20 epochs.

Original: We approximate the Fisher information matrix usingrandom samples drawn from the current dataset. 
Revised: We approximate the Fisher information matrices in EWC using 600 random samples drawn from the current dataset. For diagonal

Original: The best value of λ for this method is 102 , thus we use this value  asthe baseline for our experiment.
Revised: LP, we compare the performance of λ = 0 . 1 , 1 , 10 , 100 and use the best value λ = 0 . 1 as our baseline (see ﬁg. 10).

Revised: The models are also trained with prior N ( 0 , I ) , batch size 200, and 20 epochs.

Revised: The Hessians of LP are approximated using the Fisher information matrices with 200 samples.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: 10 # tasks 0 .0 .0 .0 .1 .

Revised: A cc u r a c y λ = 0 .λ = 0 .λ

Original:  B E XPERIMENT S ETTINGS FORTHE S PLIT MNIST E XPERIMENT
Revised: = λ =λ = B F URTHER D ETAILS FOR S PLIT MNIST E XPERIMENT

Original: In this experiment, we use fully connected multi-head networks with two hidden layers, each of which contains 256 hidden units with ReLU activations.
Revised: In this experiment, we use fully connected multi-head networks with two hidden layers, each of which contains 256 hidden units with ReLU activations.

Original: At each time step, we compare the test set accuracy of the current model on all observed tasks separately.
Revised: At each time step, we compare the test set accuracy of the current model on all observed tasks separately.

Original: We also plot the average accuracy over all tasks in the last column of ﬁg. 4.
Revised: We also plot the average accuracy over all tasks in the last column of ﬁg. 4.

Original: All the results for this experiment are the averages overruns of the algorithms with different random seeds.
Revised: All the results for this experiment are the averages overruns of the algorithms with different random seeds.

Original: Wealso use the Adam optimizer with learning rate 10 − 3 for all models in this experiment.
Revised: We use the Adam optimizer with learning rate 10 − 3 for all models.

Original: All the VCL algorithms are trained with batch size equal to the size of the training set and 120 epochs.
Revised: All the VCL algorithms are trained with batch size equal to the size of the training set and 120 epochs.

Revised: We use the prior N ( 0 , I ) and initialize our optimizer for the ﬁrst task at the mean of the maximum likelihood model and a very small initial variance ( 10 − 6 ).

Original: For the coresets, we choose 40 examples from each task to include into the coresets.
Revised: For the coresets, we choose 40 examples from each task to include into the coresets.

Original: In this experiment, the ﬁnal approximate posterior used for prediction in eq.
Revised: In this experiment, the ﬁnal approximate posterior used for prediction in eq.

Original: (3) is computed for each task separately using the coreset points corresponding to the task.
Revised: (3) is computed for each task separately using the coreset points corresponding to the task.

Original: The algorithms that use only the coresets are trained using the VFE method with batch size equal to the coreset size and 120 epochs.
Revised: The algorithms that use only the coresets are trained using the VFE method with batch size equal to the coreset size and 120 epochs.

Original: We compare the performance ofthe SImethod with c = 0 .
Revised: We compare the performance of SI with λ = 0 .

Original: and use the best value c = 1 asthe baseline for our experiment (see ﬁg. 10).
Revised: and use the best value λ = 1 as our baseline (see ﬁg. 11).

Original: We also comparethe EWCmethod with both single-head and multi-head models and λ = 1 , 10 , 10 2 , 10 3 , 10 4 (see ﬁg. 11).
Revised: We also compare EWC with both single-head and multi-head models and λ = 1 , 10 , 10 2 , 10 3 , 10 4 (see ﬁg. 12).

Revised: We approximate the Fisher information matrices usingrandom samples drawn from the current dataset.

Original: The ﬁgure shows that the multi-head models work better than the single-head models for EWC, and the performance is insensitive to the choice of λ .
Revised: The ﬁgure shows that the multi-head models work better than the single-head models for EWC, and the performance is insensitive to the choice of λ .

Original: Thus, we use the multi-head model with λ = 1 as the EWC baseline for our experiment
Revised: Thus, we use the multi-head model with λ = 1 as the EWC baseline for our experiment.

Original: [Figure]
Revised: For diagonal LP, we also use the multi-head model with λ = 1 , prior N ( 0 , I ) , and approximate the

Original: [Figure]
Revised: Hessians using the Fisher information matrices with 200 samples.

Original: C E XPERIMENT S ETTINGS FORTHE S PLIT NOT MNIST E XPERIMENT
Revised: C F URTHER D ETAILS FOR S PLIT NOT MNIST E XPERIMENT

Original: The settings for this experiment are the same as those in the Split MNIST experiment above, except that we use deeper networks with four hidden layers, each of which contains 150 hidden units.
Revised: The settings for this experiment are the same as those in the Split MNIST experiment above, except that we use deeper networks with 4 hidden layers, each of which contains 150 hidden units.

Original: Figures12 and 13  show the performance of SI and EWC with different hyper-parameter values respectively.
Revised: Figures 13 and 14 show the performance of SI and EWC with different hyper-parameter values respectively.

Original: In the experiment, we choose λ = 10 4 for multi-head EWC and c = 0 .
Revised: In the experiment, we choose λ = 10 4 for multi-head EWC, λ = 1 for multi-head LP, and λ = 0 .for SI. D A DDITIONAL E XPERIMENT ON A T OY 2D D ATASET Here we consider a small experiment on a toy 2D dataset to understand some of the properties of EWC with λ = 1 and VCL.

Revised: The experiment comprises two sequential binary classiﬁcation tasks.

Original: 1 for SI.
Revised: The ﬁrst task contains two classes generated from two Gaussian distributions. The data points (with green and black classes) for this task are shown in the ﬁrst column of ﬁg. 15. The second task contains two classes also generated from two Gaussian distributions. The green class for this task

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: 2 x     x0 .0 .
Revised: 2 x     x0 .0 .

Original: 5 0 .ewc_iter_0_first_task
Revised: 5 0 .ewc_iter_0_first_task

Original:    2 x     x0 .0 .

Revised:    2 x     x0 .0 .

Original: .ewc_iter_1_first_task
Revised: .ewc_iter_1_first_task

Original:    2 x     x0 .0 .0 .ewc_iter_1_current_task
Revised:    2 x     x0 .0 .0 .ewc_iter_1_current

Original: [Figure]
Revised: [Figure]

Original: 2 x     x0 .0 .0 .mean_field_vi_iter_0_first_task{+.mean_field_vi_iter_
+}
Revised: 2 x     x0 .0 .0 {+.mean_field_vi_iter_
+}

Revised: [Figure]

Original: 2 x     x0 .0 .0 .mean_field_vi_iter_1_first_task
Revised: [Figure]

Original: 2 x     x0 .0 .0 .{+.
 .mean+}
Revised: 2 x     x0 .0 {+.
 .mean+}

Original: [Figure]
Revised: [Figure]

Original: D A DDITIONAL E XPERIMENT ON A T OY 2D D ATASET
Revised: _vi_ [Figure] [Figure] [Figure]

Original: In this small experiment, we compare VCL and EWC on a toy 2D dataset.
Revised: 2 x     x0 . [Figure] [Figure] [Figure]

Original: The experiment has two sequential binary classiﬁcation tasks. The ﬁrst task contains two classes generated from two Gaussian distributions.
Revised: has the same input distribution as the ﬁrst task, while the input distribution for the black class is different.

Original: The data points(with green and black classes) for this tasks are shown in the ﬁrst column of ﬁg. 14.
Revised: The data points for this task are shown in the third columns of ﬁg. 15.

Original: The second task contains two classes also generated from two Gaussian
Revised: Each task contains 200 data points with 100 data points in each class.

Original: We compare the multi-head models trained by VCL and EWC on these two tasks.
Revised: We compare the multi-head models trained by VCL and EWC on these two tasks.

Original: In this experiment, we use fully connected networks with one hidden layers containing 20 hidden units with ReLU activations.
Revised: In this experiment, we use fully connected networks with one hidden layer containing 20 hidden units with ReLU activations.

Original: The ﬁrst column of ﬁg. 14 shows the contours of the prediction probabilities after observing the ﬁrst task, and both methods perform reasonably well for this task.
Revised: The ﬁrst column of ﬁg. 15 shows the contours of the prediction probabilities after observing the ﬁrst task, and both methods perform reasonably well for this task.

Original: However, after observing the second task, the EWC method fails to learn the classiﬁers for both tasks, while the
Revised: However, after observing the second task, the EWC method fails to learn the classiﬁers for both tasks, while the

Original: VCL method are still able to learn good classiﬁers for them.
Revised: VCL method are still able to learn good classiﬁers for them.

Original: E E XPERIMENT S ETTINGS AND M ORE S AMPLES FROM THE DGM S
Revised: E F URTHER D ETAILS ON D EEP G ENERATIVE M ODEL E XPERIMENTS

Original: Figure 15 shows additional images generated from the generators described in the main text.

Original: In this experiment, the learning rates and numbers of optimization epochs are tuned on separate training of each tasks.
Revised: In the experiments on Deep Generative Models, the learning rates and numbers of optimization epochs are tuned on separate training of each tasks.

Original: This gives a learning rate of 10 − 4 and the number of epochs 200 for MNIST  and 400 for notMNIST.
Revised: This gives a learning rate of 10 − 4 and the number of epochs 200 for MNIST (except for SI) and 400 for notMNIST.

Revised: For SI we optimize for 400 epochs on MNIST.

Original: For the VCL approach, the parameters of q t ( θ ) are initialized to have the same mean as q t − 1 ( θ ) and the log standard deviation  to 10 − 6 .
Revised: For the VCL approach, the parameters of q t ( θ ) are initialized to have the same mean as q t − 1 ( θ ) and the log standard deviation is set to 10 − 6 .

Original: The generative model consists of shared and task-speciﬁc components, each represented by a one hidden layer neural network with 500 hidden units (see ﬁg. 1(b)).
Revised: The generative model consists of shared and task-speciﬁc components, each represented by a one hidden layer neural network with 500 hidden units (see ﬁg. 1(b)).

Original: The dimensionality of the latent variable z and the intermediate representation h are 50 and 500, respectively.
Revised: The dimensionality of the latent variable z and the intermediate representation h are 50 and 500, respectively.

Original: We use task-speciﬁc encoders that are neural networks with symmetric architectures to the generator.
Revised: We use task-speciﬁc encoders that are neural networks with symmetric architectures to the generator.

Original: [Figure]

Original: F M EMORY OF A L INEAR R EGRESSION M ODEL T RAINED WITH VCL ON R ANDOM P ATTERNS
Revised: F M EMORY OF A L INEAR R EGRESSION M ODEL T RAINED WITH VCL ON R ANDOM P ATTERNS

Original: In many probabilistic models with conjugate priors, the exact posterior of the parameters/latent variables can be obtained.
Revised: In many probabilistic models with conjugate priors, the exact posterior of the parameters/latent variables can be obtained.

Original: For example, a Bayesian linear regression model with a Gaussian prior over the parameters and a Gaussian observation model has a Gaussian posterior.
Revised: For example, a Bayesian linear regression model with a Gaussian prior over the parameters and a Gaussian observation model has a Gaussian posterior.

Original: If we insist on using a diagonal Gaussian approximation to this posterior and use either the variational free energy method or the Laplace approximation, we will end up at the same solution – a Gaussian distribution with the same mean as that of the exact posterior and the diagonal precisions being the diagonal precisions of the exact posterior.
Revised: If we insist on using a diagonal Gaussian approximation to this posterior and use either the variational free energy method or the Laplace’s approximation, we will end up at the same solution – a Gaussian distribution with the same mean as that of the exact posterior and the diagonal precisions being the diagonal precisions of the exact posterior.

Original: Consequently, the online variational Gaussian approximation will give the same result to that given by the online Laplace approximation.
Revised: Consequently, the online variational Gaussian approximation will give the same result to that given by the online Laplace’s approximation.

Original: However, when a diagonal
Revised: However, when a diagonal

Original: Gaussian approximation is used, the batch and sequential solutions are different.
Revised: Gaussian approximation is used, the batch and sequential solutions are different.

Original: In the following, we will explicitly detail the sequential variational updates for a Bayesian linear regression model to associate random binary patterns to binary outcomes (Kirkpatrick et al., 2017), and show its relationship to the online Laplace approximation and the EWC approach of
Revised: In the following, we will explicitly detail the sequential variational updates for a Bayesian linear regression model to associate random binary patterns to binary outcomes (Kirkpatrick et al., 2017), and show its relationship to the online Laplace’s approximation and the EWC approach of

Original: Kirkpatrick et al.
Revised: Kirkpatrick et al.

Original: The task consists of associating a random D -dimensional binary vectors x t to a random binary output y t by learning a weight vector W .
Revised: The task consists of associating a random D -dimensional binary vectors x t to a random binary output y t by learning a weight vector W .

Original: Note that the possible values of the features and outputs are 1 and − 1 , and not 0 and 1.
Revised: Note that the possible values of the features and outputs are 1 and − 1 , and not 0 and 1.

Original: We also assume that the model sees only one input-output pair, { x t , y t } at the t -th time step and the previous approximate posterior q t − 1 ( W 
Revised: We also assume that the model sees only one input-output pair, { x t , y t } at the t -th time step and the previous approximate posterior q t − 1 ( W ) =

Original: [Equation]
Revised: [Equation]

Original: = [Equation] Dd =
Revised: N

Original: [Equation]
Revised: [Equation]

Original: N ( w

Original: [Equation]
Revised: [Equation]

Original: m t − 1 ,d , v t − 1 ,d ) , and that the observation noise variance σ 2 y is ﬁxed.
Revised: t − 1 ,d ) , and that the observation noise variance σ 2 y is ﬁxed.

Original: The variational updates for the mean and precisions are available in closed-form as follows:
Revised: The variational updates for the mean and precisions are available in closed-form as follows:

Original: [Equation]
Revised: [Equation]

Original: By further assuming that σ y = 1 and (cid:107) x t (cid:107) 2 = 1 , the equations above become:
Revised: By further assuming that σ y = 1 and (cid:107) x t (cid:107) 2 = 1 , the equations above become:

Original: [Equation]
Revised: [Equation]

Original: where ¯ x t = y t x t .
Revised: where ¯ x t = y t x t .

Original: When v − 1 0 ,d = 0, i.e. the prior is ignored, the update for the mean above is exactly equation S4 in the supplementary material of Kirkpatrick et al.
Revised: When v − 1 0 ,d = 0, i.e. the prior is ignored, the update for the mean above is exactly equation S4 in the supplementary material of Kirkpatrick et al.

Original: Therefore, in this case, the memory of the network trained by online variational inference is identical to that of the online
Revised: Therefore, in this case, the memory of the network trained by online variational inference is identical to that of the online

Original: Laplace method, provided in (Kirkpatrick et al., 2017).
Revised: Laplace’s method, provided in (Kirkpatrick et al., 2017).

Original: These methods differ, in practice, when the prior is not ignore or when the parameter regularization constraints are accumulated as discussed in the main text.
Revised: These methods differ, in practice, when the prior is not ignored or when the parameter regularization constraints are accumulated as discussed in the main text.

Original: This equivalence also does not hold in the general case, as discussed by Opper &
Revised: This equivalence also does not hold in the general case, as discussed by Opper &

Original: Archambeau (2009) that the Gaussian variational approximation is like an averaged and smoothed Laplace approximation.
Revised: Archambeau (2009) where the Gaussian variational approximation can be interpreted as an averaged and smoothed Laplace’s approximation.