Original: V ARIATIONAL C ONTINUAL L EARNING
Revised: V ARIATIONAL C ONTINUAL L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper develops variational continual learning (VCL), a simple but general framework for continual learning that fuses online variational inference (VI) and recent advances in Monte Carlo VI for neural networks.
Revised: This paper develops variational continual learning (VCL), a simple but general framework for continual learning that fuses online variational inference (VI) and recent advances in Monte Carlo VI for neural networks.

Original: The framework can successfully train both deep discriminative models and deep generative models in complex continual learning settings where existing tasks evolve over time and entirely new tasks emerge.
Revised: The framework can successfully train both deep discriminative models and deep generative models in complex continual learning settings where existing tasks evolve over time and entirely new tasks emerge.

Original: Experimental results show that variational continual learning outperforms state-of-the-art continual learning methods on a variety of tasks, avoiding catastrophic forgetting in a fully automatic way.
Revised: Experimental results show that variational continual learning outperforms state-of-the-art continual learning methods on a variety of tasks, avoiding catastrophic forgetting in a fully automatic way.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Continual learning (also called life-long learning and incremental learning) is a very general form of online learning in which data continuously arrive in a possibly non i.i.d.
Revised: Continual learning (also called life-long learning and incremental learning) is a very general form of online learning in which data continuously arrive in a possibly non i.i.d.

Original: way, tasks may change over time (e.g. new classes may be discovered), and entirely new tasks can emerge (Schlimmer & Fisher, 1986; Sutton & Whitehead, 1993; Ring, 1997).
Revised: way, tasks may change over time (e.g. new classes may be discovered), and entirely new tasks can emerge (Schlimmer & Fisher, 1986; Sutton & Whitehead, 1993; Ring, 1997).

Original: What is more, continual learning systems must adapt to perform well on the entire set of tasks in an online way that avoids revisiting all previous data at each stage.
Revised: What is more, continual learning systems must adapt to perform well on the entire set of tasks in an online way that avoids revisiting all previous data at each stage.

Original: This is a key problem in machine learning sincemost real world tasks continually evolve over time (e.g. they suffer from covariate and dataset shift) and the size of dataset often prohibits frequent batch updating.
Revised: This is a key problem in machine learning since real world tasks continually evolve over time (e.g. they suffer from covariate and dataset shift) and the size of dataset often prohibits frequent batch updating.

Original: Moreover, practitioners are often interested in solving a set of related tasks that beneﬁt from being handled jointly in order to leverage multi-task transfer.
Revised: Moreover, practitioners are often interested in solving a set of related tasks that beneﬁt from being handled jointly in order to leverage multi-task transfer.

Original: Continual learning is also of interest to cognitive science, being an intrinsic human ability.
Revised: Continual learning is also of interest to cognitive science, being an intrinsic human ability.

Original: The ubiquity of deep learning means that it is important to develop deep continual learning methods.
Revised: The ubiquity of deep learning means that it is important to develop deep continual learning methods.

Original: However, it is challenging to strike a balance between adapting to recent data and retaining knowledge from old data.
Revised: However, it is challenging to strike a balance between adapting to recent data and retaining knowledge from old data.

Original: Too much plasticity leads to the infamous catastrophic forgetting problem (McCloskey & Cohen, 1989; Ratcliff, 1990; Goodfellow et al., 2014a) and too much stability leads to an inability to adapt.
Revised: Too much plasticity leads to the infamous catastrophic forgetting problem (McCloskey & Cohen, 1989; Ratcliff, 1990; Goodfellow et al., 2014a) and too much stability leads to an inability to adapt.

Original: Recently there has been a resurgence of interest in this area.
Revised: Recently there has been a resurgence of interest in this area.

Original: One approach trains individual models on each task and then carries out a second stage of training to combine them (Lee et al., 2017).
Revised: One approach trains individual models on each task and then carries out a second stage of training to combine them (Lee et al., 2017).

Original: A more elegant and more ﬂexible approach maintains a single model and uses a single type of regularized training that prevents drastic changes in the parameters which have a large inﬂuence on prediction, but allows other parameters to change more freely (Li & Hoiem, 2016; Kirkpatrick et al., 2017; Zenke et al., 2017).
Revised: A more elegant and more ﬂexible approach maintains a single model and uses a single type of regularized training that prevents drastic changes in the parameters which have a large inﬂuence on prediction, but allows other parameters to change more freely (Li & Hoiem, 2016; Kirkpatrick et al., 2017; Zenke et al., 2017).

Original: The approach developed here follows this venerable work, but is arguably more principled, extensible and automatic.
Revised: The approach developed here follows this venerable work, but is arguably more principled, extensible and automatic.

Original: This paper is built on the observation that there already exists an extremely general, ﬂexible and theoretically grounded approach forhandling continual learning: Bayesian inference.
Revised: This paper is built on the observation that there already exists an extremely general framework for continual learning: Bayesian inference.

Original: Critically, Bayesian inference retains a distribution over model parameters that indicates the plausibility of any setting given the observed data.
Revised: Critically, Bayesian inference retains a distribution over model parameters that indicates the plausibility of any setting given the observed data.

Original: When new data arrive, weneed to combine what previous data have told us about the model parameters (the previous posterior over model parameters) with what the current data are telling us (the likelihood of the parameters).
Revised: When new data arrive, we combine what previous data have told us about the model parameters (the previous posterior) with what the current data are telling us (the likelihood).

Original: Multiplying and renormalizing yields the new posterior, from which point we can recurse.
Revised: Multiplying and renormalizing yields the new posterior, from which point we can recurse.

Original: Critically, the previous posterior constrainsthe parameters which strongly inﬂuence prediction, preventing them from changing drastically, butat the same time it allows unimportant parameters to change.
Revised: Critically, the previous posterior constrains parameters that strongly inﬂuence prediction, preventing them from changing drastically, but it allows other parameters to change.

Original: The wrinklein this approach is that, of course, exact Bayesian inferencein many models is  intractable and so approximations are required.
Revised: The wrinkle is that exact Bayesian inference is typically intractable and so approximations are required.

Original: Fortunately, there is a well established literature on approximate inference for neural networks.
Revised: Fortunately, there is an extensive literature on approximate inference for neural networks.

Original: We mergeadvances in online variational inference (VI) (Ghahramani & Attias, 2000; Sato, 2001; Broderick et al., 2013) with Monte Carlo variational inference for neural networks (Blundell et al., 2015) to yield variational continual learning (VCL).
Revised: We merge online variational inference (VI) (Ghahramani & Attias, 2000; Sato, 2001; Broderick et al., 2013) with Monte Carlo VI for neural networks (Blundell et al., 2015) to yield variational continual learning (VCL).

Original: In addition, wealso extend the method to include a selective episodic memory by combining variational inference with the coreset data summarization method (Bachem et al., 2015; Huggins et al., 2016).
Revised: In addition, we extend VCL to include a small episodic memory by combining VI with the coreset data summarization method (Bachem et al., 2015; Huggins et al., 2016).

Original: We demonstrate that the framework is general, applicable to both deep discriminative models and deep generative models, and that it yields excellent performance.
Revised: We demonstrate that the framework is general, applicable to both deep discriminative models and deep generative models, and that it yields excellent performance.

Original:  C ONTINUAL L EARNING BY A PPROXIMATE B AYESIAN I NFERENCE
Revised:  C ONTINUAL L EARNING BY A PPROXIMATE B AYESIAN I NFERENCE

Original: Consider a discriminative model that returns a probability distribution over an output y given an input x and parameters θ , that is p ( y | θ , x ) .
Revised: Consider a discriminative model that returns a probability distribution over an output y given an input x and parameters θ , that is p ( y | θ , x ) .

Original: Below we consider the speciﬁc case of a softmax distribution returned by a neural network with weight and bias parameters, but we keep the development general for now.
Revised: Below we consider the speciﬁc case of a softmax distribution returned by a neural network with weight and bias parameters, but we keep the development general for now.

Original: In the continual learning setting, the goal is to learn the parameters of the model from a set of sequentially arriving datasets { x ( n ) t , y ( n ) t } N t n =1 where, in principle, each might contain a single datum, N t = 1 .
Revised: In the continual learning setting, the goal is to learn the parameters of the model from a set of sequentially arriving datasets { x ( n ) t , y ( n ) t } N t n =1 where, in principle, each might contain a single datum, N t = 1 .

Original: Following a Bayesian approach, a prior distribution p ( θ ) is placed over θ .
Revised: Following a Bayesian approach, a prior distribution p ( θ ) is placed over θ .

Original: The posterior distribution after seeing T datasets is recovered by applying Bayes’ rule
Revised: The posterior distribution after seeing T datasets is recovered by applying Bayes’ rule

Original: [Equation]
Revised: [Equation]

Original: Here the input dependence has been suppressed on the right hand side to lighten notation.
Revised: Here the input dependence has been suppressed on the right hand side to lighten notation.

Original: We have used the shorthand D t = { y ( n ) t } N t n =1 .
Revised: We have used the shorthand D t = { y ( n ) t } N t n =1 .

Original: Importantly, a recursion has been identiﬁed whereby the posterior after seeing the T -th dataset is produced by taking the posterior after seeing the ( T − 1) -th dataset, multiplying by the likelihood and renormalizing.
Revised: Importantly, a recursion has been identiﬁed whereby the posterior after seeing the T -th dataset is produced by taking the posterior after seeing the ( T − 1) -th dataset, multiplying by the likelihood and renormalizing.

Original: In other words, online updating emerges naturally from Bayes’ rule.
Revised: In other words, online updating emerges naturally from Bayes’ rule.

Original: In most cases the posterior distribution is intractable and approximation is required, even when forming the ﬁrst posterior p ( θ |D 1 )
Revised: In most cases the posterior distribution is intractable and approximation is required, even when forming the ﬁrst posterior p ( θ |D 1 )

Original: ≈ q 1 ( θ ) =
Revised: ≈ q 1 ( θ ) =

Original: proj( p ( θ ) p ( D 1 | θ )) .
Revised: proj( p ( θ ) p ( D 1 | θ )) .

Original: Here q ( θ ) =
Revised: Here q ( θ ) =

Original: proj( p ∗ ( θ )) denotes a projection operation that takes the intractable un-normalized distribution p ∗ ( θ ) and returns a tractable normalized approximation q ( θ ) .
Revised: proj( p ∗ ( θ )) denotes a projection operation that takes the intractable un-normalized distribution p ∗ ( θ ) and returns a tractable normalized approximation q ( θ ) .

Original: The ﬁeld of approximate inference provides several choices for the projection operation including i) Laplace approximation, ii) variational KL minimization, iii) moment matching, and iv) importance sampling.
Revised: The ﬁeld of approximate inference provides several choices for the projection operation including i) Laplace approximation, ii) variational KL minimization, iii) moment matching, and iv) importance sampling.

Original: Having approximated the ﬁrst posterior distribution, subsequent approximations can be produced recursively by combining the approximate posterior distribution with the likelihood and projecting, that is p ( θ |D 1: T )
Revised: Having approximated the ﬁrst posterior distribution, subsequent approximations can be produced recursively by combining the approximate posterior distribution with the likelihood and projecting, that is p ( θ |D 1: T )

Original: ≈ q T ( θ ) = proj( q T − 1 ( θ ) p ( D T | θ )) .
Revised: ≈ q T ( θ ) = proj( q T − 1 ( θ ) p ( D T | θ )) .

Original: In this way online updating is supported.
Revised: In this way online updating is supported.

Original: This general approach leads, for the four projection operators previously identiﬁed, to i) Laplace propagation (Smola et al., 2004), ii) online VI (Ghahramani & Attias, 2000; Sato, 2001) also known as streaming VB (Broderick et al., 2013), iii) assumed density ﬁltering (Maybeck, 1982) and iv) sequential Monte Carlo (Liu & Chen, 1998).
Revised: This general approach leads, for the four projection operators previously identiﬁed, to i) Laplace propagation (Smola et al., 2004), ii) online VI (Ghahramani & Attias, 2000; Sato, 2001) also known as streaming VB (Broderick et al., 2013), iii) assumed density ﬁltering (Maybeck, 1982) and iv) sequential Monte Carlo (Liu & Chen, 1998).

Original: In this paper the online VI approach is used as it typically outperforms the other methods for complex models in the static setting (Bui et al., 2016) and yet it has not been applied to continual learning of neural networks.
Revised: In this paper the online VI approach is used as it typically outperforms the other methods for complex models in the static setting (Bui et al., 2016) and yet it has not been applied to continual learning of neural networks.

Original: 2. V ARIATIONAL C ONTINUAL L EARNING (VCL) AND E PISODIC M EMORY E NHANCEMENT
Revised: 2. V ARIATIONAL C ONTINUAL L EARNING (VCL) AND E PISODIC M EMORY E NHANCEMENT

Original: Variational continual learning employs a projection operator deﬁned through a KL divergence minimization over the set of allowed approximate posteriors Q ,
Revised: Variational continual learning employs a projection operator deﬁned through a KL divergence minimization over the set of allowed approximate posteriors Q ,

Original: [Equation]
Revised: [Equation]

Original: The zeroth approximate distribution is deﬁned to be the prior, q 0 ( 
Revised: The zeroth approximate distribution is deﬁned to be the prior, q 0 ( θ ) =

Original: [Equation]
Revised: p ( θ ) .

Original: = [Equation] Z t is the intractable normalizing constant of p ∗ ( θ ) =
Revised: Z t is the intractable normalizing constant of p ∗ ( θ ) =

Original: q t − 1 ( θ ) p
Revised: q t − 1 ( θ ) p

Original: ( D t | θ ) and is not required to compute the optimum.
Revised: ( D t | θ ) and is not required to compute the optimum.

Original: VCL will perform exact Bayesian inference if the true posterior is a member of the approximating family p ( θ |D 1 , D 2 , . . .
Revised: VCL will perform exact Bayesian inference if the true posterior is a member of the approximating family p ( θ |D 1 , D 2 , . . .

Original: , D t ) ∈ Q at every step t .
Revised: , D t ) ∈ Q at every step t .

Original: Typically this will not be the case and we might worry that performing repeated approximations may accumulate errors causing the algorithm to forget old tasks.
Revised: Typically this will not be the case and we might worry that performing repeated approximations may accumulate errors causing the algorithm to forget old tasks.

Original: Furthermore, the minimization at each step is also approximate (e.g. due to employing an additional Monte Carlo approximation) and so additional information may be lost.
Revised: Furthermore, the minimization at each step is also approximate (e.g. due to employing an additional Monte Carlo approximation) and so additional information may be lost.

Original: In order to mitigate this potential problem, we extend VCL to include a small representative set of data from previously observed tasks that we call the coreset.
Revised: In order to mitigate this potential problem, we extend VCL to include a small representative set of data from previously observed tasks that we call the coreset.

Original: The coreset is analogous to an episodic memory that retains key information (in our case, important training data points) from previous tasks which the algorithm can revisit in order to refresh its memory of them.Coresets have previously been [Table] [Table]
Revised: The coreset is analogous to an episodic memory that retains key information (in our case, important training data points) from previous tasks which the algorithm can revisit in order to refresh its memory of them.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table] [Table] [Table] explored fordata summarization in the batch learning setting for Bayesian models (Bachem et al., 2015; Huggins et al., 2016), but have not been considered in the continual learning setting.
Revised: This idea of using an episodic memory has been explored for continual learning by Lopez-Paz & Ranzato (2017).

Original: Algorithm 1 describes coreset VCL.
Revised: Algorithm 1 describes coreset VCL.

Original: The algorithm maintains a pair ( C t , ˜ q t ) comprising the coreset and the variational distribution for non-coreset data at each time step t .
Revised: The algorithm maintains a pair ( C t , ˜ q t ) comprising the coreset and the variational distribution for non-coreset data at each time step t .

Original: The distribution ˜ q t is continuously updated using eq.
Revised: The distribution ˜ q t is continuously updated using eq.

Original: When performing prediction, ˜ q t and C t are combined to form the ﬁnal variational distribution q t using eq.
Revised: When performing prediction, ˜ q t and C t are combined to form the ﬁnal variational distribution q t using eq.

Original: This step ensures recent exposure to data from each task and helps mitigate any  forgetting.
Revised: This step ensures recent exposure to data from each task and helps mitigate any residual forgetting.

Original: From a more general perspective, coreset VCL is equivalent to a message-passing implementation of VI in which the coreset data point updates are scheduled after updating the other data.

Original: The coreset VCL algorithm adds new data points to the coreset after observing each new dataset D t (it is simple to remove them too if T is large).
Revised: The coreset VCL algorithm adds new data points to the coreset after observing each new dataset D t (it is simple to remove them too if T is large).

Original: Any method can be used to update the coreset.
Revised: Any method can be used to update the coreset.

Original: For example, K data points can be selected at random from D t and added to C t − 1 to form an unbiased new coreset C t .
Revised: For example, K data points can be selected at random from D t and added to C t − 1 to form an unbiased new coreset C t .

Original: Alternatively, the greedy K -center algorithm (Gonzalez, 1985) can be used to return K data points that are guaranteed to be spread throughout the input space.
Revised: Alternatively, the greedy K -center algorithm (Gonzalez, 1985) can be used to return K data points that are guaranteed to be spread throughout the input space.

Original:  V ARIATIONAL C ONTINUAL L EARNING IN D EEP D ISCRIMINATIVE M ODELS
Revised:  V ARIATIONAL C ONTINUAL L EARNING IN D EEP D ISCRIMINATIVE M ODELS

Original: The VCL framework is general and can be applied to many discriminative probabilistic models.
Revised: The VCL framework is general and can be applied to many discriminative probabilistic models.

Original: Here we apply it to continual learning of a deep fully-connected neural network classiﬁers.
Revised: Here we apply it to continual learning of a deep fully-connected neural network classiﬁers.

Original: Before turning to application of VCL, we ﬁrst consider the architecture of neural networks suitable for performing continual learning.
Revised: Before turning to application of VCL, we ﬁrst consider the architecture of neural networks suitable for performing continual learning.

Original: In simple instances of discriminative continual learning, where data are arriving in an i.i.d.
Revised: In simple instances of discriminative continual learning, where data are arriving in an i.i.d.

Original: way or where only the input distribution p ( x 1: T ) changes over time, a standard  discriminative neural network sufﬁces.
Revised: way or where only the input distribution p ( x 1: T ) changes over time, a standard single-head discriminative neural network sufﬁces.

Original: The permuted MNIST experiment considered later, where each task involves classifying MNIST digits under a new permutation (Goodfellow et al., 2014a;

Original: Kirkpatrick et al., 2017), falls into this category.

Original: We refer to these as single-head networks.

Original: In many cases, the tasks, although related, might involve different output variables.
Revised: In many cases, the tasks, although related, might involve different output variables.

Original: Standard practice in multi-task learning then uses networks with a shared lower part (closer to the inputs) and separate top parts for each task (closer to the outputs). This type of architecture is used for the split MNIST experiment (Zenke et al., 2017) which involves a sequence of binary MNIST digit classiﬁcation tasks. We refer to these as multi-head networks.
Revised: Standard practice in multi-task learning (Bakker & Heskes, 2003) uses networks that share parameters close to the inputs but with separate heads for each output, hence multi-head networks.

Original: Graphical models depicting the network architecture for deep discriminative and deep generative models are shown in ﬁg. 1.
Revised: Graphical models depicting the network architecture for deep discriminative and deep generative models are shown in ﬁg. 1.

Original: A general solution to continual learning would perform automatic model building that adds new structure to the existing model as new tasks are encountered (Rusu et al., 2016).
Revised: A general solution to continual learning would perform automatic model building that adds new structure to the existing model as new tasks are encountered (Rusu et al., 2016).

Original: Although this is an interesting research direction, here we make the simplifying assumption that the model structure is known a priori .
Revised: Although this is an interesting research direction, here we make the simplifying assumption that the model structure is known a priori .

Original: VCL requires speciﬁcation of q ( θ ) where θ in the current case is a D dimensional vector formed by stacking the network’s biases and weights.
Revised: VCL requires speciﬁcation of q ( θ ) where θ in the current case is a D dimensional vector formed by stacking the network’s biases and weights.

Original: For simplicity we use a Gaussian mean-ﬁeld approximate posterior q t ( θ ) =
Revised: For simplicity we use a Gaussian mean-ﬁeld approximate posterior q t ( θ ) =

Original: (cid:81)
Revised: (cid:81)

Original: Dd =1 N ( θ t,d ; µ t,d , σ 2 t,d ) .
Revised: Dd =1 N ( θ t,d ; µ t,d , σ 2 t,d ) .

Original: Taking the most general case of a multi-head network, before task k is encountered the posterior distribution over the associated head parameters will remain at the prior and so q ( θ H k ) =
Revised: Taking the most general case of a multi-head network, before task k is encountered the posterior distribution over the associated head parameters will remain at the prior and so q ( θ Hk ) =

Original: p ( θ H k ) .
Revised: p ( θ Hk ) .

Original: This is convenient as it means the variational[Figure] approximation can be grown incrementally, starting from the prior, as each task emerges.
Revised: This is convenient as it means the variational approximation can be grown incrementally, starting from the prior, as each task emerges.

Original: Moreover, only tasks present in the current dataset D t need to have their posterior distributions over head parameters updated.
Revised: Moreover, only tasks present in the current dataset D t need to have their posterior distributions over head parameters updated.

Original: The shared parameters, on the other hand, will be constantly updated.
Revised: The shared parameters, on the other hand, will be constantly updated.

Original: Training the network using the VFE approach in eq.
Revised: Training the network using the VFE approach in eq.

Original: (1) is equivalent to maximizing the negative online variational free energy or the variational lower bound to the online marginal likelihood:[Equation]
Revised: (1) is equivalent to maximizing the negative online variational free energy or the variational lower bound to the online marginal likelihood:

Original: [Equation]
Revised: [Equation]

Original: ) with respect to the variational parameters { µ t,d , σ 2 t,d } Dd =1 .
Revised: with respect to the variational parameters { µ t,d , σ 2 t,d } Dd =1 .

Original: Whilst the KL-divergence
Revised: Whilst the KL-divergence

Original: KL( q t ( θ ) ||
Revised: KL( q t ( θ ) ||

Original: q t − 1 ( θ )) can be computed in closed-form, the expected log likelihood requires further approximation.
Revised: q t − 1 ( θ )) can be computed in closed-form, the expected log likelihood requires further approximation.

Original: Here we take the usual approach of employing simple Monte Carlo and use the  reparameterization trick to compute the gradients (Salimans & Knowles, 2013; Kingma & Welling, 2014).
Revised: Here we take the usual approach of employing simple Monte Carlo and use the [Figure] reparameterization trick to compute the gradients (Salimans & Knowles, 2013; Kingma & Welling, 2014).

Original: At the ﬁrst time step, the prior distribution, and therefore q 0 ( θ ) is chosen to be a multivariate
Revised: At the ﬁrst time step, the prior distribution, and therefore q 0 ( θ ) is chosen to be a multivariate

Original: Gaussian distribution (see e.g. Graves, 2011; Blundell et al., 2015).
Revised: Gaussian distribution (see e.g. Graves, 2011; Blundell et al., 2015).

Original:  V ARIATIONAL C ONTINUAL L EARNING IN D EEP G ENERATIVE M ODELS
Revised:  V ARIATIONAL C ONTINUAL L EARNING IN D EEP G ENERATIVE M ODELS

Original: Deep generative models (DGMs) have garnered much recent attention.
Revised: Deep generative models (DGMs) have garnered much recent attention.

Original: By passing a simple noise variable (e.g. Gaussian noise) through a deep neural network, these models have been shown to be able to generate realistic images, sounds and videos sequences (Chung et al., 2015; Kingma et al., 2016; Vondrick et al., 2016).
Revised: By passing a simple noise variable (e.g. Gaussian noise) through a deep neural network, these models have been shown to be able to generate realistic images, sounds and videos sequences (Chung et al., 2015; Kingma et al., 2016; Vondrick et al., 2016).

Original: Standard approaches for learning DGMs have focused on the batch learning, i.e. the observed instances are assumed to be i.i.d.
Revised: Standard approaches for learning DGMs have focused on the batch learning, i.e. the observed instances are assumed to be i.i.d.

Original: and are all available at the same time.
Revised: and are all available at the same time.

Original: In this section we extend the VCL framework to encompass variational auto-encoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014), a form of DGMs.
Revised: In this section we extend the VCL framework to encompass variational auto-encoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014), a form of DGMs.

Original: The approach could be extended to generative adversarial networks (GAN) (Goodfellow et al., 2014b) for which continual learning is an open problem, but see Seff et al.
Revised: The approach could be extended to generative adversarial networks (GAN) (Goodfellow et al., 2014b) for which continual learning is an open problem, but see Seff et al.

Original: for an initial attempt.
Revised: for an initial attempt.

Original: Consider a model p ( x | z , θ ) p ( z ) , for observed data x and latent variables z .
Revised: Consider a model p ( x | z , θ ) p ( z ) , for observed data x and latent variables z .

Original: The prior over latent variables p ( z ) is typically Gaussian, and the distributional parameters of p ( x | z , θ ) are deﬁned by a deep neural network.
Revised: The prior over latent variables p ( z ) is typically Gaussian, and the distributional parameters of p ( x | z , θ ) are deﬁned by a deep neural network.

Original: For example, if Bernoulli likelihood is used, then p ( x | z , θ ) =
Revised: For example, if Bernoulli likelihood is used, then p ( x | z , θ ) =

Original: Bern( f θ ( z )) , where f θ denotes the deep neural network transform and θ collects all the weight matrices and bias vectors.
Revised: Bern( f θ ( z )) , where f θ denotes the deep neural network transform and θ collects all the weight matrices and bias vectors.

Original: In the batch setting, given a dataset D = { x n } Nn =1 , the standard VAE approach learns the parameters θ by approximate maximum likelihood estimation (MLE).
Revised: In the batch setting, given a dataset D = { x n } Nn =1 , the standard VAE approach learns the parameters θ by approximate maximum likelihood estimation (MLE).

Original: This proceeds by optimizing variational lower bound with respect to θ and φ :
Revised: This proceeds by optimizing variational lower bound with respect to θ and φ :

Original: [Equation]
Revised: [Equation]

Original: where φ denotes the variational parameters for the “encoder” or the approximate posterior of z given an observation x n .
Revised: where φ denotes the variational parameters for the “encoder” or the approximate posterior of z given an observation x n .

Original: The approximate MLE approach is unsuitable for the continual learning setting as it does not return parameter uncertainty estimates that are critical for weighting the information learned from old data.
Revised: The approximate MLE approach is unsuitable for the continual learning setting as it does not return parameter uncertainty estimates that are critical for weighting the information learned from old data.

Original: So, instead the VCL approach will approximate the full posterior distribution over parameters, q t ( θ )
Revised: So, instead the VCL approach will approximate the full posterior distribution over parameters, q t ( θ )

Original: ≈ p t ( θ |D 1: t ) , after observing the t -th dataset.
Revised: ≈ p t ( θ |D 1: t ) , after observing the t -th dataset.

Original: Speciﬁcally, the approximate posterior q t is obtained by maximizing the full variational lower bound with respect to q t and φ : 
Revised: Speciﬁcally, the approximate posterior q t is obtained by maximizing the full variational lower bound with respect to q t and φ : [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the encoder network q φ ( z | x ( n ) t ) is parameterized by φ which is task-speciﬁc.
Revised: where the encoder network q φ ( z | x ( n ) t ) is parameterized by φ which is task-speciﬁc.

Original: It is likely to be beneﬁcial to share (parts of) these encoder networks, but this is not investigated in this paper.
Revised: It is likely to be beneﬁcial to share (parts of) these encoder networks, but this is not investigated in this paper.

Original: As was the case for multi-head discriminative models, we can split the generative model into shared and task-speciﬁc parts.
Revised: As was the case for multi-head discriminative models, we can split the generative model into shared and task-speciﬁc parts.

Original: There are two options: (i) the generative models share across tasks the network that generates observations x from the intermediate-level representations h , but have their own “head network” for generating h from the latent variables z (see ﬁg. 1(b)), and (ii) the other way around.
Revised: There are two options: (i) the generative models share across tasks the network that generates observations x from the intermediate-level representations h , but have their own “head network” for generating h from the latent variables z (see ﬁg. 1(b)), and (ii) the other way around.

Original: In the latter architecture, if the task-speciﬁc lower-level network is reasonably deep, the information of the current task can be encoded efﬁciently in this part of the network, thus effectively ignoring the shared high-level network.
Revised: In the latter architecture, if the task-speciﬁc lower-level network is reasonably deep, the information of the current task can be encoded efﬁciently in this part of the network, thus effectively ignoring the shared high-level network.

Original: Due to this reason, we focus on the former architecture in the experiments where the network is forced to share information between tasks.
Revised: Due to this reason, we focus on the former architecture in the experiments where the network is forced to share information between tasks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Continual Learning for Deep Discriminative Models:
Revised: Continual Learning for Deep Discriminative Models:

Original: Many neural network continual learning approaches employ regularized maximum likelihood estimation, optimizing objectives of the form:
Revised: Many neural network continual learning approaches employ regularized maximum likelihood estimation, optimizing objectives of the form:

Original: [Equation]
Revised: [Equation]

Original: Here the regularization biases the new parameter estimates towards those estimated at the previous step θ t − 1 .
Revised: Here the regularization biases the new parameter estimates towards those estimated at the previous step θ t − 1 .

Original: λ t is a user-selected hyper-parameter that controls the overall contribution from previous data and Σ t − 1 is a matrix (normally diagonal in form) that encodes the relative strength of the regularization on each element of θ .
Revised: λ t is a user-selected hyper-parameter that controls the overall contribution from previous data and Σ t − 1 is a matrix (normally diagonal in form) that encodes the relative strength of the regularization on each element of θ .

Original: We now discuss speciﬁc instances of this scheme:
Revised: We now discuss speciﬁc instances of this scheme:

Original: • Maximum-likelihood estimation and MAP estimation : maximum likelihood estimation is recovered when there is no regularization ( λ t = 0 ).
Revised: • Maximum-likelihood estimation and MAP estimation : maximum likelihood estimation is recovered when there is no regularization ( λ t = 0 ).

Original: More generally, the regularization term can be interpreted as a Gaussian prior, q ( θ |D 1: t − 1 ) =
Revised: More generally, the regularization term can be interpreted as a Gaussian prior, q ( θ |D 1: t − 1 ) =

Original: N ( θ ; θ t − 1 , Σ t − 1 /λ t ) .
Revised: N ( θ ; θ t − 1 , Σ t − 1 /λ t ) .

Original: The optimization returns the maximum a posteriori estimate of the parameters, but this does not directly provide a Σ t for the next stage.
Revised: The optimization returns the maximum a posteriori estimate of the parameters, but this does not directly provide a Σ t for the next stage.

Original: A simple ﬁx is to set Σ t
Revised: A simple ﬁx is to set Σ t

Original: =
Revised: =

Original: I and use cross validation to ﬁnd λ t , but this approximation is often coarse and leads to catastrophic forgetting (Goodfellow et al., 2014a; Kirkpatrick et al., 2017).
Revised: I and use cross-validation to ﬁnd λ t , but this approximation is often coarse and leads to catastrophic forgetting (Goodfellow et al., 2014a; Kirkpatrick et al., 2017).

Original: • Laplace Propagation
Revised: • Laplace Propagation

Original: (Smola et al., 2004): applying Laplace approximation at each step leads to a recursion for Σ − 1 t , which is initialized using the covariance of the Gaussian prior, Σ
Revised: (Smola et al., 2004): applying Laplace approximation at each step leads to a recursion for Σ − 1 t , which is initialized using the covariance of the Gaussian prior, Σ

Original: [Equation]
Revised: [Equation]

Original:  t
Revised: 1 t

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: To avoid computing the full Hessian of the likelihood, diagonal-Laplace propagation retains only the diagonal terms of Σ − 1 t .
Revised: To avoid computing the full Hessian of the likelihood, diagonal-Laplace propagation retains only the diagonal terms of Σ − 1 t .

Original: • Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) builds on diagonal-Laplace propagation by approximating the average Hessian of the likelihoods using well-known identities for the Fisher information:
Revised: • Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) builds on diagonal-Laplace propagation by approximating the average Hessian of the likelihoods using well-known identities for the Fisher information:

Original: [Equation]
Revised: [Equation]

Original: EWC also modiﬁes the Laplace regularization, 12 ( 
Revised: EWC also modiﬁes the Laplace regularization, 12 ( θ − θ

Revised: t − 1 ) (cid:124)

Revised: (Σ

Original: [Equation]
Revised: [Equation]

Original:  − 
Revised: + (cid:80) t − 1 t

Original: [Equation]
Revised: [Equation]

Revised: =1 Φ t (cid:48) )

Original:  t [Equation]
Revised: θ − θ t − 1 )

Original: [Equation]
Revised: , introducing hyper-parameters, removing the prior and regularizing to intermediate parameter estimators, rather than just that derived from the last task, 12 (cid:80) t − 1 t (cid:48)

Original: θ −
Revised: =1 λ

Original: [Equation]
Revised: [Equation]

Original: −
Revised: ( θ

Original: [Equation]
Revised: [Equation]

Original: ) , introducing hyper-parameters, removing the prior and regularizing to intermediate parameter estimators
Revised: θ

Original: [Table]
Revised: [Equation]

Original:  θ t (cid:48) − 1 ) .
Revised: 1 ) (cid:124) Φ t (cid:48) ( θ − θ t (cid:48) − 1 ) .

Original: This ﬁnal change is likely to be unnecessary.• Synaptic Intelligence (SI) (Zenke et al., 2017): SI computes Σ − 1 t using a measure of the importance of each parameter to each task.
Revised: This ﬁnal change is likely to be unnecessary.• Synaptic Intelligence (SI) (Zenke et al., 2017): SI computes Σ − 1 t using a measure of the importance of each parameter to each task.

Original: Practically, this is achieved by comparing the changing rate of the gradients of the objective and the changing rate of the parameters.
Revised: Practically, this is achieved by comparing the changing rate of the gradients of the objective and the changing rate of the parameters.

Original: VCL differs from the above methods in several ways.
Revised: VCL differs from the above methods in several ways.

Original: First, unlike MAP, EWC and SI approaches above, it does not have free parameters that need to be tuned on a validation set.
Revised: First, unlike MAP, EWC and SI approaches above, it does not have free parameters that need to be tuned on a validation set.

Original: This can be especially awkward in the online setting.
Revised: This can be especially awkward in the online setting.

Original: Second, although the KL regularization penalizes the mean of the approximate posterior through a quadratic cost, a full distribution is retained and needs to be averaged over at training time and at test time.
Revised: Second, although the KL regularization penalizes the mean of the approximate posterior through a quadratic cost, a full distribution is retained and needs to be averaged over at training time and at test time.

Original: Third, and more generally, VI is generally thought to return better uncertainty estimates than approaches like Laplace method and MAP estimation, and we have argued this is critical for continual learning.
Revised: Third, and more generally, VI is generally thought to return better uncertainty estimates than approaches like Laplace method and MAP estimation, and we have argued this is critical for continual learning.

Original: There is a long history of research on approximate Bayesian approaches to training neural networks, including extended Kalman ﬁltering (Singhal & Wu, 1989), Laplace approximation (MacKay, 1992), variational inference (Hinton & Van Camp, 1993; Barber & Bishop, 1998; Graves, 2011;
Revised: There is a long history of research on approximate Bayesian approaches to training neural networks, including extended Kalman ﬁltering (Singhal & Wu, 1989), Laplace approximation (MacKay, 1992), variational inference (Hinton & Van Camp, 1993; Barber & Bishop, 1998; Graves, 2011;

Original: Blundell et al., 2015; Gal & Ghahramani, 2016), sequential Monte Carlo (de Freitas et al., 2000), EP (Hern ´ andez-Lobato & Adams, 2015), and approximate Power EP (Hern ´ andez-Lobato et al., 2016).
Revised: Blundell et al., 2015; Gal & Ghahramani, 2016), sequential Monte Carlo (de Freitas et al., 2000), EP (Hern´andez-Lobato & Adams, 2015), and approximate Power EP (Hern´andez-Lobato et al., 2016).

Original: These approaches have been focused on batch learning, but the framework described in sectionalso enables them to be applied to continual learning.
Revised: These approaches have been focused on batch learning, but the framework described in sectionalso enables them to be applied to continual learning.

Original: On the other hand, the online variational inference has been previously explored (Ghahramani & Attias, 2000; Broderick et al., 2013; Bui et al., 2017), but not for neural networks or in the context of complex tasks.
Revised: On the other hand, the online variational inference has been previously explored (Ghahramani & Attias, 2000; Broderick et al., 2013; Bui et al., 2017), but not for neural networks or in the context of complex tasks.

Original: Continual Learning for Deep Generative Models: A na ¨ ıve continual learning approach for deep generative models would directly apply the VAE algorithm to the new dataset D t with the model parameters initialized at the previous parameter values θ t − 1 .
Revised: Continual Learning for Deep Generative Models: A na¨ıve continual learning approach for deep generative models would directly apply the VAE algorithm to the new dataset D t with the model parameters initialized at the previous parameter values θ t − 1 .

Original: Our experiments show that this approach leads to catastrophic forgetting, in the sense that the generator can only generate instances that are similar to the data points from the most recently observed dataset.
Revised: Our experiments show that this approach leads to catastrophic forgetting, in the sense that the generator can only generate instances that are similar to the data points from the most recently observed dataset.

Original: Alternatively, EWC regularization can be added to the VAE objective:
Revised: Alternatively, EWC regularization can be added to the VAE objective:

Original: [Equation]
Revised: [Equation]

Original: However computing Φ t requires the gradient ∇ θ log p ( x | θ ) which is intractable.
Revised: However computing Φ t requires the gradient ∇ θ log p ( x | θ ) which is intractable.

Original: Instead, we can approximate the marginal likelihood by the variational lower bound, i.e.
Revised: Instead, we can approximate the marginal likelihood by the variational lower bound, i.e.

Original: [Equation]
Revised: [Equation]

Original: An importance sampling estimate could also be used using Burda et al.
Revised: An importance sampling estimate could also be used using Burda et al.

Original: In our experiments with DGMs, this version of EWC is used as a baseline to compare with VCL.
Revised: In our experiments with DGMs, this version of EWC is used as a baseline to compare with VCL.

Original:  E XPERIMENTS 6. E XPERIMENTS WITH D EEP D ISCRIMINATIVE M ODELS
Revised:  E XPERIMENTS 6. E XPERIMENTS WITH D EEP D ISCRIMINATIVE M ODELS

Original: We consider two continual learning experiments for deep discriminative models: the permuted
Revised: We consider two continual learning experiments for deep discriminative models: the permuted

Original: MNIST experiment and the split MNIST experiment.
Revised: MNIST experiment and the split MNIST experiment.

Original: More details of the experiment settings and an additional experiment are available in the appendix.
Revised: More details of the experiment settings and an additional experiment are available in the appendix.

Original: Permuted MNIST:
Revised: Permuted MNIST:

Original: This is a popular continual learning benchmark (Goodfellow et al., 2014a; Kirkpatrick et al., 2017; Zenke et al., 2017).
Revised: This is a popular continual learning benchmark (Goodfellow et al., 2014a; Kirkpatrick et al., 2017; Zenke et al., 2017).

Original: The dataset received at each time step D t consists of labeled
Revised: The dataset received at each time step D t consists of labeled

Original: MNIST images whose pixels have undergone a ﬁxed random permutation.
Revised: MNIST images whose pixels have undergone a ﬁxed random permutation.

Original: We compare VCL to
Revised: We compare VCL to

Original: EWC and SI.
Revised: EWC and SI.

Original: For all algorithms, we use fully connected single-head networks with two hidden layers, where each layer contains 100 hidden units with ReLU activations.
Revised: For all algorithms, we use fully connected single-head networks with two hidden layers, where each layer contains 100 hidden units with ReLU activations.

Original: We evaluate three versions of VCL: VCL with no coreset, VCL with a random coreset, and VCL with a coreset selected by the
Revised: We evaluate three versions of VCL: VCL with no coreset, VCL with a random coreset, and VCL with a coreset selected by the

Original: K-center method.
Revised: K-center method.

Original: For the coresets, we select 200 data points from each task.
Revised: For the coresets, we select 200 data points from each task.

Original: [Figure] [Figure] Figure 2 compares the average test set accuracy on all observed tasks.
Revised: Figure 2 compares the average test set accuracy on all observed tasks.

Original: From this ﬁgure, VCL outperforms the EWC and SI by large margins, even though they beneﬁted from an extensive hyperparameter search for λ .
Revised: From this ﬁgure, VCL outperforms the EWC and SI by large margins, even though they beneﬁted from an extensive hyperparameter search for λ .

Original: After 10 tasks, VCL achieves 90% average accuracy, while EWC and SI only achieve 63% and 86% respectively.
Revised: After 10 tasks, VCL achieves 90% average accuracy, while EWC and SI only achieve 63% and 86% respectively.

Original: The results also show that using the coresets alone is not useful for this task, but combining them with VCL leads to a modest improvement over VCL: both random coresets and K-center coresets achieve 93% accuracy.
Revised: The results also show that using the coresets alone is not useful for this task, but combining them with VCL leads to a modest improvement over VCL: both random coresets and K-center coresets achieve 93% accuracy.

Original: Split MNIST:
Revised: Split MNIST:

Original: This experiment was used by Zenke et al.
Revised: This experiment was used by Zenke et al.

Original: (2017) to assess the SI method.
Revised: (2017) to assess the SI method.

Original: Five binary classiﬁcation tasks from the MNIST dataset arrive in sequence: 0/1, 2/3, 4/5, 6/7, and 8/9.
Revised: Five binary classiﬁcation tasks from the MNIST dataset arrive in sequence: 0/1, 2/3, 4/5, 6/7, and 8/9.

Original: We use fully connected multi-head networks with two hidden layers comprising 256 hidden units with
Revised: We use fully connected multi-head networks with two hidden layers comprising 256 hidden units with

Original: ReLU activations.
Revised: ReLU activations.

Original: We compare VCL (with and without coresets) to EWC and SI.
Revised: We compare VCL (with and without coresets) to EWC and SI.

Original: For the coresets, 40 data points from each task are selected through random sampling or the K-center method.
Revised: For the coresets, 40 data points from each task are selected through random sampling or the K-center method.

Original: Figure 3 compares the test set accuracy on individual tasks (averaged over 10 runs) as well as the accumulated accuracy averaged over tasks (right most column).
Revised: Figure 3 compares the test set accuracy on individual tasks (averaged over 10 runs) as well as the accumulated accuracy averaged over tasks (right most column).

Original: As an upper bound on the algorithms’  performance, we compare to batch VI trained on the full dataset.
Revised: As an upper bound on the algorithms’ [Figure] [Figure] performance, we compare to batch VI trained on the full dataset.

Original: From this ﬁgure, VCL signiﬁcantly outperforms EWC although it is slightly worse than SI.
Revised: From this ﬁgure, VCL signiﬁcantly outperforms EWC although it is slightly worse than SI.

Original: Again, EWC and SI beneﬁted from a hyper-parameter search for λ , but a value close to 1 performs well in both cases.
Revised: Again, EWC and SI beneﬁted from a hyper-parameter search for λ , but a value close to 1 performs well in both cases.

Original: After 5 tasks,
Revised: After 5 tasks,

Original: VCL achieves 97.0% average accuracy on all tasks, while EWC and SI attain 63.6% and 98.9% respectively.
Revised: VCL achieves 97.0% average accuracy on all tasks, while EWC and SI attain 63.6% and 98.9% respectively.

Original: Adding the coreset improves the performance of VCL to around 98.4% accuracy.
Revised: Adding the coreset improves the performance of VCL to around 98.4% accuracy.

Original: 6. E XPERIMENTS WITH D EEP G ENERATIVE M ODELS
Revised: 6. E XPERIMENTS WITH D EEP G ENERATIVE M ODELS

Original: We consider two continual learning experiments for deep generative models: MNIST digit generation and notMNIST (small) character generation.
Revised: We consider two continual learning experiments for deep generative models: MNIST digit generation and notMNIST (small) character generation.

Original: In both cases, ten datasets are received in sequence.
Revised: In both cases, ten datasets are received in sequence.

Original: For MNIST, the ﬁrst dataset comprises exclusively of images of the digit zero, the second[Figure] dataset ones and so on.
Revised: For MNIST, the ﬁrst dataset comprises exclusively of images of the digit zero, the second dataset ones and so on.

Original: For notMNIST (small), the datasets contain the characters A to J in sequence.
Revised: For notMNIST (small), the datasets contain the characters A to J in sequence.

Original: The generative model consists of shared and task-speciﬁc components, each represented by a one hidden layer neural network with 500 hidden units (see ﬁg. 1(b)).
Revised: The generative model consists of shared and task-speciﬁc components, each represented by a one hidden layer neural network with 500 hidden units (see ﬁg. 1(b)).

Original: The dimensionality of the latent variable z and the intermediate representation h are 50 and 500, respectively.
Revised: The dimensionality of the latent variable z and the intermediate representation h are 50 and 500, respectively.

Original: We use task-speciﬁc encoders that are neural networks with symmetric architectures to the generator.
Revised: We use task-speciﬁc encoders that are neural networks with symmetric architectures to the generator.

Original: We compare VCL to na ¨ ıve online learning using the standard VAE objective and EWC (with hyperparameters λ
Revised: We compare VCL to na¨ıve online learning using the standard VAE objective and EWC (with hyperparameters λ

Original: = 1 , 10 , 100 ).
Revised: = 1 , 10 , 100 ).

Original: For full details of the experimental settings see appendix D. Samples from the generative models attained at different time steps are shown in ﬁg. 4.
Revised: For full details of the experimental settings see appendix D. Samples from the generative models attained at different time steps are shown in ﬁg. 4.

Original: The na ¨ ıve online learning method fails catastrophically and so we have not included numerical results in the main text.
Revised: The na ¨ ıve online learning method fails catastrophically and so we have not included numerical results in the main text.

Original: EWC and VCL manage to remember the previous tasks.
Revised: EWC and VCL manage to remember the previous tasks.

Original: EWC and VCL are quantitatively evaluated using two metrics in ﬁg. 5: an importance sampling estimate of the test log-likelihood (test-LL) using 5 , 000 samples and a measure of quality we term “classiﬁer uncertainty”.
Revised: EWC and VCL are quantitatively evaluated using two metrics in ﬁg. 5: an importance sampling estimate of the test log-likelihood (test-LL) using 5 , 000 samples and a measure of quality we term “classiﬁer uncertainty”.

Original: For the latter, we train a discriminative classiﬁer for the digits/alphabets.
Revised: For the latter, we train a discriminative classiﬁer for the digits/alphabets.

Revised: [Figure]

Original: The quality of generated samples can then be assessed by computing the entropy of the output classiﬁcation probability vector.
Revised: The quality of generated samples can then be assessed by computing the entropy of the output classiﬁcation probability vector.

Original: A well-trained generator will produce images that are conﬁdently classiﬁed resulting in zero entropy.
Revised: A well-trained generator will produce images that are conﬁdently classiﬁed resulting in zero entropy.

Original: Initially, VCL performs slightly worse in terms of test-LL on the most recent task when compared to EWC.
Revised: Initially, VCL performs slightly worse in terms of test-LL on the most recent task when compared to EWC.

Original: This is likely due to the fact that VCL optimizes a distribution over random weights θ while EWC computes point estimates, and the latter strategy often produces higher test-LL.
Revised: This is likely due to the fact that VCL optimizes a distribution over random weights θ while EWC computes point estimates, and the latter strategy often produces higher test-LL.

Original: However, importantly VCL has a superior long-term memory of previous tasks which leads to better overall performance on both metrics.
Revised: However, importantly VCL has a superior long-term memory of previous tasks which leads to better overall performance on both metrics.

Original: Again, we note that EWC beneﬁted here from carefully tuning the hyper-parameter.
Revised: Again, we note that EWC beneﬁted here from carefully tuning the hyper-parameter.

Original: This again reveals a key advantage of the
Revised: This again reveals a key advantage of the

Original: VCL approach: its objective function is hyper-parameter free.
Revised: VCL approach: its objective function is hyper-parameter free.

Original: Very interestingly, all EWC methods suffer from performance deteriorations when moving from task “digit 0” to “digit 1”, probably because these two digits are very different.
Revised: Very interestingly, all EWC methods suffer from performance deteriorations when moving from task “digit 0” to “digit 1”, probably because these two digits are very different.

Original: Future work will investigate continual learning on a sequence of tasks that follows “adversarial ordering”, i.e. the ordering that makes the next task maximally different from the current task.[Figure]
Revised: Future work will investigate continual learning on a sequence of tasks that follows “adversarial ordering”, i.e. the ordering that makes the next task maximally different from the current task.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Approximate Bayesian inference provides a natural framework for continual learning.
Revised: Approximate Bayesian inference provides a natural framework for continual learning.

Original: VCL, developed in this paper, is an approach in this vein that extends online variational inference to handle more general continual learning tasks and complex neural network models.
Revised: VCL, developed in this paper, is an approach in this vein that extends online variational inference to handle more general continual learning tasks and complex neural network models.

Original: VCL can be enhanced by including a small episodic memory that leverages coreset algorithms from statistics.
Revised: VCL can be enhanced by including a small episodic memory that leverages coreset algorithms from statistics.

Original: We demonstrated how the VCL framework can be applied to both discriminative and generative models.
Revised: We demonstrated how the VCL framework can be applied to both discriminative and generative models.

Original: Experimental results showed state-of-the-art performance when compared to previous continual learning approaches, even though VCL has no free parameters in its objective function.
Revised: Experimental results showed state-of-the-art performance when compared to previous continual learning approaches, even though VCL has no free parameters in its objective function.

Original: Future work should explore alternative approximate inference methods using the same framework and also develop more sophisticated episodic memories.
Revised: Future work should explore alternative approximate inference methods using the same framework and also develop more sophisticated episodic memories.

Original: Finally, we note that VCL is ideally suited for efﬁcient model reﬁnement in sequential decision making problems, such as RL and active learning. 
Revised: Finally, we note that VCL is ideally suited for efﬁcient model reﬁnement in sequential decision making problems, such as RL and active learning. [Figure]

Original: A PPENDIX A E XPERIMENT S ETTINGS FOR THE P ERMUTED MNIST E XPERIMENT
Revised: A PPENDIX A E XPERIMENT S ETTINGS FOR THE P ERMUTED MNIST E XPERIMENT

Original: In this experiment we use fully connected single-head networks with two hidden layers, where each layer contains 100 hidden units with ReLU activations.
Revised: In this experiment we use fully connected single-head networks with two hidden layers where each layer contains 100 hidden units with ReLU activations.

Original: The metric used for comparison is the test set accuracy on all observed tasks.
Revised: The metric used for comparison is the test set accuracy on all observed tasks.

Original: We train all the models using the Adam optimizer (Kingma &
Revised: We train all the models using the Adam optimizer (Kingma &

Original: Ba, 2015) with learning rate 10 − 3 since we found that it works best for all models.
Revised: Ba, 2015) with learning rate 10 − 3 since we found that it works best for all models.

Original: All the VCL algorithms are trained with batch size 256 and 100 epochs.
Revised: All the VCL algorithms are trained with batch size 256 and 100 epochs.

Original: For all the algorithms with coresets, we choose 200 examples from each task to include into the coresets.
Revised: For all the algorithms with coresets, we choose 200 examples from each task to include into the coresets.

Original: The algorithms that use only the coresets are trained using the VFE method with batch size equal to the coreset size and 100 epochs.
Revised: The algorithms that use only the coresets are trained using the VFE method with batch size equal to the coreset size and 100 epochs.

Original: We compare the performance of the SI method that uses c = 0 .
Revised: We compare the performance of the SI method that uses c = 0 .

Original: 5 , 1 , 2 , where c is the regularization strength parameter (see ﬁg. 6).
Revised: 5 , 1 , 2 , where c is the regularization strength parameter (see ﬁg. 6).

Original: Following the settings in (Zenke et al., 2017), we train these models with batch size 256 and 20 epochs.
Revised: Following the settings in (Zenke et al., 2017), we train these models with batch size 256 and 20 epochs.

Original: The best value of c for this method is 0 .
Revised: The best value of c for this method is 0 .

Original: 5 , so we use this value as the baseline for our experiment.
Revised: 5 , so we use this value as the baseline for our experiment.

Original: We also compare the performance of the EWC method that uses λ = 1 , 10 , 10 2 , . .
Revised: We also compare the performance of the EWC method that uses λ = 1 , 10 , 10 2 , . .

Original: , 10 7 , where λ is the regularization strength parameter (see ﬁg. 7).
Revised: , 10 7 , where λ is the regularization strength parameter (see ﬁg. 7).

Original: The models are trained with batch size 200 andepochs.
Revised: The models are trained with batch size 200 andepochs.

Original: We approximate the Fisher information matrix using 600 random samples drawn from the current dataset.
Revised: We approximate the Fisher information matrix using 600 random samples drawn from the current dataset.

Original: The best value of λ for this method is 10 6 , thus we use this value as the baseline for our experiment.
Revised: The best value of λ for this method is 10 6 , thus we use this value as the baseline for our experiment.

Original: # tasks 0 .0 .0 .0 .0 .0 .0 . A cc u r a c y c=0. c=0. c=0.
Revised: [Figure]

Original: c= c=
Revised: [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original:  λ =λ = λ = 10λ = 10λ = 10λ = 10λ = 10λ
Revised: A cc u r a c y λ =λ =λ = 10λ = 10λ = 10λ = 10λ = 10λ

Original: = 

Revised: =

Original: B E XPERIMENT S ETTINGS FOR THE S PLIT MNIST E XPERIMENT
Revised: B E XPERIMENT S ETTINGS FOR THE S PLIT MNIST E XPERIMENT

Original: In this experiment, we use fully connected multi-head networks with two hidden layers, each of which contains 256 hidden units with ReLU activations.
Revised: In this experiment, we use fully connected multi-head networks with two hidden layers, each of which contains 256 hidden units with ReLU activations.

Original: At each time step, we compare the test set accuracy of the current model on all observed tasks separately.
Revised: At each time step, we compare the test set accuracy of the current model on all observed tasks separately.

Original: We also plot the average accuracy over all tasks in the last column of ﬁg. 3.
Revised: We also plot the average accuracy over all tasks in the last column of ﬁg. 3.

Original: All the results for this experiment are the averages overruns of the algorithms with different random seeds.
Revised: All the results for this experiment are the averages overruns of the algorithms with different random seeds.

Original: We also use the Adam optimizer with learning rate 10 − 3 for all models in this experiment.
Revised: We also use the Adam optimizer with learning rate 10 − 3 for all models in this experiment.

Original: All the
Revised: All the

Original: VCL algorithms are trained with batch size equal to the size of the training set and 120 epochs.
Revised: VCL algorithms are trained with batch size equal to the size of the training set and 120 epochs.

Original: For the coresets, we choose 40 examples from each task to include into the coresets.
Revised: For the coresets, we choose 40 examples from each task to include into the coresets.

Original: The algorithms that use only the coresets are trained using the VFE method with batch size equal to the coreset size and 120 epochs.
Revised: The algorithms that use only the coresets are trained using the VFE method with batch size equal to the coreset size and 120 epochs.

Original: We compare the performance of the SI method with c = 0 .
Revised: We compare the performance of the SI method with c = 0 .

Original: and use the best value c = 1 as the baseline for our experiment (see ﬁg. 8).
Revised: and use the best value c = 1 as the baseline for our experiment (see ﬁg. 8).

Original: We also compare the EWC method with both single-head and multi-head models and λ = 1 , 10 , 10 2 , 10 3 , 10 4 (see ﬁg. 9).
Revised: We also compare the EWC method with both single-head and multi-head models and λ = 1 , 10 , 10 2 , 10 3 , 10 4 (see ﬁg. 9).

Original: The ﬁgure shows that the multi-head models work better than the single-head models for EWC, and the performance is insensitive to the choice of λ .
Revised: The ﬁgure shows that the multi-head models work better than the single-head models for EWC, and the performance is insensitive to the choice of λ .

Original: Thus, we use the multi-head model with λ = 10 4 as the EWC baseline for our experiment.
Revised: Thus, we use the multi-head model with λ = 10 4 as the EWC baseline for our experiment.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: method (Kirkpatrick et al., 2017) with different regularization strength parameter ( λ ) values.

Original:  2 x     x [Figure] [Figure]
Revised: [Figure] 2 x     x0 .0 .0 .

Original: [Figure]
Revised: [Figure]

Original:  x     {+x0 .0 .
 .mean+}
Revised: 2 x     {+x0 .0 .
 .mean+}

Original: [Figure]
Revised: [Figure]

Original: 2 x     x
Revised: _vi_iter_1_

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: 2 x     x0 .0 .0 .

Original: [Figure]
Revised: [Figure]

Original: C A DDITIONAL E XPERIMENT ON A T OY 2D D ATASET
Revised: C A DDITIONAL E XPERIMENT ON A T OY 2D D ATASET

Original: In this small experiment, we compare VCL and EWC on a toy 2D dataset.
Revised: In this small experiment, we compare VCL and EWC on a toy 2D dataset.

Original: The experiment has two sequential binary classiﬁcation tasks.
Revised: The experiment has two sequential binary classiﬁcation tasks.

Original: The ﬁrst task contains two classes generated from two
Revised: The ﬁrst task contains two classes generated from two

Original: Gaussian distributions.
Revised: Gaussian distributions.

Original: The data points (with green and black classes) for this tasks are shown in the ﬁrst column of ﬁg. 10.
Revised: The data points (with green and black classes) for this tasks are shown in the ﬁrst column of ﬁg. 10.

Original: The second task contains two classes also generated from two Gaussian distributions.
Revised: The second task contains two classes also generated from two Gaussian distributions.

Original: The green class for this task has the same input distribution as the ﬁrst task, while the input distribution for the black class is different.
Revised: The green class for this task has the same input distribution as the ﬁrst task, while the input distribution for the black class is different.

Original: The data points for this task are shown in the second and third columns of ﬁg. 10.
Revised: The data points for this task are shown in the second and third columns of ﬁg. 10.

Original: Each task contains 200 data points with 100 data points in each class.
Revised: Each task contains 200 data points with 100 data points in each class.

Revised: [Figure]

Original: We compare the multi-head models trained by VCL and EWC on these two tasks.
Revised: We compare the multi-head models trained by VCL and EWC on these two tasks.

Original: In this experiment, we use fully connected networks with one hidden layers containing 20 hidden units with ReLU activations.
Revised: In this experiment, we use fully connected networks with one hidden layers containing 20 hidden units with ReLU activations.

Original: The ﬁrst column of ﬁg. 10 shows the contours of the prediction probabilities after observing the ﬁrst task, and both methods perform reasonably well for this task.
Revised: The ﬁrst column of ﬁg. 10 shows the contours of the prediction probabilities after observing the ﬁrst task, and both methods perform reasonably well for this task.

Original: However, after observing the second task, the EWC method fails to learn the classiﬁers for both tasks, while the
Revised: However, after observing the second task, the EWC method fails to learn the classiﬁers for both tasks, while the

Original: VCL method are still able to learn good classiﬁers for them.
Revised: VCL method are still able to learn good classiﬁers for them.

Original: D E XPERIMENTAL S ETTINGS AND M ORE S AMPLES FROM THE DGM S
Revised: D E XPERIMENTAL S ETTINGS AND M ORE S AMPLES FROM THE DGM S

Original: Figure 11 shows additional images generated from the generators.
Revised: Figure 11 shows additional images generated from the generators.

Original: In this experiment, the learning rates and numbers of optimization epochs are tuned on separate training of each tasks.
Revised: In this experiment, the learning rates and numbers of optimization epochs are tuned on separate training of each tasks.

Original: This gives a learning rate of 10 − 4 and the number of epochs 200 for MNIST and 400 for notMNIST.
Revised: This gives a learning rate of 10 − 4 and the number of epochs 200 for MNIST and 400 for notMNIST.

Original: For the
Revised: For the

Original: VCL approach, the parameters of q t ( θ ) are initialized to have the same mean as q t − 1 ( θ ) and the log standard deviation to 10 − 6 .
Revised: VCL approach, the parameters of q t ( θ ) are initialized to have the same mean as q t − 1 ( θ ) and the log standard deviation to 10 − 6 .

Original: The generative model consists of shared and task-speciﬁc components, each represented by a one hidden layer neural network with 500 hidden units (see ﬁg. 1(b)).
Revised: The generative model consists of shared and task-speciﬁc components, each represented by a one hidden layer neural network with 500 hidden units (see ﬁg. 1(b)).

Original: The dimensionality of the latent variable z and the intermediate representation h are 50 and 500, respectively.
Revised: The dimensionality of the latent variable z and the intermediate representation h are 50 and 500, respectively.

Original: We use task-speciﬁc encoders that are neural networks with symmetric architectures to the generator.
Revised: We use task-speciﬁc encoders that are neural networks with symmetric architectures to the generator.

Original: E M EMORY OF A L INEAR R EGRESSION M ODEL T RAINED WITH VCL ON R ANDOM P ATTERNS
Revised: E M EMORY OF A L INEAR R EGRESSION M ODEL T RAINED WITH VCL ON R ANDOM P ATTERNS

Original: In many probabilistic models with conjugate priors, the exact posterior of the parameters/latent variables can be obtained.
Revised: In many probabilistic models with conjugate priors, the exact posterior of the parameters/latent variables can be obtained.

Original: For example, a Bayesian linear regression model with a Gaussian prior over the parameters and a Gaussian observation model has a Gaussian posterior.
Revised: For example, a Bayesian linear regression model with a Gaussian prior over the parameters and a Gaussian observation model has a Gaussian posterior.

Original: If we insist on using a diagonal Gaussian approximation to this posterior and use either the variational free energy method or the Laplace approximation, we will end up at the same solution – a Gaussian distribution with the same mean as that of the exact posterior and the diagonal precisions being the diagonal precisions of the exact posterior.
Revised: If we insist on using a diagonal Gaussian approximation to this posterior and use either the variational free energy method or the Laplace approximation, we will end up at the same solution – a Gaussian distribution with the same mean as that of the exact posterior and the diagonal precisions being the diagonal precisions of the exact posterior.

Original: Consequently, the online variational Gaussian approximation will give the same result to that given by the online Laplace approximation.
Revised: Consequently, the online variational Gaussian approximation will give the same result to that given by the online Laplace approximation.

Original: However, when a diagonal
Revised: However, when a diagonal

Original: Gaussian approximation is used, the batch and sequential solutions are different.
Revised: Gaussian approximation is used, the batch and sequential solutions are different.

Original: In the following, we will explicitly detail the sequential variational updates for a Bayesian linear regression model to associate random binary patterns to binary outcomes (Kirkpatrick et al., 2017), and show its relationship to the online Laplace approximation and the EWC approach of
Revised: In the following, we will explicitly detail the sequential variational updates for a Bayesian linear regression model to associate random binary patterns to binary outcomes (Kirkpatrick et al., 2017), and show its relationship to the online Laplace approximation and the EWC approach of

Original: Kirkpatrick et al.
Revised: Kirkpatrick et al.

Original: The task consists of associating a random D -dimensional binary vectors x t to a random binary output y t by learning a weight vector W .
Revised: The task consists of associating a random D -dimensional binary vectors x t to a random binary output y t by learning a weight vector W .

Original: Note that the possible values of the features and outputs are 1 and − 1 , and not 0 and 1.
Revised: Note that the possible values of the features and outputs are 1 and − 1 , and not 0 and 1.

Original: We also assume that the model sees only one input-output pair, { x t , y t } at the t -th time step and the previous approximate posterior q t − 1 ( W ) =
Revised: We also assume that the model sees only one input-output pair, { x t , y t } at the t -th time step and the previous approximate posterior q t − [Equation] N

Original: (cid:81)
Revised: [Equation] d [Equation] t − 1 ,d ,

Original: Dd =1 N ( w d ; m t − 1 ,d , v t − 1 ,d ) , and that the observation noise variance σ 2 y is ﬁxed.
Revised: [Equation] t − [Equation] d ) , and that the observation noise variance σ 2 y is ﬁxed.

Original: The variational updates for the mean and precisions are available in closed-form as follows:
Revised: The variational updates for the mean and precisions are available in closed-form as follows:

Original: [Equation]
Revised: [Equation]

Original: By further assuming that σ y = 1 and (cid:107) x t (cid:107) 2 = 1 , the equations above become:
Revised: By further assuming that σ y = 1 and (cid:107) x t (cid:107) 2 = 1 , the equations above become:

Original: [Equation]
Revised: [Equation]

Original: where ¯ x t = y t x t .
Revised: where ¯ x t = y t x t .

Original: When v − 1 0 ,d = 0, i.e. the prior is ignored, the update for the mean above is exactly equation S4 in the supplementary material of Kirkpatrick et al.
Revised: When v − 1 0 ,d = 0, i.e. the prior is ignored, the update for the mean above is exactly equation S4 in the supplementary material of Kirkpatrick et al.

Original: Therefore, in this case, the memory of the network trained by online variational inference is identical to that of the online
Revised: Therefore, in this case, the memory of the network trained by online variational inference is identical to that of the online

Original: Laplace method, provided in Kirkpatrick et al.
Revised: Laplace method, provided in Kirkpatrick et al.

Original: These methods differ, in practice, when the prior is not ignore or when the parameter regularization constraints are accumulated as discussed in the main text.
Revised: These methods differ, in practice, when the prior is not ignore or when the parameter regularization constraints are accumulated as discussed in the main text.

Original: This equivalence also does not hold in the general case, as discussed by Opper &
Revised: This equivalence also does not hold in the general case, as discussed by Opper &

Original: Archambeau (2009) that the Gaussian variational approximation is like an averaged and smoothed
Revised: Archambeau (2009) that the Gaussian variational approximation is like an averaged and smoothed

Original: Laplace approximation.
Revised: Laplace approximation.