Original: Task Agnostic Continual Learning via Meta Learning
Revised: Task Agnostic Continual Learning via Meta Learning

Original: Abstract
Revised: Abstract

Original: Most continual learning approaches implicitly assume that there exists a multi-task solution for the sequence of tasks.
Revised: Most continual learning approaches implicitly assume that there exists a multi-task solution for the sequence of tasks.

Original: In this work, we motivate and discuss realistic scenarios when this assumption does not hold.
Revised: In this work, we motivate and discuss realistic scenarios when this assumption does not hold.

Original: We argue that the traditional metric of zero-shot remembering is not appropriate in such settings, and, inspired by the meta-learning literature, we focus on the speed of remembering previous tasks.
Revised: We argue that the traditional metric of zero-shot remembering is not appropriate in such settings, and, inspired by the meta-learning literature, we focus on the speed of remembering previous tasks.

Original: A natural approach to deal with this case is to separate the concerns into what task is currently being solved and how the task should be solved.
Revised: A natural approach to deal with this case is to separate the concerns into what task is currently being solved and how the task should be solved.

Original: At each step, the what algorithm performs task inference, which allows our framework to work in absence of task boundaries.
Revised: At each step, the what algorithm performs task inference, which allows our framework to work in absence of task boundaries.

Original: The how algorithm is conditioned on the inferred task, allowing for task-speciﬁc behaviour, hence relaxing the assumption of a multi-task solution.
Revised: The how algorithm is conditioned on the inferred task, allowing for task-speciﬁc behaviour, hence relaxing the assumption of a multi-task solution.

Original: From the perspective of meta-learning, our framework is able to deal with a sequential presentation of tasks, rather than having access to the distribution of all tasks.
Revised: From the perspective of meta-learning, our framework is able to deal with a sequential presentation of tasks, rather than having access to the distribution of all tasks.

Original: We empirically validate the effectiveness of our approach and discuss variations of the proposed algorithm.
Revised: We empirically validate the effectiveness of our approach and discuss variations of the proposed algorithm.

Original: 1. Introduction
Revised: 1. Introduction

Original: Connectionist networks are known to suffer from catastrophic forgetting (CF) (McCloskey & Cohen, 1989), when the underlying data stream is not independently and identically distributed (i.i.d).
Revised: Connectionist networks are known to suffer from catastrophic forgetting (CF) (McCloskey & Cohen, 1989), when the underlying data stream is not independently and identically distributed (i i d).

Original: Continual Learning (CL) explores this problem, where non-stationarity of data is given as a sequence of distinct tasks.
Revised: Continual Learning (CL) explores this problem, where non-stationarity of data is given as a sequence of distinct tasks.

Original: One typical goal of many CL algorithms is to ensure that, after training on a sequence of tasks, the performance of the network is close to a network trained on all tasks at the same time.
Revised: One typical goal of many CL algorithms is to ensure that, after training on a sequence of tasks, the performance of the network is close to a network trained on all tasks at the same time.

Original: Hence, all these methods implicitly assume that there is always a multi-task solution that ﬁts all previous tasks.
Revised: Hence, all these methods implicitly assume that there is always a multi-task solution that ﬁts all previous tasks.

Original: Another common assumption is that either the identities of different tasks or the boundaries between them are available to the learner, and many CL methods often depend on this information to know when to consolidate the knowledge learned so far.
Revised: Another common assumption is that either the identities of different tasks or the boundaries between them are available to the learner, and many CL methods often depend on this information to know when to consolidate the knowledge learned so far.

Original: However, there are many realistic scenarios where both of the above-mentioned assumptions do not hold.
Revised: However, there are many realistic scenarios where both of the above-mentioned assumptions do not hold.

Original: Consider a multi-agent game in reinforcement learning (RL), where all agents are learning and adapting their policies.
Revised: Consider a multi-agent game in reinforcement learning (RL), where all agents are learning and adapting their policies.

Original: For any of these agents, the objective it tries to optimize (in other words, its task) depends not only on itself and the environment, but also on the policies and conﬁgurations of others, which are usually not directly observable.
Revised: For any of these agents, the objective it tries to optimize (in other words, its task) depends not only on itself and the environment, but also on the policies and conﬁgurations of others, which are usually not directly observable.

Original: Moreover, the other agents might change their policies at any moment as they are also learning.
Revised: Moreover, the other agents might change their policies at any moment as they are also learning.

Original: As a result, the task for this agent is changing all the time and there are no clearly deﬁned boundaries available to the agent.
Revised: As a result, the task for this agent is changing all the time and there are no clearly deﬁned boundaries available to the agent.

Original: It has been observed that such non-stationarity in multi-agent systems usually causes catastrophic forgetting of the agent (Hernandez-Leal et al., 2019).
Revised: It has been observed that such non-stationarity in multi-agent systems usually causes catastrophic forgetting of the agent (Hernandez-Leal et al., 2019).

Original: For example, Vinyals et al.
Revised: For example, Vinyals et al.

Original: (2019) trained agents to play the video game StarCraft II by self-play (Tesauro, 1995), and they noticed that one salient drawback of this approach is in fact forgetting: the agent may forget how to defeat a previous version of itself as training progresses, and this may lead to a “tail-chasing” cycle where the agents always relearn a previously learned strategy and training never converges.
Revised: (2019) trained agents to play the video game StarCraft II by self-play (Tesauro, 1995), and they noticed that one salient drawback of this approach is in fact forgetting: the agent may forget how to defeat a previous version of itself as training progresses, and this may lead to a “tail-chasing” cycle where the agents always relearn a previously learned strategy and training never converges.

Original: Furthermore, since the tasks now depend on the conﬁgurations of other agents, in general, there is no guarantee that a multi-task solution would exist in these settings.
Revised: Furthermore, since the tasks now depend on the conﬁgurations of other agents, in general, there is no guarantee that a multi-task solution would exist in these settings.

Original: A potential example is Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), where a generator G and a discriminator D are trained together by playing a minimax game.
Revised: A potential example is Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), where a generator G and a discriminator D are trained together by playing a minimax game.

Original: The objective of D is to classify the data as real or fake, whereas the goal of G is to fool D as much as possible by generating fake data.
Revised: The objective of D is to classify the data as real or fake, whereas the goal of G is to fool D as much as possible by generating fake data.

Original: It was shown in (Goodfellow et al., 2014) that the optimal discriminator D ∗ ( x ) is a function of the generator probability density function p G (x ) ,
Revised: It was shown in (Goodfellow et al., 2014) that the optimal discriminator D ∗ ( x ) is a function of the generator probability density function p G (

Original: [Equation]
Revised: [Equation]

Original:  Therefore, if we take two snapshots G 1 , G 2 of the generator at different moments of the training process such that p G 1 ( x ) (cid:54) = p G 2 ( x ) for some x where p data ( x ) (cid:54) = 0 , then their corresponding optimal discriminators have to be different.
Revised: [Equation] Therefore, if we take two snapshots G 1 , G 2 of the generator at different moments of the training process such that [Equation] ) for some x where p data ( x ) (cid:54) = 0 , then their corresponding optimal discriminators have to be different.

Original: In other words, there is no multi-task solution for the discriminator to be optimal for both generators.
Revised: In other words, there is no multi-task solution for the discriminator to be optimal for both generators.

Original: As a result, optimizing the discriminator with respect to a new version of the generator will inevitably lead to degradation of its performance with respect to a past generator, which is considered forgetting by the traditional metric of continual learning.
Revised: As a result, optimizing the discriminator with respect to a new version of the generator will inevitably lead to degradation of its performance with respect to a past generator, which is considered forgetting by the traditional metric of continual learning.

Original: Indeed, it has been shown empirically by Thanh-Tung &
Revised: Indeed, it has been shown empirically by Thanh-Tung &

Original: Tran (2019); Liang et al.
Revised: Tran (2019); Liang et al.

Original: (2019) that GANs suffer from catastrophic forgetting,
Revised: (2019) that GANs suffer from catastrophic forgetting,

Original: and they adapted CL methods such as Elastic Weight Consolidation (EWC)
Revised: and they adapted CL methods such as Elastic Weight Consolidation (EWC)

Original: (Kirkpatrick et al., 2017) and Synaptic Intelligence (SI) (Zenke et al., 2017) to alleviate forgetting in GANs .
Revised: (Kirkpatrick et al., 2017) and Synaptic Intelligence (SI) (Zenke et al., 2017) to alleviate forgetting in GANs .

Original: However, as we pointed out before, these methods were initially designed with implicit assumptions that a multitask solution always exists and precise task boundaries are available, which makes them unsuitable for the setting of multi-agent games like GANs.
Revised: However, as we pointed out before, these methods were initially designed with implicit assumptions that a multitask solution always exists and precise task boundaries are available, which makes them unsuitable for the setting of multi-agent games like GANs.

Original: In this work, we propose a CL framework that does not make these assumptions and is applicable in a task agnostic scenario where the tasks can potentially be conﬂicting with each other.
Revised: In this work, we propose a CL framework that does not make these assumptions and is applicable in a task agnostic scenario where the tasks can potentially be conﬂicting with each other.

Original: Furthermore, to evaluate our framework, we shift our focus from less forgetting to faster remembering: to rapidly recover the performance on a previously learned task, given the right context as a cue.
Revised: Furthermore, to evaluate our framework, we shift our focus from less forgetting to faster remembering: to rapidly recover the performance on a previously learned task, given the right context as a cue.

Original: 2. Formal Statement
Revised: 2. Formal Statement

Original: We consider an online learning scenario similar to Hochreiter et al.
Revised: We consider an online learning scenario similar to Hochreiter et al.

Original: ; Nagabandi et al.
Revised: ; Nagabandi et al.

Original: (2019), where at each time step t , a model ˆ f parametrised by θ t receives an observation x t and makes a prediction ˆ y t :
Revised: (2019), where at each time step t , a model ˆ f parametrised by θ t receives an observation x t and makes a prediction ˆ y t :

Original: = ˆ f ( x t ; θ t ) .
Revised: = ˆ f ( x t ; θ t ) .

Original: It then gets the ground truth y t on that task, which can be used to optimize its parameters for better performance in the future.
Revised: It then gets the ground truth y t on that task, which can be used to optimize its parameters for better performance in the future.

Original: If the data distribution is non-stationary (for example, ( x, y ) are sampled from task A for a while, then the task switches to task B at some moment t (cid:48) ), then training on the new data might lead to catastrophic forgetting – the new parameters θ (cid:48) can solve task B but not task A anymore.
Revised: If the data distribution is non-stationary (for example, ( x, y ) are sampled from task A for a while, then the task switches to task B at some moment t (cid:48) ), then training on the new data might lead to catastrophic forgetting – the new parameters θ (cid:48) can solve task B but not task A anymore.

Original: Many continual learning methods were proposed to alleviate the problem of catastrophic forgetting.
Revised: Many continual learning methods were proposed to alleviate the problem of catastrophic forgetting.

Original: Most of them require either the task identities ( A and B in the example) or at least the moment when the task switches ( t (cid:48) in this case).
Revised: Most of them require either the task identities ( A and B in the example) or at least the moment when the task switches ( t (cid:48) in this case).

Original: This information, however, is not available when the ground truth y t depends not only on the observation x t but also on some hidden task (or context) variable T t : y t
Revised: This information, however, is not available when the ground truth y t depends not only on the observation x t but also on some hidden task (or context) variable T t : y t

Original: = f
Revised: = f

Original: ( x t , T t ) , a common situation in partially observable environments (Monahan, 1982; Cassandra et al., 1994).
Revised: ( x t , T t ) , a common situation in partially observable environments (Monahan, 1982; Cassandra et al., 1994).

Original: Only recently, the
Revised: Only recently, the

Original: CL community started to look at the task agnostic setting (Zeno et al., 2018; Aljundi et al., 2019).
Revised: CL community started to look at the task agnostic setting (Zeno et al., 2018; Aljundi et al., 2019).

Original: However, all these methods have the underlying assumption that no matter what tasks the learner has been learning, at any time t , it is always possible to ﬁnd parameters θ t that ﬁt all previous tasks:
Revised: However, all these methods have the underlying assumption that no matter what tasks the learner has been learning, at any time t , it is always possible to ﬁnd parameters θ t that ﬁt all previous tasks:

Original: ∃ θ t s.t.
Revised: ∃ θ t s.t.

Original: ∀ t (cid:48) ≤ t, ˆ f ( x t (cid:48) , θ t )
Revised: ∀ t (cid:48) ≤ t, ˆ f ( x t (cid:48) , θ t )

Original: ≈ y t (cid:48) .
Revised: ≈ y t (cid:48) .

Original: As discussed in the previous section, this assumption does not hold in many realistic scenarios where different tasks conﬂict with each other: f ( x t , T t ) (cid:54) =
Revised: As discussed in the previous section, this assumption does not hold in many realistic scenarios where different tasks conﬂict with each other: f ( x t , T t ) (cid:54) =

Original: f ( x t (cid:48) , T t (cid:48) ) even when x t = x t (cid:48) .
Revised: f ( x t (cid:48) , T t (cid:48) ) even when x t = x t (cid:48) .

Original: It follows that, in those settings, catastrophic forgetting cannot be avoided if the model ˆ f ( · ; θ t ) does not depend on the hidden task variable T t .
Revised: It follows that, in those settings, catastrophic forgetting cannot be avoided if the model ˆ f ( · ; θ t ) does not depend on the hidden task variable T t .

Original: 3. What & How Framework
Revised: 3. What & How Framework

Original: Here we propose a framework for task agnostic continual learning that explicitly infers the current task from some context data D ctx t and makes predictions based on both the inputs x t and the inferred task representations c t .
Revised: Here we propose a framework for task agnostic continual learning that explicitly infers the current task from some context data D ctx t and makes predictions based on both the inputs x t and the inferred task representations c t .

Original: The framework consists of two modules: a task inference encoder algorithm F what : D ctx t → c t that predicts the current task representation c t based on the context data D ctx t , and a decoder algorithm F
Revised: The framework consists of two modules: a task inference encoder algorithm F what : D ctx t → c t that predicts the current task representation c t based on the context data D ctx t , and a decoder algorithm F

Original: How : c t → ˆ f t that maps the task representation c t to a task speciﬁc model ˆ f t : x → ˆ y .
Revised: How : c t → ˆ f t that maps the task representation c t to a task speciﬁc model ˆ f t : x → ˆ y .

Original: Under this framework, even when the inputs x t and x t (cid:48) are the same, the predictions ˆ y t and ˆ y t (cid:48) can be different from each other depending on the context.
Revised: Under this framework, even when the inputs x t and x t (cid:48) are the same, the predictions ˆ y t and ˆ y t (cid:48) can be different from each other depending on the context.

Original: In this work, we choose the recent k observations { ( x t − k , y t − k ) , · · · ( x t − 1 , y t − 1 ) } as the context dataset D ctx t .
Revised: In this work, we choose the recent k observations { ( x t − k , y t − k ) , · · · ( x t − 1 , y t − 1 ) } as the context dataset D ctx t .

Original: This choice is reasonable in an environment where the task variable T t is piece-wise stationary or changes smoothly.
Revised: This choice is reasonable in an environment where the task variable T t is piece-wise stationary or changes smoothly.

Original: An overview of this framework is illustrated in Figure 1.
Revised: An overview of this framework is illustrated in Figure 1.

Original: 3.1. Meta Learning as Task Inference
Revised: 3.1. Meta Learning as Task Inference

Original: In fact, many recently proposed meta-learning methods can be seen as decomposing the problem into What and How modules.
Revised: In fact, many recently proposed meta-learning methods can be seen as decomposing the problem into What and How modules.

Original: For example, Conditional Neural Processes (CNP) (Garnelo et al., 2018) embed the observation and target pairs in context data ( x i , y i )
Revised: For example, Conditional Neural Processes (CNP) (Garnelo et al., 2018) embed the observation and target pairs in context data ( x i , y i )

Original: ∈ D ctx t by an encoder network r i
Revised: ∈ D ctx t by an encoder network r i

Original: =
Revised: =

Original: h
Revised: h

Original: ( x i , y i ; θ h ) .
Revised: ( x i , y i ; θ h ) .

Original: The embeddings are then aggregated by a commutative operation ⊕ (such as the mean operation) to obtain a single embedding of the context: r [Equation] = F What ( D ctx t ; θ h ) =
Revised: The embeddings are then aggregated by a commutative operation ⊕ (such as the mean operation) to obtain a single embedding of the context: r t = F What ( D ctx t ; θ h ) =

Original: (cid:76) x i ,y i ∈D ctx
Revised: (cid:76) x i ,y i ∈D ctx

Original: t h ( x i , y i ; θ h ) .
Revised: t h ( x i , y i ; θ h ) .

Original: At inference time, the context embedding is passed as an additional input to a decoder g to produce the conditional outputs:
Revised: At inference time, the context embedding is passed as an additional input to a decoder g to produce the conditional outputs:

Original: F How ( r t ) =
Revised: F How ( r t ) =

Original: g ( · , r t ; θ g ) .
Revised: g ( · , r t ; θ g ) .

Original: Model-Agnostic Meta-Learning (MAML)
Revised: Model-Agnostic Meta-Learning (MAML)

Original: (Finn et al., 2017) infers the current task by applying one or a few steps of gradient descent on the context data D ctx t .
Revised: (Finn et al., 2017) infers the current task by applying one or a few steps of gradient descent on the context data D ctx t .

Original: In this case, the gradient descent algorithm is the What encoder and the resulting task-speciﬁc parameters can be considered a high-dimensional representation of the current task: 
Revised: In this case, the gradient descent algorithm is the What encoder and the resulting task-speciﬁc parameters can be consid- ered a high-dimensional representation of the current task: θ kt = F

Original: [Equation] ; θ init )
Revised: What ( D ctx t ; θ init )

Original: =
Revised: =

Original: U k
Revised: U k

Original: ( θ init , D ctx t , λ in ) :
Revised: ( θ init , D ctx t , λ in ) :

Original: = θ k − 1 t − λ in ∇ θ L in ( ˆ f ( · ; θ k − 1 t ) , D ctx t ) , where the meta parameters θ init t are the initial values of the model parameters, U k is the operator that updates θ init by k steps of gradient descent on the context data D ctx t with an inner loop learning rate λ in and an inner loop loss function L in .
Revised: = θ k − 1 t − λ in ∇ θ L in ( ˆ f ( · ; θ k − 1 t ) , D ctx t ) , where the meta parameters θ init t are the initial values of the model parameters, U k is the operator that updates θ init by k steps of gradient descent on the context data D ctx t with an inner loop learning rate λ in and an inner loop loss function L in .

Original: The How decoder of MAML returns the task-speciﬁc model by simply re-parametrizing the model ˆ f with θ t : F How ( θ t ) :
Revised: The How decoder of MAML returns the task-speciﬁc model by simply re-parametrizing the model ˆ f with θ t : F How ( θ t ) :

Original: = ˆ f ( · ; θ t ) .
Revised: = ˆ f ( · ; θ t ) .

Original: In Fast

Original: Context

Original: Adaptation via

Original: CAVIA) (Zintgraf et al., 2019), a neural network model ˆ f takes a context vector c t as an additional input: ˆ y
Revised: In Fast Context Adaptation via Meta-Learning (CAVIA) (Zintgraf et al., 2019), a neural network model ˆ f takes a context vector c t as an additional input: ˆ y

Original: = ˆ f ( x, c t ; θ ) .
Revised: = ˆ f ( x, c t ; θ ) .

Original: The context vector is inferred from context data by a few steps of gradient descent: c t = F
Revised: The context vector is inferred from context data by a few steps of gradient descent: c t = F

Original: What ( D ctx t ; θ ) :
Revised: What ( D ctx t ; θ ) :

Original: = c init − λ in ∇ c L in ( ˆ f ( · , c ; θ ) , D ctx t ) .
Revised: = c init − λ in ∇ c L in ( ˆ f ( · , c ; θ ) , D ctx t ) .

Original: Then a context-dependent model is returned by the How decoder: F How ( c t ) :
Revised: Then a context-dependent model is returned by the How decoder: F How ( c t ) :

Original: = ˆ f ( · , c t ; θ ) .
Revised: = ˆ f ( · , c t ; θ ) .

Original: In this work, to showcase our framework, we choose a simple meta learning method called Reptile (Nichol et al., 2018), mainly for its simplicity and for being computationally inexpenssive as it does not require second order gradients.
Revised: In this work, to showcase our framework, we choose a simple meta learning method called Reptile (Nichol et al., 2018), mainly for its simplicity and for being computationally inexpenssive as it does not require second order gradients.

Original: Similar to MAML, Reptile tries to learn an initialization of model parameters θ init t such that optimization on a test task is fast, so its What encoder and How decoder are exactly the same as those of MAML.
Revised: Similar to MAML, Reptile tries to learn an initialization of model parameters θ init t such that optimization on a test task is fast, so its What encoder and How decoder are exactly the same as those of MAML.

Original: To update the meta parameters θ init t , Reptile simply uses the difference between the task-speciﬁc parameters and the initialization as the gradient direction: g θ init :
Revised: To update the meta parameters θ init t , Reptile simply uses the difference between the task-speciﬁc parameters and the initialization as the gradient direction: g θ init :

Original: = θ init t − θ kt = θ init t − U k
Revised: = θ init t − θ kt = θ init t − U k

Original: ( θ init t , D ctx t , λ in ) .
Revised: ( θ init t , D ctx t , λ in ) .

Original: 3.2. Continual Meta Learning
Revised: 3.2. Continual Meta Learning

Original: In order to train a meta-learning model, one normally needs access to a task distribution so that i.i.d task samples are available at the same time during training.
Revised: In order to train a meta-learning model, one normally needs access to a task distribution so that i.i.d task samples are available at the same time during training.

Original: This is not possible in the online learning setting where tasks are presented sequentially one after the other.
Revised: This is not possible in the online learning setting where tasks are presented sequentially one after the other.

Original: Finn et al.
Revised: Finn et al.

Original: (2019) proposed an online meta-learning algorithm called follow the meta leader (FTML) based on the framework of regretminimization.
Revised: (2019) proposed an online meta-learning algorithm called follow the meta leader (FTML) based on the framework of regretminimization.

Original: However, the computational cost of FTML grows linearly over time as new losses are accumulated, and it requires to store all datapoints from previous tasks, which is usually considered infeasible in continual learning due to limited resources or privacy reasons.
Revised: However, the computational cost of FTML grows linearly over time as new losses are accumulated, and it requires to store all datapoints from previous tasks, which is usually considered infeasible in continual learning due to limited resources or privacy reasons.

Original: In this work, we choose an alternative framework of online learning called online variational Bayes (Minka et al., 2009; Opper, 1998), since it does not require unbounded computational and memory budget.
Revised: In this work, we choose an alternative framework of online learning called online variational Bayes (Minka et al., 2009; Opper, 1998), since it does not require unbounded computational and memory budget.

Original: Furthermore, when additional memory budget are available for storing datapoints, online variational Bayes can also be extended by combining it with memory-based online learning methods (Minka et al., 2009; Nguyen et al., 2017; Kurle et al., 2020).
Revised: Furthermore, when additional memory budget are available for storing datapoints, online variational Bayes can also be extended by combining it with memory-based online learning methods (Minka et al., 2009; Nguyen et al., 2017; Kurle et al., 2020).

Original: In this work, we focus on an algorithm that does not have a growing memory cost over time.
Revised: In this work, we focus on an algorithm that does not have a growing memory cost over time.

Original: Formally, let φ be the collection of meta parameters in F
Revised: Formally, let φ be the collection of meta parameters in F

Original: What and F How (for instance, θ init in MAML and Reptile)
Revised: What and F How (for instance, θ init in MAML and Reptile)

Original: such that ˆ f t = F How ◦ F What ( D ctx t ; φ ) .
Revised: such that ˆ f t = F How ◦ F What ( D ctx t ; φ ) .

Original: Using the Bayes rule, the posterior p ( φ |D 0: t ) can be recursively updated by
Revised: Using the Bayes rule, the posterior p ( φ |D 0: t ) can be recursively updated by

Original: [Equation]
Revised: [Equation]

Original: where D t = { ( x t , y t ) } and D 0: t is the union of all datapoints up to t .
Revised: where D t = { ( x t , y t ) } and D 0: t is the union of all datapoints up to t .

Original: By our assumption that a moving window of context data is informative enough about the task variable, we have
Revised: By our assumption that a moving window of context data is informative enough about the task variable, we have

Original: [Equation]
Revised: [Equation]

Original: In online variational Bayes, the true posterior is approximated by a parametric distribution q t ( φ ) by minimizing the
Revised: In online variational Bayes, the true posterior is approximated by a parametric distribution q t ( φ ) by minimizing the

Original: Kullback-Leibler divergence
Revised: Kullback-Leibler divergence

Original: [Equation]
Revised: [Equation]

Original: and if we use a parametric distribution at every time step, the optimization problem above can be simpliﬁed as maximizing the evidence lower bound (ELBO)
Revised: and if we use a parametric distribution at every time step, the optimization problem above can be simpliﬁed as maximizing the evidence lower bound (ELBO)

Original: [Equation]
Revised: [Equation]

Original: In this work, we choose the parametric distribution to be a factorized Gaussian q t ( φ ) =
Revised: In this work, we choose the parametric distribution to be a factorized Gaussian q t ( φ ) =

Original: (cid:81) i N ( φ i | µ i ( t ) , σ i ( t )) , where φ i is the i -th component of φ .
Revised: (cid:81) i N ( φ i | µ i ( t ) , σ i ( t )) , where φ i is the i -th component of φ .

Original: Using the re-parametrization trick: φ i = µ[Equation] i 
Revised: Using the re-parametrization trick: φ i = µ i + σ i (cid:15) i , (cid:15) i ∼ N (0 , 1) and let q t − 1 ( φ ) =

Original: [Equation]
Revised: (cid:81)

Original: i ,
Revised: i N

Original: [Equation]
Revised: [Equation]

Original:  i
Revised: φ i

Original: [Equation]
Revised: [Equation]

Original: N
Revised: µ i

Original: [Equation]
Revised: [Equation]

Original: 0 , 1) and let q t − 1 ( φ ) = (cid:81) i N ( φ i | µ i ( t − 1) , σ i ( t − 1)) , we can ﬁnd the maximum of the ELBO by solving the following equations
Revised: 1) , σ i ( t − 1)) , we can ﬁnd the maximum of the ELBO by solving the following equations

Original: [Equation]
Revised: [Equation]

Original: and the results are update rules for µ i and σ i that are similar to Bayesian Online Learning (Opper, 1998) and Bayesian
Revised: and the results are update rules for µ i and σ i that are similar to Bayesian Online Learning (Opper, 1998) and Bayesian

Original: Gradient Descent (BGD)
Revised: Gradient Descent (BGD)

Original: (Zeno et al., 2018), but on the meta-level:
Revised: (Zeno et al., 2018), but on the meta-level:

Original: [Equation]
Revised: [Equation]

Original: where L t ( φ ) = − log p ( D t | φ, D 0: t − 1 ) .
Revised: where L t ( φ ) = − log p ( D t | φ, D 0: t − 1 ) .

Original: An intuitive interpretation of these learning rules is that weights µ i withsmaller uncertainty σ i are more important for the knowledge accumulated so far, thus they should change slower in the future in order to preserve the learned knowledge.
Revised: An intuitive interpretation of these learning rules is that weights µ i with

Original: Input: µ, σ, λ in , k, J, D ctx 0 , η µ , η σ , for t = 0 , 1 , . . .
Revised: Input: µ, σ, λ in , k, J, D ctx 0 , η µ , η σ , for t = 0 , 1 , . . .

Original: do c t ← θ t =
Revised: do c t ← θ t =

Original: U k ( µ ; D ctx t , λ in ) =
Revised: U k ( µ ; D ctx t , λ in ) =

Original: F What ( D ctx t ; µ ) , ˆ y t ← ˆ f ( x t ; θ t ) =
Revised: F What ( D ctx t ; µ ) , ˆ y t ← ˆ f ( x t ; θ t ) =

Original: F How ( c t )( x t ) ∆ µ ← η µ ∂ L t ( φ ) ∂φ (cid:12)(cid:12)(cid:12) φ
Revised: F How ( c t )( x t ) ∆ µ ← η µ ∂ L t ( φ ) ∂φ (cid:12)(cid:12)(cid:12) φ

Original: = µ ≈ η µ ( µ − U k ( µ, D ctx t , λ in )) for j = 1 to J do (cid:15) j ∼ N (0 , 1)
Revised: = µ ≈ η µ ( µ − U k ( µ, D ctx t , λ in )) for j = 1 to J do (cid:15) j ∼ N (0 , 1)

Original: φ j = (cid:15)
Revised: φ j = (cid:15)

Original: j σ + µ ∆ jσ ← (cid:15) j
Revised: j σ + µ ∆ jσ ← (cid:15) j

Original: ∂ L t ( φ ) ∂φ (cid:12)(cid:12)(cid:12) φ = φ j ≈ (cid:15) j ( φ j − U k ( φ j , D ctx t , λ in )) end for ∆ σ ← η σ 1 J (cid:80) j =1 ∆ jσ µ i ← µ i − σ 2 i ∆ µ i σ i ← σ i (cid:114) 1 + (cid:16) 12 σ i ∆ σ i (cid:17) 2 − 12 σ 2 i ∆ σ i ,
Revised: ∂ L t ( φ ) ∂φ (cid:12)(cid:12)(cid:12) φ = φ j ≈ (cid:15) j ( φ j − U k ( φ j , D ctx t , λ in )) end for ∆ σ ← η σ 1 J (cid:80) j =1 ∆ jσ µ i ← µ i − σ 2 i ∆ µ i σ i ← σ i (cid:114) 1 + (cid:16) 12 σ i ∆ σ i (cid:17) 2 − 12 σ 2 i ∆ σ i ,

Original: Update D ctx t with { ( x t , y t ) } to get D ctx t + end for 
Revised: Update D ctx t with { ( x t , y t ) } to get D ctx t + end for smaller uncertainty σ i are more important for the knowledge accumulated so far, thus they should change slower in the future in order to preserve the learned knowledge.

Original: In practice, we introduce learning rates for both 6 and 7.
Revised: In practice, we introduce learning rates for both 6 and 7.

Original: Maximum a posterior (MAP) estimate of φ is used for prediction.
Revised: Maximum a posterior (MAP) estimate of φ is used for prediction.

Original: We approximate the expectation in the µ update rule 6 by the gradient at the mean, and for the expectation in the σ update rule 7, we estimate it by Monte Carlo sampling method.
Revised: We approximate the expectation in the µ update rule 6 by the gradient at the mean, and for the expectation in the σ update rule 7, we estimate it by Monte Carlo sampling method.

Original: The ﬁnal algorithm called W & H is described in
Revised: The ﬁnal algorithm called W & H is described in

Original: Algorithm 1.
Revised: Algorithm 1.

Original: Complexity
Revised: Complexity

Original: The number of parameters required by the
Revised: The number of parameters required by the

Original: W & H algorithm is 3 times that of the base model, since it needs to store the mean and the standard deviation of the initialization and a copy of the current task-speciﬁc parameters.
Revised: W & H algorithm is 3 times that of the base model, since it needs to store the mean and the standard deviation of the initialization and a copy of the current task-speciﬁc parameters.

Original: In terms of time complexity, the computation of the mean update ∆ µ and the Monte Carlo (MC) sample of the standard deviation update ∆ jσ can be parallelized.
Revised: In terms of time complexity, the computation of the mean update ∆ µ and the Monte Carlo (MC) sample of the standard deviation update ∆ jσ can be parallelized.

Original: In that case, W & H has only constant computational overhead compared to Reptile due to sampling and gradient averaging.
Revised: In that case, W & H has only constant computational overhead compared to Reptile due to sampling and gradient averaging.

Original: If the MC sampling process is implemented sequentially, the total time complexity is O
Revised: If the MC sampling process is implemented sequentially, the total time complexity is O

Original: ( J + 1) times that of the
Revised: ( J + 1) times that of the

Original: Reptile algorithm.
Revised: Reptile algorithm.

Original: Similar to the ﬁndings in Zeno et al.
Revised: Similar to the ﬁndings in Zeno et al.

Original: (2018), we ﬁnd that in practice the number of MC samples has negligible effect on the performance of the algorithm.
Revised: (2018), we ﬁnd that in practice the number of MC samples has negligible effect on the performance of the algorithm.

Original: 4. Related Work
Revised: 4. Related Work

Original: Continual learning has seen a surge in popularity in the last few years, with multiple approaches being proposed to address the problem of catastrophic forgetting.
Revised: Continual learning has seen a surge in popularity in the last few years, with multiple approaches being proposed to address the problem of catastrophic forgetting.

Original: These approaches can be largely categorized into the following types (Parisi et al., 2019): Rehearsal based methods focus on techniques to either efﬁciently store data points from previous tasks (Robins, 1995; Lopez-Paz et al., 2017) or to create pseudo datasets that are representative of past tasks.
Revised: These approaches can be largely categorized into the following types (Parisi et al., 2019): Rehearsal based methods focus on techniques to either efﬁciently store data points from previous tasks (Robins, 1995; Lopez-Paz et al., 2017) or to create pseudo datasets that are representative of past tasks.

Original: Then the stored or generated data can be used to approximate the losses of previous tasks.
Revised: Then the stored or generated data can be used to approximate the losses of previous tasks.

Original: For example, Learning without
Revised: For example, Learning without

Original: Forgetting (LwF) (Li & Hoiem, 2017)
Revised: Forgetting (LwF) (Li & Hoiem, 2017)

Original: ﬁrst labels the inputs of the current task with the previous model, then use the resulting input-output pairs for rehearsal.
Revised: ﬁrst labels the inputs of the current task with the previous model, then use the resulting input-output pairs for rehearsal.

Original: Deep Generative
Revised: Deep Generative

Original: Replay (DGR) (Shin et al., 2017) trains a generative model together with a classiﬁer, and when the task switches, the previous generative model can be used to produce pseudoexamples for rehearsing the old tasks.
Revised: Replay (DGR) (Shin et al., 2017) trains a generative model together with a classiﬁer, and when the task switches, the previous generative model can be used to produce pseudoexamples for rehearsing the old tasks.

Original: Structural based methods exploit modularity to reduce interference, localizing the updates to a subset of weights.
Revised: Structural based methods exploit modularity to reduce interference, localizing the updates to a subset of weights.

Original: Rusu et al.
Revised: Rusu et al.

Original: (2016) proposed to learn a new module for each task with lateral connection to previous modules.
Revised: (2016) proposed to learn a new module for each task with lateral connection to previous modules.

Original: This prevents catastrophic forgetting by construction and allows forward transfer, at the cost of quadratic growth in model size.
Revised: This prevents catastrophic forgetting by construction and allows forward transfer, at the cost of quadratic growth in model size.

Original: He & Jaeger (2018) proposed to use Conceptors to identify the linear subspaces in a network that are not used by previous tasks for learning future tasks.
Revised: He & Jaeger (2018) proposed to use Conceptors to identify the linear subspaces in a network that are not used by previous tasks for learning future tasks.

Original: This method does not increase the size of the network as long as the linear subspaces are not exhausted, but the network capacity will eventually saturate.
Revised: This method does not increase the size of the network as long as the linear subspaces are not exhausted, but the network capacity will eventually saturate.

Original: In (Golkar et al., 2019), pruning techniques were applied to minimize the growth of the model after each task.
Revised: In (Golkar et al., 2019), pruning techniques were applied to minimize the growth of the model after each task.

Original: Finally, Regularization based methods draw inspiration from Bayesian learning, and can be seen as utilizing the posterior after learning a sequence of tasks as a prior to regularize learning of the new task.
Revised: Finally, Regularization based methods draw inspiration from Bayesian learning, and can be seen as utilizing the posterior after learning a sequence of tasks as a prior to regularize learning of the new task.

Original: These methods differ from each other in how the prior and implicitly the posterior are parametrized and approximated.
Revised: These methods differ from each other in how the prior and implicitly the posterior are parametrized and approximated.

Original: For instance, Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) relies on a Gaussian approximation with a diagonal covariance, estimated using a Laplace approximation.
Revised: For instance, Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) relies on a Gaussian approximation with a diagonal covariance, estimated using a Laplace approximation.

Original: Variational Continual Learning (VCL) (Nguyen et al., 2017) learns directly the parameters of the Gaussian relying on the re-parametrization trick.
Revised: Variational Continual Learning (VCL) (Nguyen et al., 2017) learns directly the parameters of the Gaussian relying on the re-parametrization trick.

Original: (Ritter et al., 2018) achieved better approximation with a block-diagonal covariance.
Revised: (Ritter et al., 2018) achieved better approximation with a block-diagonal covariance.

Original: Synaptic Intelligence (SI) by
Revised: Synaptic Intelligence (SI) by

Original: Zenke et al.
Revised: Zenke et al.

Original: (2017) proposed to estimate the importance of parameters by the path length of the updates on the previous task, then discourage future changes on important parameters by a quadratic penalty.
Revised: (2017) proposed to estimate the importance of parameters by the path length of the updates on the previous task, then discourage future changes on important parameters by a quadratic penalty.

Original: While effective at preventing forgetting, the abovementioned methods either rely on knowledge of task boundaries or require task labels to select a sub-module for adaptation and prediction, hence cannot be directly applied in the task agnostic scenario considered here.
Revised: While effective at preventing forgetting, the abovementioned methods either rely on knowledge of task boundaries or require task labels to select a sub-module for adaptation and prediction, hence cannot be directly applied in the task agnostic scenario considered here.

Original: To circumvent this issue, (Kirkpatrick et al., 2017) used Forget-Me-Not (FMN) (Milan et al., 2016) to detect task boundaries and combined it with EWC to consolidate memory when task switches.
Revised: To circumvent this issue, (Kirkpatrick et al., 2017) used Forget-Me-Not (FMN) (Milan et al., 2016) to detect task boundaries and combined it with EWC to consolidate memory when task switches.

Original: However, FMN requires a generative model that computes exact data likelihood, which limits it from scaling to complex tasks.
Revised: However, FMN requires a generative model that computes exact data likelihood, which limits it from scaling to complex tasks.

Original: Bayesian Gradient Descent (BGD) (Zeno et al., 2018), as we discussed before, adopts the framework of online variational Bayes, and approximates the posterior with a diagonal Gaussian distribution.
Revised: Bayesian Gradient Descent (BGD) (Zeno et al., 2018), as we discussed before, adopts the framework of online variational Bayes, and approximates the posterior with a diagonal Gaussian distribution.

Original: More recently, (Aljundi et al., 2019) proposed a rehearsal-based method to select a ﬁnite number of data that are representative of all data seen so far.
Revised: More recently, (Aljundi et al., 2019) proposed a rehearsal-based method to select a ﬁnite number of data that are representative of all data seen so far.

Original: All of these methods assume that it is possible to learn one model that ﬁts all previous data, neglecting the scenario where different tasks may conﬂict each other, hence does not allow task-speciﬁc adaptations.
Revised: All of these methods assume that it is possible to learn one model that ﬁts all previous data, neglecting the scenario where different tasks may conﬂict each other, hence does not allow task-speciﬁc adaptations.

Original: Meta-learning , or learning to learn (Schmidhuber, 1987), trains a model on a distribution of tasks and focuses on its ability to quickly learn a new task at meta-testing time.
Revised: Meta-learning , or learning to learn (Schmidhuber, 1987), trains a model on a distribution of tasks and focuses on its ability to quickly learn a new task at meta-testing time.

Original: As with continual learning, different families of approaches exist for meta-learning.
Revised: As with continual learning, different families of approaches exist for meta-learning.

Original: Memory based methods Santoro et al.
Revised: Memory based methods Santoro et al.

Original: (2016) rely on a recurrent model (optimizer) such as
Revised: (2016) rely on a recurrent model (optimizer) such as

Original: LSTM to learn a history-dependent update function for the lower-level learner (optimizee).
Revised: LSTM to learn a history-dependent update function for the lower-level learner (optimizee).

Original: Andrychowicz et al.
Revised: Andrychowicz et al.

Original: (2016) trained an LSTM to replace the stochastic gradient descent algorithm by minimizing the sum of the losses of the optimizees on multiple prior tasks.
Revised: (2016) trained an LSTM to replace the stochastic gradient descent algorithm by minimizing the sum of the losses of the optimizees on multiple prior tasks.

Original: Ravi & Larochelle (2016) use an LSTM-based meta-learner to transform the gradient and loss of the base-learners on every new example to the ﬁnal updates of the model parameters.
Revised: Ravi & Larochelle (2016) use an LSTM-based meta-learner to transform the gradient and loss of the base-learners on every new example to the ﬁnal updates of the model parameters.

Original: Metric based methods learn an embedding space in which new tasks can be solved efﬁciently.
Revised: Metric based methods learn an embedding space in which new tasks can be solved efﬁciently.

Original: Koch (2015) trained siamese networks to tell if two images are similar by converting the distance between their feature embeddings to the probability of whether they are from the same class.
Revised: Koch (2015) trained siamese networks to tell if two images are similar by converting the distance between their feature embeddings to the probability of whether they are from the same class.

Original: Vinyals et al.
Revised: Vinyals et al.

Original: (2016) proposed the matching network to improve the embeddings of a test image and the support images by taking the entire support set as context input.
Revised: (2016) proposed the matching network to improve the embeddings of a test image and the support images by taking the entire support set as context input.

Original: The approaches discussed in Section 3.1 instead belong to the family of optimization based metalearning methods.
Revised: The approaches discussed in Section 3.1 instead belong to the family of optimization based metalearning methods.

Original: Beside the online meta learning work (Finn et al., 2019) discussed before, the most relevant work in this domain is from (Nagabandi et al., 2019), where they studied fast adaptation in a non-stationary environment by learning an ensemble of networks, one for each task.
Revised: Beside the online meta learning work (Finn et al., 2019) discussed before, the most relevant work in this domain is from (Nagabandi et al., 2019), where they studied fast adaptation in a non-stationary environment by learning an ensemble of networks, one for each task.

Original: Unlike our work, they used MAML for initialization of new networks in the ensemble instead of task inference.
Revised: Unlike our work, they used MAML for initialization of new networks in the ensemble instead of task inference.

Original: A drawback of this approach is that the size of the ensemble grows over time and is unbounded, hence can become memoryconsuming when there are many tasks.
Revised: A drawback of this approach is that the size of the ensemble grows over time and is unbounded, hence can become memoryconsuming when there are many tasks.

Original: 5. Experiments
Revised: 5. Experiments

Original: We design a series of experiments to thoroughly evaluate the effectiveness of the What & How framework, and compare it to BGD and other CL methods (EWC, online EWC, SI, LwF,
Revised: We design a series of experiments to thoroughly evaluate the effectiveness of the What & How framework, and compare it to BGD and other CL methods (EWC, online EWC, SI, LwF,

Original: DGR, DGR+Distill) implemented by van de Ven & Tolias (2019).
Revised: DGR, DGR+Distill) implemented by van de Ven & Tolias (2019).

Original: We also include the following baselines:
Revised: We also include the following baselines:

Original: None : the model is trained sequentially using Adam (Kingma & Ba, 2014) in the standard way without applying any continual learning method; Joint : all tasks seen so far are trained atthe same time, this scheme usually sets the upper bound for continual learning methods.
Revised: None : the model is trained sequentially using Adam (Kingma & Ba, 2014) in the standard way without applying any continual learning method; Joint : all tasks seen so far are trained at

Original: [Table]
Revised: [Table]

Revised: 88.88 ± 1.

Revised: the same time, this scheme usually sets the upper bound for continual learning methods.

Original: 5.1. Label-Permuted MNIST
Revised: 5.1. Label-Permuted MNIST

Original: In this experiment, we ﬁrst create a different permutation of 10 classes for every task, with which we shufﬂe the classes in the labels.
Revised: In this experiment, we ﬁrst create a different permutation of 10 classes for every task, with which we shufﬂe the classes in the labels.

Original: For instance, digit 0 might be the ﬁrst class in one task but the second class in another task.
Revised: For instance, digit 0 might be the ﬁrst class in one task but the second class in another task.

Original: The reason for this design is to ensure that a multi-task solution does not exist since the network has to map the same image to different labels for different tasks.
Revised: The reason for this design is to ensure that a multi-task solution does not exist since the network has to map the same image to different labels for different tasks.

Original: In this way, we can test whether our framework is able to quickly adapt its behavior according to the current context.
Revised: In this way, we can test whether our framework is able to quickly adapt its behavior according to the current context.

Original: Five tasks are created with this method and are presented sequentially foriterations each to an MLP with 2 hidden layers ofneurons.
Revised: Five tasks are created with this method and are presented sequentially foriterations each to an MLP with 2 hidden layers ofneurons.

Original: In each iteration, a mini-batch of 128 images is presented to the network.
Revised: In each iteration, a mini-batch of 128 images is presented to the network.

Original: Note that except BGD, None and our method, all the other baselines simply cannot be directly applied in the task agnostic scenario, so we provide the necessary task information for these methods in order to perform the comparison.
Revised: Note that except BGD, None and our method, all the other baselines simply cannot be directly applied in the task agnostic scenario, so we provide the necessary task information for these methods in order to perform the comparison.

Original: Zero-shot vs. Few-shot Recall at the end of the entire learning process, we test the learner’s classiﬁcation accuracy of each task in two ways.
Revised: Zero-shot vs. Few-shot Recall at the end of the entire learning process, we test the learner’s classiﬁcation accuracy of each task in two ways.

Original: The ﬁrst way is to directly apply the ﬁnal model on the testing data without any adaptations, this corresponds to the zero-shot recall accuracy traditionally used in continual learning.
Revised: The ﬁrst way is to directly apply the ﬁnal model on the testing data without any adaptations, this corresponds to the zero-shot recall accuracy traditionally used in continual learning.

Original: For the W & H algorithm, this means we apply the model simply with the learned initialization.
Revised: For the W & H algorithm, this means we apply the model simply with the learned initialization.

Original: In the second way, we provide the ﬁnal model with a mini-batch of 128 images from the training set of each task as context data, then let the model take k =steps of gradient descent on the context data before it is tested on the testing data of that task.
Revised: In the second way, we provide the ﬁnal model with a mini-batch of 128 images from the training set of each task as context data, then let the model take k =steps of gradient descent on the context data before it is tested on the testing data of that task.

Original: We call this metric few-shot recall accuracy.
Revised: We call this metric few-shot recall accuracy.

Original:  Table 1 summarizes the performance of our method and other CL methods.
Revised: [Table] [Table] Table 1 summarizes the performance of our method and other CL methods.

Original: Note that in this experiment, it is impossible to achieve good performance with zero-shot recall [Table] [Table] [Table]
Revised: Note that in this experiment, it is impossible to achieve good performance with zero-shot recall since a multi-task solution does not exist.

Original: [Table]

Original: since a multi-task solution does not exist.

Original: Even the joint training scheme which is considered the upper bound for
Revised: Even the joint training scheme which is considered the upper bound for

Original: CL achieves very poor accuracies.
Revised: CL achieves very poor accuracies.

Original: In the few-shot setting, our framework signiﬁcantly outperforms the other baselines, even without access to any task information during training.
Revised: In the few-shot setting, our framework signiﬁcantly outperforms the other baselines, even without access to any task information during training.

Original: Interestingly, some baselines perform worse in the few-shot setting due to over-ﬁtting on the context data.
Revised: Interestingly, some baselines perform worse in the few-shot setting due to over-ﬁtting on the context data.

Original: 5.2. Permuted and Split MNIST
Revised: 5.2. Permuted and Split MNIST

Original: We also test the What & How framework on the standard
Revised: We also test the What & How framework on the standard

Original: CL benchmarks called Permuted MNIST (Goodfellow et al., 2013; Kirkpatrick et al., 2017) and Split MNIST (Zenke et al., 2017).
Revised: CL benchmarks called Permuted MNIST (Goodfellow et al., 2013; Kirkpatrick et al., 2017) and Split MNIST (Zenke et al., 2017).

Original: In these experiments, tasks are not conﬂicting with each other, so multi-task solutions do exist.
Revised: In these experiments, tasks are not conﬂicting with each other, so multi-task solutions do exist.

Original: van de Ven & Tolias (2019) described three scenarios for these experiments based on what task information are available at test time: task-incremental learning : models are always informed about which task is presented; domainincremental learning : task ID is not provided at test time; class-incremental learning : task ID is not provided and should be inferred at test time.
Revised: van de Ven & Tolias (2019) described three scenarios for these experiments based on what task information are available at test time: task-incremental learning : models are always informed about which task is presented; domainincremental learning : task ID is not provided at test time; class-incremental learning : task ID is not provided and should be inferred at test time.

Original: However, they assumed that during training there are clear and well-deﬁned task boundaries available for the learner.
Revised: However, they assumed that during training there are clear and well-deﬁned task boundaries available for the learner.

Original: Since our focus is task agnostic CL during training, we only consider the domainincremental scenario at test time, because in the other two scenarios, the task boundaries are anyways available during training (in the class-incremental case, a class corresponds to a task, which can be simply detected from the labels), it is not necessary to apply a task-agnostic method.
Revised: Since our focus is task agnostic CL during training, we only consider the domainincremental scenario at test time, because in the other two scenarios, the task boundaries are anyways available during training (in the class-incremental case, a class corresponds to a task, which can be simply detected from the labels), it is not necessary to apply a task-agnostic method.

Original: In the permuted MNIST protocol, a new task is created by shufﬂing the pixels of all images in MNIST by a ﬁxed permutation.
Revised: In the permuted MNIST protocol, a new task is created by shufﬂing the pixels of all images in MNIST by a ﬁxed permutation.

Original: We present 10 such tasks sequentially foriterations each.
Revised: We present 10 such tasks sequentially foriterations each.

Original: After all tasks are learned, the network has to predict the digit from an image without knowing the permutation.
Revised: After all tasks are learned, the network has to predict the digit from an image without knowing the permutation.

Original: For the split MNIST protocol, the original
Revised: For the split MNIST protocol, the original

Original: MNIST dataset is divided into 5 subsets with 2 digits each.
Revised: MNIST dataset is divided into 5 subsets with 2 digits each.

Original: We present one of these subsets at a time for 2000 iterations.
Revised: We present one of these subsets at a time for 2000 iterations.

Original: At the end of the training, the network has to predict if an image was the ﬁrst class or the second class in its subset, without knowing which subset the image is from.
Revised: At the end of the training, the network has to predict if an image was the ﬁrst class or the second class in its subset, without knowing which subset the image is from.

Original: To be comparable with the results from van de Ven & Tolias (2019), we use exactly the same network architecture, experiment setup and hyper-parameters for these two experiments.
Revised: To be comparable with the results from van de Ven & Tolias (2019), we use exactly the same network architecture, experiment setup and hyper-parameters for these two experiments.

Original: The results of these two experiments are displayed in Table 2 and Table 3.
Revised: The results of these two experiments are displayed in Table 2 and Table 3.

Original: Again, BGD and our method do not have access to any task information , while the other methods cannot be directly applied without task information.
Revised: Again, BGD and our method do not have access to any task information , while the other methods cannot be directly applied without task information.

Original: For our method, we use the previous mini-batch as context data during training, and at testing time, the learned initialization was directly used without any context-dependent adaption, it can be seen from the tables that our method performs the same as BGD, which also has similar performance to the regularization-based task-aware methods.
Revised: For our method, we use the previous mini-batch as context data during training, and at testing time, the learned initialization was directly used without any context-dependent adaption, it can be seen from the tables that our method performs the same as BGD, which also has similar performance to the regularization-based task-aware methods.

Original: This means our framework is also applicable when multi-task solutions exist.
Revised: This means our framework is also applicable when multi-task solutions exist.

Original: 5.3. Sine Regression
Revised: 5.3. Sine Regression

Original: One desideratum of continual learning is the ability of forward transfer .
Revised: One desideratum of continual learning is the ability of forward transfer .

Original: Lopez-Paz et al.
Revised: Lopez-Paz et al.

Original: (2017) deﬁned a metric for forward transfer based on a model’s “zero-shot” performance on a future task.
Revised: (2017) deﬁned a metric for forward transfer based on a model’s “zero-shot” performance on a future task.

Original: We generalize the concept of forward transfer to the “few-shot” setting: positive forward transfer should allow a model to learn faster on a future task similar to the ones it has learned.
Revised: We generalize the concept of forward transfer to the “few-shot” setting: positive forward transfer should allow a model to learn faster on a future task similar to the ones it has learned.

Original: This deﬁnition coincides with generalization in meta-learning, which refers to how fast a meta-learner can learn a new task at meta-test time.
Revised: This deﬁnition coincides with generalization in meta-learning, which refers to how fast a meta-learner can learn a new task at meta-test time.

Original: We show that our method is capable of forward transfer by comparing it to Reptile, BGD and SGD on the sine regression tasks commonly used in meta learning literature.
Revised: We show that our method is capable of forward transfer by comparing it to Reptile, BGD and SGD on the sine regression tasks commonly used in meta learning literature.

Original: We randomly generate 200 different sine curves and present them sequentially to the learners for 100 iterations each.
Revised: We randomly generate 200 different sine curves and present them sequentially to the learners for 100 iterations each.

Original: At the end of the learning process, we evaluate the few-shot performance (mean squared error) of all learners on thesesine curves as well as 200 new sine curves it has not seen before .
Revised: At the end of the learning process, we evaluate the few-shot performance (mean squared error) of all learners on thesesine curves as well as 200 new sine curves it has not seen before .

Original: In particular, the learners are evaluated after they takes k = 5 steps of SGD on a mini-batch of 64 context data.
Revised: In particular, the learners are evaluated after they takes k = 5 steps of SGD on a mini-batch of 64 context data.

Original: Table 4 summarizes the results of this experiment.
Revised: Table 4 summarizes the results of this experiment.

Original: The other baselines perform poorly due to catastrophic forgetting and no multi-task solution.
Revised: The other baselines perform poorly due to catastrophic forgetting and no multi-task solution.

Original: Figure 2 visualizes the predictions of different learners on the ﬁrst seen and a unseen sine curve at the end of the entire training procedure.
Revised: Figure 2 visualizes the predictions of different learners on the ﬁrst seen and a unseen sine curve at the end of the entire training procedure.

Original: 5.4. Continual GAN
Revised: 5.4. Continual GAN

Original: Finally, we apply our framework in the setting of GAN training, where the ultimate goal is to ﬁnd the optimal parameters θ ∗ G for a generator G ( z ; θ G ) by optimizing a minimax objective (e.g. Goodfellow et al., 2014;
Revised: Finally, we apply our framework in the setting of GAN training, where the ultimate goal is to ﬁnd the optimal parameters θ ∗ G for a generator G ( z ; θ G ) by optimizing a minimax objective (e.g. Goodfellow et al., 2014;

Original: Metz et al., 2016):
Revised: Metz et al., 2016):

Original: [Equation]
Revised: [Equation]

Original: where θ ∗ D ( θ G ) = argmax θ D l ( θ G , θ D ) and l ( θ G , θ D ) =
Revised: where θ ∗ D ( θ G ) = argmax θ D l [Equation] G , θ D ) and l ( θ G , θ D ) =

Original: E x ∼ p data ( x ) [log( D ( x ; θ D ))]
Revised: E x ∼ p data ( x ) [log( D ( x ; θ D ))]

Original: +
Revised: +

Original: [Equation]

Original:  ∼N [Equation] [Equation] 1) [log
Revised: E z ∼N (0 , 1) [log( −

Original: [Equation]

Original: D ( G ( z ; θ G ); θ D ))] .
Revised: D ( G ( z ; θ G ); θ D ))] .

Original: Following our discussion in the Section 1, continually learning D ( · ; θ D ) is impossible, because at different moments t and t (cid:48) during training, the generator distributions may be different ( p G (cid:54)
Revised: Following our discussion in the Section 1, continually learning D ( · ; θ D ) is impossible, because at different moments t and t (cid:48) during training, the generator distributions may be different ( p G (cid:54)

Original: =
Revised: =

Original: p G (cid:48) ), hence the corresponding optimal discriminator parameters cannot be the same: θ ∗ D ( θ G ) (cid:54)
Revised: p G (cid:48) ), hence the corresponding optimal discriminator parameters cannot be the same: θ ∗ D ( θ G ) (cid:54)

Original: = θ ∗ D ( θ G (cid:48) ) .
Revised: = θ ∗ D ( θ G (cid:48) ) .

Original: In other words, the tasks for the discriminator at time t and t (cid:48) are conﬂicting with each other.
Revised: In other words, the tasks for the discriminator at time t and t (cid:48) are conﬂicting with each other.

Original: Adopting the What &
Revised: Adopting the What &

Original: How framework, we focus on continually learning a model for θ ∗ D ( · ) in Eq.8 instead of the discriminator D ( · ) .
Revised: How framework, we focus on continually learning a model for θ ∗ D ( · ) in Eq.8 instead of the discriminator D ( · ) .

Original: This model corresponds to our What encoder: given a set of data points D G :=
Revised: This model corresponds to our What encoder: given a set of data points D G :=

Original: { G ( z n ; θ G ) } n sampled from the current generator as context data, the What encoder is trained to approximate the optimal discriminator parameters θ ∗ D ( θ G )
Revised: { G ( z n ; θ G ) } n sampled from the current generator as context data, the What encoder is trained to ap-

Original: ≈
Revised: [Table]

Original: F What ( D  G; φ ) by k steps of inner loop updates on the con-
Revised: proximate the optimal discriminator parameters θ ∗ D ( θ G ) ≈

Original: [Table]
Revised: [Table]

Original: text data, where the meta parameter φ corresponding to an initialization of the discriminator φ := θ init D is learned by
Revised: G ; φ ) by k steps of inner loop updates on the context data, where the meta parameter φ corresponding to an initialization of the discriminator φ := θ init D is learned by

Original: Alg.1.
Revised: Alg.1.

Original: The generator-speciﬁc discriminator returned by the
Revised: The generator-speciﬁc discriminator returned by the

Original: How decoder F How ◦F What ( D G ; φ ) :
Revised: How decoder F How ◦F What ( D G ; φ ) :

Original: = D ( · ; F What ( D G ; φ )) is used to compute the loss of θ G and to update the current generator 1 .
Revised: = D ( · ; F What ( D G ; φ )) is used to compute the loss of θ G and to update the current generator 1 .

Original: This way, the conﬂict at the level of θ D is resolved at the level of φ : it is possible, in theory, to ﬁnd a single φ that maximizes both l ( θ G , F What ( D G ; φ )) and l ( θ G (cid:48) , F What ( D G (cid:48) ; φ )) , even when θ G (cid:54)
Revised: This way, the conﬂict at the level of θ D is resolved at the level of φ : it is possible, in theory, to ﬁnd a single φ that maximizes both l ( θ G , F What ( D G ; φ )) and l ( θ G (cid:48) , F What ( D G (cid:48) ; φ )) , even when θ G (cid:54)

Original: = θ G (cid:48) .
Revised: = θ G (cid:48) .

Original: Thanh-Tung & Tran (2019) and Liang et al.
Revised: Thanh-Tung & Tran (2019) and Liang et al.

Original: (2019) showed that a notorious problem for GAN training called mode collapse (Che et al., 2016) is interrelated with catastrophic forgetting and can cause the training process to never converge, since the generator is always optimized to revisit a mode that the discriminator has forgotten.
Revised: (2019) showed that a notorious problem for GAN training called mode collapse (Che et al., 2016) is interrelated with catastrophic forgetting and can cause the training process to never converge, since the generator is always optimized to revisit a mode that the discriminator has forgotten.

Original: Therefore, overcoming catastrophic forgetting problem in the discriminator should be able to break this mode revisiting cycles and reduce mode collapse, as shown in the experiments below.
Revised: Therefore, overcoming catastrophic forgetting problem in the discriminator should be able to break this mode revisiting cycles and reduce mode collapse, as shown in the experiments below.

Original: 2D Mixture of Gaussian To directly visualize the effect of our method, we ﬁrst apply it to train a simple GAN from synthetic data generated from a mixture of 8 Gaussian distributions on 2D space.
Revised: 2D Mixture of Gaussian To directly visualize the effect of our method, we ﬁrst apply it to train a simple GAN from synthetic data generated from a mixture of 8 Gaussian distributions on 2D space.

Original: The network architecture and experiment setup are exactly the same as in (Metz et al., 2016).
Revised: The network architecture and experiment setup are exactly the same as in (Metz et al., 2016).

Original: We ﬁrst train a vanilla GAN with standard techniques on this dataset, the ﬁrst row (Vanilla GAN) of Figure 3 shows that it enters a non-convergent cycle of revisiting the modes.
Revised: We ﬁrst train a vanilla GAN with standard techniques on this dataset, the ﬁrst row (Vanilla GAN) of Figure 3 shows that it enters a non-convergent cycle of revisiting the modes.

Original: We then apply the What & How method (with k = 3 steps in the inner loop) to the discriminator while keeping the rest of the experiment setup and hyper-parameters the same.
Revised: We then apply the What & How method (with k = 3 steps in the inner loop) to the discriminator while keeping the rest of the experiment setup and hyper-parameters the same.

Original: As can be seen in the second row (WHGAN) of Figure 3, although mode collapse also occurs at the beginning of the training  process, our networks can avoid the repeating cycle, and eventually converge to a distribution that covers all modes.
Revised: As can be seen in the second row (WHGAN) of Figure 3, although mode collapse also occurs at the beginning of the training [Table] process, our networks can avoid the repeating cycle

Original: Metrics
Revised: [Table] and eventually converge to a distribution that covers all modes

Original: [Table]
Revised: [Table]

Original: DCGAN on CIFAR10 In this experiment, we compare the differences between a DCGAN (Radford et al., 2015) trained with and without our method on the CIFAR dataset (Krizhevsky et al., 2009).
Revised: DCGAN on CIFAR10 In this experiment, we compare the differences between a DCGAN (Radford et al., 2015) trained with and without our method on the CIFAR dataset (Krizhevsky et al., 2009).

Original: Since it is hard to visualize mode collapse for high dimensional image data, we use two bin-based metrics called NDB and JSD (Richardson & Weiss, 2018) to evaluate the resulting GANs.
Revised: Since it is hard to visualize mode collapse for high dimensional image data, we use two bin-based metrics called NDB and JSD (Richardson & Weiss, 2018) to evaluate the resulting GANs.

Original: To compute NDB, the real samples are ﬁrst clustered by K-means into K = 200 bins, which can be considered as modes of the data distribution.
Revised: To compute NDB, the real samples are ﬁrst clustered by K-means into K = 200 bins, which can be considered as modes of the data distribution.

Original: Then N = 50000 images sampled from the generator are assigned to their nearest bins.
Revised: Then N = 50000 images sampled from the generator are assigned to their nearest bins.

Original: For each bin, a two-sample test is performed to decide if the synthesized samples are statistically different from the real samples.
Revised: For each bin, a two-sample test is performed to decide if the synthesized samples are statistically different from the real samples.

Original: NDB is then simply the number of statistically different bins and JSD is the Jensen-Shannon divergence between the real distribution and the generator distribution over these bins.
Revised: NDB is then simply the number of statistically different bins and JSD is the Jensen-Shannon divergence between the real distribution and the generator distribution over these bins.

Original: Lower NDB and JSD scores imply more similarity between two distributions, and hence less mode collapse.
Revised: Lower NDB and JSD scores imply more similarity between two distributions, and hence less mode collapse.

Original: In addition, we also evaluate the resulting GANs with the Inception Score (IS) (Salimans et al., 2016) and the Fr´echet Inception Distance (FID) (Heusel et al., 2017), which are metrics based on the image features extracted by the Inception Network (Szegedy et al., 2015).
Revised: In addition, we also evaluate the resulting GANs with the Inception Score (IS) (Salimans et al., 2016) and the Fr´echet Inception Distance (FID) (Heusel et al., 2017), which are metrics based on the image features extracted by the Inception Network (Szegedy et al., 2015).

Original: Higher IS and lower FID indicate better quality of the generated images.
Revised: Higher IS and lower FID indicate better quality of the generated images.

Original: Table 5 compares the performance of the original DCGAN and one trained with our framework (WHGAN).
Revised: Table 5 compares the performance of the original DCGAN and one trained with our framework (WHGAN).

Original: Again, we keep the network architecture and all hyper-parameters the same, except that the discriminator in WHGAN is trained with the What & How method.
Revised: Again, we keep the network architecture and all hyper-parameters the same, except that the discriminator in WHGAN is trained with the What & How method.

Original: In both cases, we train the networks for 50000 iterations with a mini-batch size 64.
Revised: In both cases, we train the networks for 50000 iterations with a mini-batch size 64.

Original: For our method, k = 3 steps are used in the inner loop of the
Revised: For our method, k = 3 steps are used in the inner loop of the

Original: What encoder.
Revised: What encoder.

Original: The results show that WHGAN achieved signiﬁcantly lower NDB and JSD while maintaining the same image quality.
Revised: The results show that WHGAN achieved signiﬁcantly lower NDB and JSD while maintaining the same image quality.

Original: 6. Conclusions
Revised: 6. Conclusions

Original: In this work, we showed that when a multi-task solution does not exist, catastrophic forgetting is inevitable.
Revised: In this work, we showed that when a multi-task solution does not exist, catastrophic forgetting is inevitable.

Original: A framework that can infer task information explicitly from context data was proposed to resolve this problem.
Revised: A framework that can infer task information explicitly from context data was proposed to resolve this problem.

Original: The framework separates the inference process into two components: one for representing What task is presented, and the other for describing How to solve the given task.
Revised: The framework separates the inference process into two components: one for representing What task is presented, and the other for describing How to solve the given task.

Original: In addition, our framework uniﬁes many meta learning methods and establishes a connection between continual learning and meta learning, leveraging the advantages of both.
Revised: In addition, our framework uniﬁes many meta learning methods and establishes a connection between continual learning and meta learning, leveraging the advantages of both.

Original: From the meta learning perspective, our framework addresses the continual meta learning problem by applying CL techniques on the meta variables, therefore allowing meta knowledge to accumulate over an extended period; from the continual learning perspective, our framework addresses the task agnostic continual learning problem by explicitly inferring the task when the task information is not available and a multi-task solution does not exist.
Revised: From the meta learning perspective, our framework addresses the continual meta learning problem by applying CL techniques on the meta variables, therefore allowing meta knowledge to accumulate over an extended period; from the continual learning perspective, our framework addresses the task agnostic continual learning problem by explicitly inferring the task when the task information is not available and a multi-task solution does not exist.

Original: This allows us to shift the focus of continual learning from less forgetting to faster remembering, given the right context.
Revised: This allows us to shift the focus of continual learning from less forgetting to faster remembering, given the right context.