Original: L EARNING TO L EARN WITHOUT F ORGETTING B Y M AXIMIZING T RANSFER AND M INIMIZING I NTERFERENCE
Revised: L EARNING TO L EARN WITHOUT F ORGETTING B Y M AXIMIZING T RANSFER AND M INIMIZING I NTERFERENCE

Original: A BSTRACT
Revised: A BSTRACT

Original: Lack of performance when it comes to continual learning over non-stationary distributions of data remains a major challenge in scaling neural network learning to more human realistic settings.
Revised: Lack of performance when it comes to continual learning over non-stationary distributions of data remains a major challenge in scaling neural network learning to more human realistic settings.

Original: In this work we propose a new conceptualization of the continual learning problem in terms of a temporally symmetric trade-off between transfer and interference that can be optimized by enforcing gradient alignment across examples.
Revised: In this work we propose a new conceptualization of the continual learning problem in terms of a temporally symmetric trade-off between transfer and interference that can be optimized by enforcing gradient alignment across examples.

Original: We then propose a new algorithm, Meta-Experience Replay (MER), that directly exploits this view by combining experience replay with optimization based meta-learning.
Revised: We then propose a new algorithm, Meta-Experience Replay (MER), that directly exploits this view by combining experience replay with optimization based meta-learning.

Original: This method learns parameters that make interference based on future gradients less likely and transfer based on future gradients more likely.
Revised: This method learns parameters that make interference based on future gradients less likely and transfer based on future gradients more likely.

Original: 1 We conduct experiments across continual lifelong supervised learning benchmarks and non-stationary reinforcement learning environments demonstrating that our approach consistently outperforms recently proposed baselines for continual learning.
Revised: 1 We conduct experiments across continual lifelong supervised learning benchmarks and non-stationary reinforcement learning environments demonstrating that our approach consistently outperforms recently proposed baselines for continual learning.

Original: Our experiments show that the gap between the performance of MER and baseline algorithms grows both as the environment gets more non-stationary and as the fraction of the total experiences stored gets smaller
Revised: Our experiments show that the gap between the performance of MER and baseline algorithms grows both as the environment gets more non-stationary and as the fraction of the total experiences stored gets smaller.

Original:  S OLVING THE C ONTINUAL L EARNING P ROBLEM
Revised:  S OLVING THE C ONTINUAL L EARNING P ROBLEM

Original: A long-held goal of AI is to build agents capable of operating autonomously for long periods.
Revised: A long-held goal of AI is to build agents capable of operating autonomously for long periods.

Original: Such agents must incrementally learn and adapt to a changing environment while maintaining memories of what they have learned before, a setting known as lifelong learning (Thrun, 1994; 1996).
Revised: Such agents must incrementally learn and adapt to a changing environment while maintaining memories of what they have learned before, a setting known as lifelong learning (Thrun, 1994; 1996).

Original: In this paper we explore a variant called continual learning (Ring, 1994).
Revised: In this paper we explore a variant called continual learning (Ring, 1994).

Original: In continual learning we assume that the learner is exposed to a sequence of tasks, where each task is a sequence of experiences from the same distribution (see Appendix A for details).
Revised: In continual learning we assume that the learner is exposed to a sequence of tasks where each task is a sequence of experiences from the same distribution (see Appendix A for details).

Original: We would like to develop a solution in this setting by discovering notions of tasks without supervision while learning incrementally after every experience.
Revised: We would like to develop a solution in this setting by discovering notions of tasks without supervision while learning incrementally after every experience.

Original: This is challenging because in standard ofﬂine single task and multi-task learning (Caruana, 1997) it is implicitly assumed that the data is drawn from an i.i.d.
Revised: This is challenging because in standard ofﬂine single task and multi-task learning (Caruana, 1997) it is implicitly assumed that the data is drawn from an i.i.d.

Original: stationary distribution.
Revised: stationary distribution.

Original: Unfortunately, neural networks tend to struggle whenever this is not the case (Goodrich, 2015).
Revised: Unfortunately, neural networks tend to struggle whenever this is not the case (Goodrich, 2015).

Original: Over the years, solutions to the continual learning problem have been largely driven by prominent conceptualizations of the issues faced by neural networks.
Revised: Over the years, solutions to the continual learning problem have been largely driven by prominent conceptualizations of the issues faced by neural networks.

Original: One popular view is catastrophic forgetting (interference) (McCloskey & Cohen, 1989), in which the primary concern is the lack of stability in neural networks, and the main solution is to limit the extent of weight sharing across experiences by focusing on preserving past knowledge (Kirkpatrick et al., 2017; Zenke et al., 2017; Lee et al., 2017).
Revised: One popular view is catastrophic forgetting (interference) (McCloskey & Cohen, 1989), in which the primary concern is the lack of stability in neural networks, and the main solution is to limit the extent of weight sharing across experiences by focusing on preserving past knowledge (Kirkpatrick et al., 2017; Zenke et al., 2017; Lee et al., 2017).

Original: Another popular and more complex conceptualization is the stability-plasticity dilemma (Carpenter & Grossberg, 1987).
Revised: Another popular and more complex conceptualization is the stability-plasticity dilemma (Carpenter & Grossberg, 1987).

Original: In this view, the primary concern is the balance between network
Revised: In this view, the primary concern is the balance between network

Original: Stability – Plasticity Dilemma Stability – Plasticity Dilemma
Revised: Stability – Plasticity Dilemma Stability – Plasticity Dilemma

Original: A. Transfer – Interference Trade-o ﬀ
Revised: A. Transfer – Interference Trade-o ﬀ

Original: Transfer
Revised: Transfer

Original: Old Learning
Revised: Old Learning

Original: Current Learning
Revised: Current Learning

Original: Future Learning
Revised: Future Learning

Original: Sharing Sharing
Revised: Sharing Sharing

Original: B. Transfer
Revised: B. Transfer

Original: C. Interference ∂ L i ∂ θ ∂ L j ∂ θ ∂ L i ∂ θ ∂ L j ∂ θ
Revised: C. Interference ∂ L i ∂ θ ∂ L j ∂ θ ∂ L i ∂ θ ∂ L j ∂ θ

Original: FUTURE PAST
Revised: FUTURE PAST

Original: Transfer
Revised: Transfer

Original: Interference
Revised: Interference

Original: Interference stability (to preserve past knowledge) and plasticity (to rapidly learn the current experience).
Revised: Interference stability (to preserve past knowledge) and plasticity (to rapidly learn the current experience).

Original: For example, these techniques focus on balancing limited weight sharing with some mechanism to ensure fast learning (Li & Hoiem, 2016; Riemer et al., 2016a; Lopez-Paz & Ranzato, 2017; Rosenbaum et al., 2018; Lee et al., 2018;
Revised: For example, these techniques focus on balancing limited weight sharing with some mechanism to ensure fast learning (Li & Hoiem, 2016; Riemer et al., 2016a; Lopez-Paz & Ranzato, 2017; Rosenbaum et al., 2018; Lee et al., 2018;

Original: Serr ` a et al., 2018).
Revised: Serr ` a et al., 2018).

Original: In this paper, we extend this view by noting that for continual learning over an unbounded number of distributions, we need to consider weight sharing and the stability-plasticity trade-off in both the forward and backward directions in time (Figure 1A).
Revised: In this paper, we extend this view by noting that for continual learning over an unbounded number of distributions, we need to consider weight sharing and the stability-plasticity trade-off in both the forward and backward directions in time (Figure 1A).

Original: The transfer-interference trade-off proposed in this paper (section 2) presents a novel perspective on the goal of gradient alignment for the continual learning problem.
Revised: The transfer-interference trade-off proposed in this paper (section 2) presents a novel perspective on the goal of gradient alignment for the continual learning problem.

Original: This is right at the heart of the problem as these gradients are the update steps for SGD based optimizers during learning and there is a clear connection between gradients angles and managing the extent of weight sharing.
Revised: This is right at the heart of the problem as these gradients are the update steps for SGD based optimizers during learning and there is a clear connection between gradients angles and managing the extent of weight sharing.

Original: The key difference in perspective with past conceptualizations of continual learning is that we are not just concerned with current transfer and interference with respect to past examples, but also with the dynamics of transfer and interference moving forward as we learn.
Revised: The key difference in perspective with past conceptualizations of continual learning is that we are not just concerned with current transfer and interference with respect to past examples, but also with the dynamics of transfer and interference moving forward as we learn.

Original: Other approaches have certainly explored operational notions of transfer and interference in forward and backward directions (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018), the link to weight sharing (French, 1991; Ajemian et al., 2013), and the idea of inﬂuencing gradient alignment for continual learning before (Lopez-Paz & Ranzato, 2017).
Revised: Other approaches have certainly explored operational notions of transfer and interference in forward and backward directions (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018), the link to weight sharing (French, 1991; Ajemian et al., 2013), and the idea of inﬂuencing gradient alignment for continual learning before (Lopez-Paz & Ranzato, 2017).

Original: However, in past work, ad hoc changes have been made to the dynamics of weight sharing based on current learning and past learning without formulating a consistent theory about the optimal weight sharing dynamics.
Revised: However, in past work, ad hoc changes have been made to the dynamics of weight sharing based on current learning and past learning without formulating a consistent theory about the optimal weight sharing dynamics.

Original: This new view of the problem leads to a natural meta-learning (Schmidhuber, 1987) perspective on continual learning: we would like to learn to modify our learning to affect the dynamics of transfer and interference in a general sense.
Revised: This new view of the problem leads to a natural meta-learning (Schmidhuber, 1987) perspective on continual learning: we would like to learn to modify our learning to affect the dynamics of transfer and interference in a general sense.

Original: To the extent that our meta-learning into the future generalizes, this should make it easier for our model to perform continual learning in non-stationary settings.
Revised: To the extent that our meta-learning into the future generalizes, this should make it easier for our model to perform continual learning in non-stationary settings.

Original: We achieve this by building off past work on experience replay (Murre, 1992; Lin, 1992; Robins, 1995) that has been a mainstay for solving non-stationary problems with neural networks.
Revised: We achieve this by building off past work on experience replay (Murre, 1992; Lin, 1992; Robins, 1995) that has been a mainstay for solving non-stationary problems with neural networks.

Original: We propose a novel meta-experience replay (MER) algorithm that combines experience replay with optimization based meta-learning (section 3) as a ﬁrst step towards modeling this perspective.
Revised: We propose a novel meta-experience replay (MER) algorithm that combines experience replay with optimization based meta-learning (section 3) as a ﬁrst step towards modeling this perspective.

Original: Moreover, our experiments (sections 4, 5, and 6), conﬁrm our theory.
Revised: Moreover, our experiments (sections 4, 5, and 6), conﬁrm our theory.

Original: MER shows great promise across a variety of supervised continual learning and continual reinforcement learning settings.
Revised: MER shows great promise across a variety of supervised continual learning and continual reinforcement learning settings.

Original: Critically, our approach is not reliant on any provided notion of tasks and in most of the settings we explore we must detect the concept of tasks without supervision.
Revised: Critically, our approach is not reliant on any provided notion of tasks and in most of the settings we explore we must detect the concept of tasks without supervision.

Original: See Appendix B for a more detailed positioning with respect to related research.
Revised: See Appendix B for a more detailed positioning with respect to related research.

Original:  T HE T RANSFER -I NTERFERENCE T RADE - OFF FOR C ONTINUAL L EARNING
Revised:  T HE T RANSFER -I NTERFERENCE T RADE - OFF FOR C ONTINUAL L EARNING

Original: At an instant in time with parameters θ and loss L , we can deﬁne 2 operational measures of transfer and interference between two arbitrary distinct examples ( x i , y i ) and ( x j , y j ) while training withlearning
Revised: At an instant in time with parameters θ and loss L , we can deﬁne 2 operational measures of transfer and interference between two arbitrary distinct examples ( x i , y i ) and ( x j , y j ) while training with

Original: SGD.
Revised: SGD.

Original: Transfer occurs when:
Revised: Transfer occurs when:

Original: [Equation]
Revised: [Equation]

Original: where · is the dot product operator.
Revised: where · is the dot product operator.

Original: This implies that learning example i will without repetition improve performance on example j and vice versa (Figure 1B).
Revised: This implies that learning example i will without repetition improve performance on example j and vice versa (Figure 1B).

Original: Interference occurs when:
Revised: Interference occurs when:

Original: [Equation]
Revised: [Equation]

Original: Here, in contrast, learning example i will lead to unlearning (i.e. forgetting) of example j and vice versa (Figure 1C).
Revised: Here, in contrast, learning example i will lead to unlearning (i.e. forgetting) of example j and vice versa (Figure 1C).

Original: 3 There is weight sharing between i and j when they are learned using an overlapping set of parameters.
Revised: 3 There is weight sharing between i and j when they are learned using an overlapping set of parameters.

Original: So, potential for transfer is maximized when weight sharing is maximized while potential for interference is minimized when weight sharing is minimized (Appendix C).
Revised: So, potential for transfer is maximized when weight sharing is maximized while potential for interference is minimized when weight sharing is minimized (Appendix C).

Original: Past solutions for the stability-plasticity dilemma in continual learning operate in a simpliﬁed temporal context where learning is divided into two phases: all past experiences are lumped together as old memories and the data currently being learned qualiﬁes as new learning .
Revised: Past solutions for the stability-plasticity dilemma in continual learning operate in a simpliﬁed temporal context where learning is divided into two phases: all past experiences are lumped together as old memories and the data currently being learned qualiﬁes as new learning .

Original: In this setting, the goal is to simply minimize the interference projecting backward in time, which is generally achieved by reducing the degree of weight sharing explicitly or implicitly.
Revised: In this setting, the goal is to simply minimize the interference projecting backward in time, which is generally achieved by reducing the degree of weight sharing explicitly or implicitly.

Original: In Appendix D we explain how our baseline approaches (Kirkpatrick et al., 2017; Lopez-Paz & Ranzato, 2017)
Revised: In Appendix D we explain how our baseline approaches (Kirkpatrick et al., 2017; Lopez-Paz & Ranzato, 2017)

Original: ﬁt within this paradigm.
Revised: ﬁt within this paradigm.

Original: The important issue with this perspective, however, is that the system still has learning to do and what the future may bring is largely unknown.
Revised: The important issue with this perspective, however, is that the system still has learning to do and what the future may bring is largely unknown.

Original: This makes it incumbent upon us to do nothing to potentially undermine the networks ability to effectively learn in an uncertain future.
Revised: This makes it incumbent upon us to do nothing to potentially undermine the networks ability to effectively learn in an uncertain future.

Original: This consideration makes us extend the temporal horizon of the stability-plasticity problem forward, turning it, more generally, into a continual learning problem that we label as solving the Transfer-Interference
Revised: This consideration makes us extend the temporal horizon of the stability-plasticity problem forward, turning it, more generally, into a continual learning problem that we label as solving the Transfer-Interference

Original: Trade-off (Figure 1A).
Revised: Trade-off (Figure 1A).

Original: Speciﬁcally, it is important not only to reduce backward interference from our current point in time, but we must do so in a manner that does not limit our ability to learn in the future.
Revised: Speciﬁcally, it is important not only to reduce backward interference from our current point in time, but we must do so in a manner that does not limit our ability to learn in the future.

Original: This more general perspective acknowledges a subtlety in the problem: the issue of gradient alignment and thus weight sharing across examples arises both backward and forward in time.
Revised: This more general perspective acknowledges a subtlety in the problem: the issue of gradient alignment and thus weight sharing across examples arises both backward and forward in time.

Original: With this temporally symmetric perspective, the transfer-interference trade-off becomes clear.
Revised: With this temporally symmetric perspective, the transfer-interference trade-off becomes clear.

Original: Here we propose a potential solution where we learn to learn in a way that promotes gradient alignment at each point in time.
Revised: Here we propose a potential solution where we learn to learn in a way that promotes gradient alignment at each point in time.

Original: The weight sharing across examples that enables transfer to improve future performance must not disrupt performance on what has come previously.
Revised: The weight sharing across examples that enables transfer to improve future performance must not disrupt performance on what has come previously.

Original: As such, our work adopts a meta-learning perspective on the continual learning problem.
Revised: As such, our work adopts a meta-learning perspective on the continual learning problem.

Original: We would like to learn to learn each example in a way that generalizes to other examples from the overall distribution.
Revised: We would like to learn to learn each example in a way that generalizes to other examples from the overall distribution.

Original:  A S YSTEM FOR L EARNING TO L EARN WITHOUT F ORGETTING
Revised:  A S YSTEM FOR L EARNING TO L EARN WITHOUT F ORGETTING

Original: In typical ofﬂine supervised learning, we can express our optimization objective over the stationary distribution of x, y pairs within the dataset D :
Revised: In typical ofﬂine supervised learning, we can express our optimization objective over the stationary distribution of x, y pairs within the dataset D :

Original: [Equation]
Revised: [Equation]

Original: where L is the loss function, which can be selected to ﬁt the problem.
Revised: where L is the loss function, which can be selected to ﬁt the problem.

Original: If we would like to maximize transfer and minimize interference, we can imagine it would be useful to add an auxiliary loss to the objective to bias the learning process in that direction.
Revised: If we would like to maximize transfer and minimize interference, we can imagine it would be useful to add an auxiliary loss to the objective to bias the learning process in that direction.

Original: Considering equations 1 and 2, one obviously beneﬁcial choice would be to also directly consider the gradients with respect to the loss function evaluated at randomly chosen datapoints.
Revised: Considering equations 1 and 2, one obviously beneﬁcial choice would be to also directly consider the gradients with respect to the loss function evaluated at randomly chosen datapoints.

Original: If we could maximize the dot products between gradients at these different points, it would directly encourage the network to share parameters where gradient directions align and keep parameters separate where interference is caused by gradients in opposite directions.
Revised: If we could maximize the dot products between gradients at these different points, it would directly encourage the network to share parameters where gradient directions align and keep parameters separate where interference is caused by gradients in opposite directions.

Original: So, ideally we would like to optimize for the following objective 4 :
Revised: So, ideally we would like to optimize for the following objective 4 :

Original: [Equation]
Revised: [Equation]

Original: procedure T RAIN ( D, θ, α, β, γ, s, k ) M ← {} for t
Revised: procedure T RAIN ( D, θ, α, β, γ, s, k ) M ← {} for t

Original: = 1 , ..., T do for ( x, y ) in D t do //
Revised: = 1 , ..., T do for ( x, y ) in D t do //

Original: Draw batches from buffer:
Revised: Draw batches from buffer:

Original: B 1 , ..., B s ← sample ( x, y, s, k, M )
Revised: B 1 , ..., B s ← sample ( x, y, s, k, M )

Original: θ
Revised: θ

Original: A 0 ← θ for i = 1 , ..., s do θ Wi , 0 ← θ for j = 1 , ..., k do x c , y c ← B i [ j ] θ Wi,j ← SGD ( x c , y c , θ Wi,j − 1 , α ) end for //
Revised: A 0 ← θ for i = 1 , ..., s do θ Wi , 0 ← θ for j = 1 , ..., k do x c , y c ← B i [ j ] θ Wi,j ← SGD ( x c , y c , θ Wi,j − 1 , α ) end for //

Original: Within batch Reptile meta-update: θ ← θ Wi, 0 + β ( θ Wi,k − θ Wi, 0 ) θ
Revised: Within batch Reptile meta-update: θ ← θ Wi, 0 + β ( θ Wi,k − θ Wi, 0 ) θ

Original: Ai ← θ end for //
Revised: Ai ← θ end for //

Original: Across batch Reptile meta-update: θ ← θ A 0 + γ ( θ As − θ A 0 ) //
Revised: Across batch Reptile meta-update: θ ← θ A 0 + γ ( θ As − θ A 0 ) //

Original: Reservoir sampling memory update: M ← M ∪ { ( x, y ) } (algorithm 3) end for end for return θ, M end procedure where ( x i , y i ) and ( x j , y j ) are randomly sampled unique data points.
Revised: Reservoir sampling memory update: M ← M ∪ { ( x, y ) } (algorithm 3) end for end for return θ, M end procedure where ( x i , y i ) and ( x j , y j ) are randomly sampled unique data points.

Original: We will attempt to design a continual learning system that optimizes for this objective.
Revised: We will attempt to design a continual learning system that optimizes for this objective.

Original: However, there are multiple problems that must be addressed to implement this kind of learning process in practice.
Revised: However, there are multiple problems that must be addressed to implement this kind of learning process in practice.

Original: The ﬁrst problem is that continual learning deals with learning over a non-stationary stream of data.
Revised: The ﬁrst problem is that continual learning deals with learning over a non-stationary stream of data.

Original: We address this by implementing an experience replay module that augments online learning so that we can approximately optimize over the stationary distribution of all examples seen so far.
Revised: We address this by implementing an experience replay module that augments online learning so that we can approximately optimize over the stationary distribution of all examples seen so far.

Original: Another practical problem is that the gradients of this loss depend on the second derivative of the loss function, which is expensive to compute.
Revised: Another practical problem is that the gradients of this loss depend on the second derivative of the loss function, which is expensive to compute.

Original: We address this by indirectly approximating the objective to a ﬁrst order Taylor expansion using a meta-learning algorithm with minimal computational overhead.
Revised: We address this by indirectly approximating the objective to a ﬁrst order Taylor expansion using a meta-learning algorithm with minimal computational overhead.

Original: 3. E XPERIENCE R EPLAY
Revised: 3. E XPERIENCE R EPLAY

Original: Learning objective: The continual lifelong learning setting poses a challenge for the optimization of neural networks as examples come one by one in a non-stationary stream.
Revised: Learning objective: The continual lifelong learning setting poses a challenge for the optimization of neural networks as examples come one by one in a non-stationary stream.

Original: Instead, we would like our network to optimize over the stationary distribution of all examples seen so far.
Revised: Instead, we would like our network to optimize over the stationary distribution of all examples seen so far.

Original: Experience replay (Lin, 1992; Murre, 1992) is an old technique that remains a central component of deep learning systems attempting to learn in non-stationary settings, and we will adopt here conventions from recent work (Zhang & Sutton, 2017; Riemer et al., 2017b) leveraging this approach.
Revised: Experience replay (Lin, 1992; Murre, 1992) is an old technique that remains a central component of deep learning systems attempting to learn in non-stationary settings, and we will adopt here conventions from recent work (Zhang & Sutton, 2017; Riemer et al., 2017b) leveraging this approach.

Original: The central feature of experience replay is keeping a memory of examples seen M that is interleaved with the training of the current example with the goal of making training more stable.
Revised: The central feature of experience replay is keeping a memory of examples seen M that is interleaved with the training of the current example with the goal of making training more stable.

Original: As a result, experience replay approximates the objective in equation 3 to the extent that M approximates D :
Revised: As a result, experience replay approximates the objective in equation 3 to the extent that M approximates D :

Original: [Equation]
Revised: [Equation]

Original: M has a current size M size and maximum size M max .
Revised: M has a current size M size and maximum size M max .

Original: In our work, we update the buffer with reservoir sampling (Appendix F).
Revised: In our work, we update the buffer with reservoir sampling (Appendix F).

Original: This ensures that at every time-step the probability that any of the N examples seen has of being in the buffer is equal to M size /N .
Revised: This ensures that at every time-step the probability that any of the N examples seen has of being in the buffer is equal to M size /N .

Original: The content of the buffer resembles a stationary distribution over all examples seen to the extent that the items stored captures the variation of past examples.
Revised: The content of the buffer resembles a stationary distribution over all examples seen to the extent that the items stored captures the variation of past examples.

Original: Following the standard practice in ofﬂine learning, we train by randomly sampling a batch B from the distribution captured by M .
Revised: Following the standard practice in ofﬂine learning, we train by randomly sampling a batch B from the distribution captured by M .

Original: Prioritizing the current example: the variant of experience replay we explore differs from ofﬂine learning in that the current example has a special role ensuring that it is always interleaved with the examples sampled from the replay buffer.
Revised: Prioritizing the current example: the variant of experience replay we explore differs from ofﬂine learning in that the current example has a special role ensuring that it is always interleaved with the examples sampled from the replay buffer.

Original: This is because before we proceed to the next example, we want to make sure our algorithm has the ability to optimize for the current example (particularly if it is not added to the memory).
Revised: This is because before we proceed to the next example, we want to make sure our algorithm has the ability to optimize for the current example (particularly if it is not added to the memory).

Original: Over N examples seen
Revised: Over N examples seen

Original: , this still implies that we have trained with each example as the current example with probability per step of 1 /N .
Revised: , this still implies that we have trained with each example as the current example with probability per step of 1 /N .

Original: We provide algorithms further detailing how experience replay is used in this work in Appendix G (algorithms 4 and 5).
Revised: We provide algorithms further detailing how experience replay is used in this work in Appendix G (algorithms 4 and 5).

Original: Concerns about storing examples: Obviously, it is not scalable to store every experience seen in memory.
Revised: Concerns about storing examples: Obviously, it is not scalable to store every experience seen in memory.

Original: As such, in this work we focus on showing that we can achieve greater performance than baseline techniques when each approach is provided with only a small memory buffer
Revised: As such, in this work we focus on showing that we can achieve greater performance than baseline techniques when each approach is provided with only a small memory buffer

Original: . 3. C OMBINING E XPERIENCE R EPLAY WITH O PTIMIZATION B ASED M ETA -L EARNING
Revised: . 3. C OMBINING E XPERIENCE R EPLAY WITH O PTIMIZATION B ASED M ETA -L EARNING

Original: First order meta-learning: One of the most popular meta-learning algorithms to date is Model Agnostic Meta-Learning (MAML)
Revised: First order meta-learning: One of the most popular meta-learning algorithms to date is Model Agnostic Meta-Learning (MAML)

Original: (Finn et al., 2017).
Revised: (Finn et al., 2017).

Original: MAML is an optimization based meta-learning algorithm with nice properties such as the ability to approximate any learning algorithm and the ability to generalize well to learning data outside of the previous distribution (Finn & Levine, 2017).
Revised: MAML is an optimization based meta-learning algorithm with nice properties such as the ability to approximate any learning algorithm and the ability to generalize well to learning data outside of the previous distribution (Finn & Levine, 2017).

Original: One aspect of MAML that limits its scalability is the need to explicitly compute second derivatives.
Revised: One aspect of MAML that limits its scalability is the need to explicitly compute second derivatives.

Original: The authors proposed a variant called ﬁrst-order MAML (FOMAML), which is deﬁned by ignoring the second derivative terms to address this issue and surprisingly found that it achieved very similar performance.
Revised: The authors proposed a variant called ﬁrst-order MAML (FOMAML), which is deﬁned by ignoring the second derivative terms to address this issue and surprisingly found that it achieved very similar performance.

Original: Recently, this phenomenon was explained by Nichol & Schulman (2018) who noted through Taylor expansion that the two algorithms were approximately optimizing for the same loss function.
Revised: Recently, this phenomenon was explained by Nichol & Schulman (2018) who noted through Taylor expansion that the two algorithms were approximately optimizing for the same loss function.

Original: Nichol & Schulman (2018) also proposed an algorithm, Reptile, that efﬁciently optimizes for approximately the same objective while not requiring that the data be split into training and testing splits for each task learned as MAML does.
Revised: Nichol & Schulman (2018) also proposed an algorithm, Reptile, that efﬁciently optimizes for approximately the same objective while not requiring that the data be split into training and testing splits for each task learned as MAML does.

Original: Reptile is implemented by optimizing across s batches of data sequentially with an SGD based optimizer and learning rate α .
Revised: Reptile is implemented by optimizing across s batches of data sequentially with an SGD based optimizer and learning rate α .

Original: After training on these batches, we take the initial parameters before training θ 0 and update them to θ 0 ← θ 0 + β ∗ ( θ k − θ 0 ) where β is the learning rate for the meta-learning update.
Revised: After training on these batches, we take the initial parameters before training θ 0 and update them to θ 0 ← θ 0 + β ∗ ( θ k − θ 0 ) where β is the learning rate for the meta-learning update.

Original: The process repeats for each series of s batches (algorithm 2).
Revised: The process repeats for each series of s batches (algorithm 2).

Original: Shown in terms of gradients in Nichol & Schulman (2018), Reptile approximately optimizes for the following objective over a set of s batches:
Revised: Shown in terms of gradients in Nichol & Schulman (2018), Reptile approximately optimizes for the following objective over a set of s batches:

Original: [Equation]
Revised: [Equation]

Original: where B 1 , ..., B s are batches within D .
Revised: where B 1 , ..., B s are batches within D .

Original: This is similar to our motivation in equation 4 to the extent that gradients produced on these batches approximate samples from the stationary distribution.
Revised: This is similar to our motivation in equation 4 to the extent that gradients produced on these batches approximate samples from the stationary distribution.

Original: The MER learning objective: In this work, we modify the Reptile algorithm to properly integrate it with an experience replay module, facilitating continual learning while maximizing transfer and minimizing interference.
Revised: The MER learning objective: In this work, we modify the Reptile algorithm to properly integrate it with an experience replay module, facilitating continual learning while maximizing transfer and minimizing interference.

Original: As we describe in more detail during the derivation in Appendix I, achieving the Reptile objective in an online setting where examples are provided sequentially is non-trivial and is in part only achievable because of our sampling strategies for both the buffer and batch.
Revised: As we describe in more detail during the derivation in Appendix I, achieving the Reptile objective in an online setting where examples are provided sequentially is non-trivial and is in part only achievable because of our sampling strategies for both the buffer and batch.

Original: Following our remarks about experience replay from the prior section, this allows us to optimize for the following objective in a continual learning setting using our proposed MER algorithm:
Revised: Following our remarks about experience replay from the prior section, this allows us to optimize for the following objective in a continual learning setting using our proposed MER algorithm:

Original: [Equation]
Revised: [Equation]

Original: The MER algorithm:
Revised: The MER algorithm:

Original: MER maintains an experience replay style memory M with reservoir sampling and at each time step draws s batches including k − 1 random samples from the buffer to be trained alongside the current example.
Revised: MER maintains an experience replay style memory M with reservoir sampling and at each time step draws s batches including k − 1 random samples from the buffer to be trained alongside the current example.

Original: Each of the k examples within each batch is treated as its own
Revised: Each of the k examples within each batch is treated as its own

Original: Reptile batch of size 1 with an inner loop Reptile meta-update after that batch is processed.
Revised: Reptile batch of size 1 with an inner loop Reptile meta-update after that batch is processed.

Original: We then apply the Reptile meta-update again in an outer loop across the s batches.
Revised: We then apply the Reptile meta-update again in an outer loop across the s batches.

Original: We provide further details for MER in algorithm 1.
Revised: We provide further details for MER in algorithm 1.

Original: This procedure approximates the objective of equation 7 when β = 1 .
Revised: This procedure approximates the objective of equation 7 when β = 1 .

Original: The sample function produces s batches for updates.
Revised: The sample function produces s batches for updates.

Original: Each batch is created by ﬁrst adding the current example and then interleaving k − 1 random examples from M .
Revised: Each batch is created by ﬁrst adding the current example and then interleaving k − 1 random examples from M .

Original: Controlling the degree of regularization: In light of our ideal objective in equation 4, we can see that using a SGD batch size of 1 has an advantage over larger batches because it allows for the second derivative information conveyed to the algorithm to be ﬁne grained on the example level.
Revised: Controlling the degree of regularization: In light of our ideal objective in equation 4, we can see that using a SGD batch size of 1 has an advantage over larger batches because it allows for the second derivative information conveyed to the algorithm to be ﬁne grained on the example level.

Original: Another reason to use sample level effective batches is that for a given number of samples drawn from the buffer, we maximize s from equation 6.
Revised: Another reason to use sample level effective batches is that for a given number of samples drawn from the buffer, we maximize s from equation 6.

Original: In equation 6, the typical ofﬂine learning loss has a weighting proportional to s and the regularizer term to maximize transfer and minimize interference has a weighting proportional to αs ( s − 1) / 2 .
Revised: In equation 6, the typical ofﬂine learning loss has a weighting proportional to s and the regularizer term to maximize transfer and minimize interference has a weighting proportional to αs ( s − 1) / 2 .

Original: This implies that by maximizing the effective s we can put more weight on the regularization term.
Revised: This implies that by maximizing the effective s we can put more weight on the regularization term.

Original: We found that for a ﬁxed number of examples drawn from M , we consistently performed better converting to a long list of individual samples than we did using proper batches as in Nichol & Schulman (2018) for few shot learning.
Revised: We found that for a ﬁxed number of examples drawn from M , we consistently performed better converting to a long list of individual samples than we did using proper batches as in Nichol & Schulman (2018) for few shot learning.

Original: Prioritizing current learning: To ensure strong regularization, we would like our number of batches processed in a Reptile update to be large – enough that experience replay alone would start to overﬁt to M .
Revised: Prioritizing current learning: To ensure strong regularization, we would like our number of batches processed in a Reptile update to be large – enough that experience replay alone would start to overﬁt to M .

Original: As such, we also need to make sure we provide enough priority to learning the current example, particularly because we may not store it in M .
Revised: As such, we also need to make sure we provide enough priority to learning the current example, particularly because we may not store it in M .

Original: To achieve this in algorithm 1, we sample s separate batches from M that are processed sequentially and each interleaved with the current example.
Revised: To achieve this in algorithm 1, we sample s separate batches from M that are processed sequentially and each interleaved with the current example.

Original: In Appendix H we also outline two additional variants of MER with very similar properties in that they effectively approximate for the same objective.
Revised: In Appendix H we also outline two additional variants of MER with very similar properties in that they effectively approximate for the same objective.

Original: In one we choose one big batch of size sk − s memories and s copies of the current example (algorithm 6).
Revised: In one we choose one big batch of size sk − s memories and s copies of the current example (algorithm 6).

Original: In the other, we choose one memory batch of size k − 1 with a special current item learning rate of sα (algorithm 7).
Revised: In the other, we choose one memory batch of size k − 1 with a special current item learning rate of sα (algorithm 7).

Original: Unique properties:
Revised: Unique properties:

Original: In the end, our approach amounts to a quite easy to implement and computationally efﬁcient extension of SGD, which is applied to an experience replay buffer by leveraging the machinery of past work on optimization based meta-learning.
Revised: In the end, our approach amounts to a quite easy to implement and computationally efﬁcient extension of SGD, which is applied to an experience replay buffer by leveraging the machinery of past work on optimization based meta-learning.

Original: However, the emergent regularization on learning is totally different than those previously considered.
Revised: However, the emergent regularization on learning is totally different than those previously considered.

Original: Past work on optimization based meta-learning has enabled fast learning on incoming data without considering past data.
Revised: Past work on optimization based meta-learning has enabled fast learning on incoming data without considering past data.

Original: Meanwhile, past work on experience replay only focused on stabilizing learning by approximating stationary conditions without altering model parameters to change the dynamics of transfer and interference.
Revised: Meanwhile, past work on experience replay only focused on stabilizing learning by approximating stationary conditions without altering model parameters to change the dynamics of transfer and interference.

Original:  E VALUATION FOR S UPERVISED C ONTINUAL L IFELONG L EARNING
Revised:  E VALUATION FOR S UPERVISED C ONTINUAL L IFELONG L EARNING

Original: To test the efﬁcacy of MER we compare it to relevant baselines for continual learning of many supervised tasks from Lopez-Paz & Ranzato (2017) (see Appendix D for in-depth descriptions):
Revised: To test the efﬁcacy of MER we compare it to relevant baselines for continual learning of many supervised tasks from Lopez-Paz & Ranzato (2017) (see Appendix D for in-depth descriptions):

Original: • Online: represents online learning performance of a model trained straightforwardly one example at a time on the incoming non-stationary training data by simply applying SGD.
Revised: • Online: represents online learning performance of a model trained straightforwardly one example at a time on the incoming non-stationary training data by simply applying SGD.

Original: • Independent: an independent predictor per task with less hidden units proportional to the number of tasks.
Revised: • Independent: an independent predictor per task with less hidden units proportional to the number of tasks.

Original: When useful, it can be initialized by cloning the last predictor.
Revised: When useful, it can be initialized by cloning the last predictor.

Original: • Task Input: has the same architecture as Online, but with a dedicated input layer per task.
Revised: • Task Input: has the same architecture as Online, but with a dedicated input layer per task.

Original: • EWC: Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) is an algorithm that modiﬁes online learning where the loss is regularized to avoid catastrophic forgetting.
Revised: • EWC: Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) is an algorithm that modiﬁes online learning where the loss is regularized to avoid catastrophic forgetting.

Original: • GEM:
Revised: • GEM:

Original: Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato, 2017) is an approach for making efﬁcient use of episodic storage by following gradients on incoming examples to the maximum extent while altering them so that they do not interfere with past memories.
Revised: Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato, 2017) is an approach for making efﬁcient use of episodic storage by following gradients on incoming examples to the maximum extent while altering them so that they do not interfere with past memories.

Original: An independent adhoc analysis is performed to alter each incoming gradient.
Revised: An independent adhoc analysis is performed to alter each incoming gradient.

Original: In contrast to
Revised: In contrast to

Original: MER,
Revised: MER,

Original: nothing generalizable is
Revised: nothing generalizable is

Original: learned
Revised: learned

Original: across
Revised: across

Original: examples
Revised: examples

Original: about
Revised: about

Original: how
Revised: how

Original: to alter
Revised: to alter

Original: gradients.
Revised: gradients.

Original: We follow Lopez-Paz & Ranzato (2017) and consider ﬁnal retained accuracy across all tasks after training sequentially on all tasks as our main metric for comparing approaches.
Revised: We follow Lopez-Paz & Ranzato (2017) and consider ﬁnal retained accuracy across all tasks after training sequentially on all tasks as our main metric for comparing approaches.

Original: Moving forward we will refer to this metric as retained accuracy (RA) .
Revised: Moving forward we will refer to this metric as retained accuracy (RA) .

Original: In order to reveal more characteristics of the learning behavior, we also report the learning accuracy (LA) which is the average accuracy for each task directly after it is learned.
Revised: In order to reveal more characteristics of the learning behavior, we also report the learning accuracy (LA) which is the average accuracy for each task directly after it is learned.

Original: Additionally, we report the backward transfer and interference (BTI) as the average change in accuracy from when a task is learned to the end of training.
Revised: Additionally, we report the backward transfer and interference (BTI) as the average change in accuracy from when a task is learned to the end of training.

Original: A highly negative BTI reﬂects catastrophic forgetting.
Revised: A highly negative BTI reﬂects catastrophic forgetting.

Original: Forward transfer and interference (Lopez-Paz & Ranzato, 2017) is only applicable for one task we explore, so we provide details in Appendix K.
Revised: Forward transfer and interference (Lopez-Paz & Ranzato, 2017) is only applicable for one task we explore, so we provide details in Appendix K.

Original: Question 1 How does MER perform on supervised continual learning benchmarks?
Revised: Question 1 How does MER perform on supervised continual learning benchmarks?

Original: To address this question we consider two continual learning benchmarks from Lopez-Paz & Ranzato (2017).
Revised: To address this question we consider two continual learning benchmarks from Lopez-Paz & Ranzato (2017).

Original: MNIST Permutations is a variant of MNIST ﬁrst proposed in Kirkpatrick et al.
Revised: MNIST Permutations is a variant of MNIST ﬁrst proposed in Kirkpatrick et al.

Original: (2017) where each task is transformed by a ﬁxed permutation of the MNIST pixels.
Revised: (2017) where each task is transformed by a ﬁxed permutation of the MNIST pixels.

Original: As such, the input distribution of each task is unrelated.
Revised: As such, the input distribution of each task is unrelated.

Original: MNIST Rotations is another variant of MNIST proposed in
Revised: MNIST Rotations is another variant of MNIST proposed in

Original: Lopez-Paz & Ranzato (2017) where each task contains digits rotated by a ﬁxed angle between 0 and 180 degrees.
Revised: Lopez-Paz & Ranzato (2017) where each task contains digits rotated by a ﬁxed angle between 0 and 180 degrees.

Original: We follow the standard benchmark setting from Lopez-Paz & Ranzato (2017) using a modest memory buffer of size 5120 to learn 1000 sampled examples across each of 20 tasks.
Revised: We follow the standard benchmark setting from Lopez-Paz & Ranzato (2017) using a modest memory buffer of size 5120 to learn 1000 sampled examples across each of 20 tasks.

Original: We provide detailed information about our architectures and hyperparameters in Appendix J.
Revised: We provide detailed information about our architectures and hyperparameters in Appendix J.

Original: In Table 1 we report results on these benchmarks in comparison to our baseline approaches.
Revised: In Table 1 we report results on these benchmarks in comparison to our baseline approaches.

Original: Clearly
Revised: Clearly

Original: GEM outperforms our other baselines, but our approach adds signiﬁcant value over GEM in terms of retained accuracy on both benchmarks.
Revised: GEM outperforms our other baselines, but our approach adds signiﬁcant value over GEM in terms of retained accuracy on both benchmarks.

Original: MER achieves this by striking a superior balance between transfer and interference with respect to the past and future data.
Revised: MER achieves this by striking a superior balance between transfer and interference with respect to the past and future data.

Original: MER displays the best adaption to incoming tasks, while also providing very strong retention of knowledge when learning future tasks.
Revised: MER displays the best adaption to incoming tasks, while also providing very strong retention of knowledge when learning future tasks.

Original: EWC and using a task speciﬁc input layer both also lead to gains over standard online learning in terms of retained accuracy.
Revised: EWC and using a task speciﬁc input layer both also lead to gains over standard online learning in terms of retained accuracy.

Original: However, they are quite far below the performance of approaches that make usage of episodic storage.
Revised: However, they are quite far below the performance of approaches that make usage of episodic storage.

Original: While EWC does not store examples, in storing the Fisher information for each task it accrues more incremental resources than the episodic storage approaches.
Revised: While EWC does not store examples, in storing the Fisher information for each task it accrues more incremental resources than the episodic storage approaches.

Original: Question 

Revised: Question

Original: How do the performance gains from MER vary as a function of the buffer size?
Revised: How do the performance gains from MER vary as a function of the buffer size?

Original: To make progress towards the greater goals of lifelong learning, we would like our algorithm to make the most use of even a modest buffer.
Revised: To make progress towards the greater goals of lifelong learning, we would like our algorithm to make the most use of even a modest buffer.

Original: This is because in extremely large scale settings it is
Revised: This is because in extremely large scale settings it is

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: unrealistic to assume a system can store a large percentage of previous examples in memory.
Revised: unrealistic to assume a system can store a large percentage of previous examples in memory.

Original: As such, we would like to compare MER to GEM, which is known to perform well with an extremely small memory buffer (Lopez-Paz & Ranzato, 2017).
Revised: As such, we would like to compare MER to GEM, which is known to perform well with an extremely small memory buffer (Lopez-Paz & Ranzato, 2017).

Original: We consider a buffer size of 500, that is overtimes smaller than the standard setting on these benchmarks.
Revised: We consider a buffer size of 500, that is overtimes smaller than the standard setting on these benchmarks.

Original: Additionally, we also consider a buffer size of 200, matching the smallest setting explored in Lopez-Paz & Ranzato (2017).
Revised: Additionally, we also consider a buffer size of 200, matching the smallest setting explored in Lopez-Paz & Ranzato (2017).

Original: This setting corresponds to an average storage of 1 example for each combination of task and class.
Revised: This setting corresponds to an average storage of 1 example for each combination of task and class.

Original: We report our results in Table 2.
Revised: We report our results in Table 2.

Original: The beneﬁts of MER seem to grow as the buffer becomes smaller.
Revised: The beneﬁts of MER seem to grow as the buffer becomes smaller.

Original: In the smallest setting, MER provides more than a 10% boost in retained accuracy on both benchmarks.
Revised: In the smallest setting, MER provides more than a 10% boost in retained accuracy on both benchmarks.

Original: Question 3 How effective is MER at dealing with increasingly non-stationary settings?
Revised: Question 3 How effective is MER at dealing with increasingly non-stationary settings?

Original: Another larger goal of lifelong learning is to enable continual learning with only relatively few examples per task.
Revised: Another larger goal of lifelong learning is to enable continual learning with only relatively few examples per task.

Original: This setting is particularly difﬁcult because we have less data to characterize each class to learn from and our distribution is increasingly non-stationary over a ﬁxed amount of training.
Revised: This setting is particularly difﬁcult because we have less data to characterize each class to learn from and our distribution is increasingly non-stationary over a ﬁxed amount of training.

Original: We would like to explore how various models perform in this kind of setting.
Revised: We would like to explore how various models perform in this kind of setting.

Original: To do this we consider two new benchmarks.
Revised: To do this we consider two new benchmarks.

Original: Many Permutations is a variant of MNIST Permutations that hastimes more tasks (100 total) and 5 times less training examples per task (200 each).
Revised: Many Permutations is a variant of MNIST Permutations that hastimes more tasks (100 total) and 5 times less training examples per task (200 each).

Original: Meanwhile we also explore the Omniglot (Lake et al., 2011) benchmark treating each of the 50 alphabets to be a task (see Appendix J for experimental details).
Revised: Meanwhile we also explore the Omniglot (Lake et al., 2011) benchmark treating each of the 50 alphabets to be a task (see Appendix J for experimental details).

Original: Following multi-task learning conventions, 90% of the data is used for training and 10% is used for testing (Yang & Hospedales, 2017).
Revised: Following multi-task learning conventions, 90% of the data is used for training and 10% is used for testing (Yang & Hospedales, 2017).

Original: Overall there are 1623 characters.
Revised: Overall there are 1623 characters.

Original: We learn each character and task sequentially with a task speciﬁc output layer.
Revised: We learn each character and task sequentially with a task speciﬁc output layer.

Original: We report continual learning results using these new datasets in Table 3.
Revised: We report continual learning results using these new datasets in Table 3.

Original: The effect on Many Permutations of efﬁciently using episodic storage becomes even more pronounced when the setting becomes more non-stationary.
Revised: The effect on Many Permutations of efﬁciently using episodic storage becomes even more pronounced when the setting becomes more non-stationary.

Original: GEM and MER
Revised: GEM and MER

Original: both achieve nearly double the performance of EWC and online learning.
Revised: both achieve nearly double the performance of EWC and online learning.

Original: We also see that increasingly non-stationary settings lead to a larger performance gain for MER over GEM.
Revised: We also see that increasingly non-stationary settings lead to a larger performance gain for MER over GEM.

Original: Gains are quite signiﬁcant for Many Permutations and remarkable for Omniglot .
Revised: Gains are quite signiﬁcant for Many Permutations and remarkable for Omniglot .

Original: Omniglot is even more non-stationary including slightly fewer examples per task and
Revised: Omniglot is even more non-stationary including slightly fewer examples per task and

Original: MER nearly quadruples the performance of baseline techniques.
Revised: MER nearly quadruples the performance of baseline techniques.

Original: Considering the poor performance
Revised: Considering the poor performance

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: of online learning
Revised: of online learning

Original: and EWC it is natural to question whether or not examples were learned in the ﬁrst place.
Revised: and EWC it is natural to question whether or not examples were learned in the ﬁrst place.

Original: We experiment with using as many as 100 gradient descent steps per incoming example to ensure each example is learned when ﬁrst seen.
Revised: We experiment with using as many as 100 gradient descent steps per incoming example to ensure each example is learned when ﬁrst seen.

Original: However, due to the extremely non-stationary setting no run of any variant we tried surpassed 5.5% retained accuracy.
Revised: However, due to the extremely non-stationary setting no run of any variant we tried surpassed 5.5% retained accuracy.

Original: GEM also has major deﬁcits for learning on Omniglot that are resolved by MER which achieves far better performance when it comes to quickly learning the current task.
Revised: GEM also has major deﬁcits for learning on Omniglot that are resolved by MER which achieves far better performance when it comes to quickly learning the current task.

Original: GEM maintains a buffer using a recent item based sampling strategy and thus can not deal with non-stationarity within the task nearly as well as reservoir sampling.
Revised: GEM maintains a buffer using a recent item based sampling strategy and thus can not deal with non-stationarity within the task nearly as well as reservoir sampling.

Original: Additionally, we found that the optimization based on the buffer was signiﬁcantly less effective and less reliable as the quadratic program fails for many hyperparameter values that lead to non-positive deﬁnite matrices.
Revised: Additionally, we found that the optimization based on the buffer was signiﬁcantly less effective and less reliable as the quadratic program fails for many hyperparameter values that lead to non-positive deﬁnite matrices.

Original: Unfortunately, we could not get GEM to consistently converge on
Revised: Unfortunately, we could not get GEM to consistently converge on

Original: Omniglot for a memory size of 500 (signiﬁcantly less than the number of classes), meanwhile MER handles it well.
Revised: Omniglot for a memory size of 500 (signiﬁcantly less than the number of classes), meanwhile MER handles it well.

Original: In fact, MER greatly outperforms GEM with an order of magnitude smaller buffer.
Revised: In fact, MER greatly outperforms GEM with an order of magnitude smaller buffer.

Original: We provide additional details about our experiments on Omniglot in Figure 3.
Revised: We provide additional details about our experiments on Omniglot in Figure 3.

Original: We plot retained training accuracy, retained testing accuracy, and computation time for the entire training period using one
Revised: We plot retained training accuracy, retained testing accuracy, and computation time for the entire training period using one

Original: CPU.
Revised: CPU.

Original: We ﬁnd that MER strikes the best balance of computational efﬁciency and performance even when using algorithm 1 for MER which performs more computation than algorithm 7.
Revised: We ﬁnd that MER strikes the best balance of computational efﬁciency and performance even when using algorithm 1 for MER which performs more computation than algorithm 7.

Original: The computation involved in the GEM update does not scale well to large CNN models like those that are popular for
Revised: The computation involved in the GEM update does not scale well to large CNN models like those that are popular for

Original: Omniglot.
Revised: Omniglot.

Original: MER is far better able to ﬁt the training data than our baseline models while maintaining a computational efﬁciency closer to online update methods like EWC than GEM.
Revised: MER is far better able to ﬁt the training data than our baseline models while maintaining a computational efﬁciency closer to online update methods like EWC than GEM.

Original:  E VALUATION FOR C ONTINUAL R EINFORCEMENT L EARNING
Revised:  E VALUATION FOR C ONTINUAL R EINFORCEMENT L EARNING

Original: Question 4 Can MER improve a DQN with ER in continual reinforcement learning settings?
Revised: Question 4 Can MER improve a DQN with ER in continual reinforcement learning settings?

Original: We considered the evaluation of MER in a continual reinforcement learning setting where the environment is highly non-stationary.
Revised: We considered the evaluation of MER in a continual reinforcement learning setting where the environment is highly non-stationary.

Original: In order to produce these non-stationary environments in a
Revised: In order to produce these non-stationary environments in a

Original: [Figure]
Revised: [Figure]

Original: controlled way suitable for our experimental purposes, we utilized arcade games provided by Tasﬁ (2016).
Revised: controlled way suitable for our experimental purposes, we utilized arcade games provided by Tasﬁ (2016).

Original: Speciﬁcally, we used Catcher and Flappy Bird, two simple but interesting enough environments (see Appendix M.1 for details).
Revised: Speciﬁcally, we used Catcher and Flappy Bird, two simple but interesting enough environments (see Appendix N.1 for details).

Original: For the purposes of our explanation, we will call each set of ﬁxed game-dependent parameters a task 5 .
Revised: For the purposes of our explanation, we will call each set of ﬁxed game-dependent parameters a task 5 .

Original: The multi-task setting is then built by introducing changes in these parameters, resulting in non-stationarity across tasks.
Revised: The multi-task setting is then built by introducing changes in these parameters, resulting in non-stationarity across tasks.

Original: Each agent is once again evaluated based on its performance over time on all tasks.
Revised: Each agent is once again evaluated based on its performance over time on all tasks.

Original: Our model uses a standard DQN model, developed for Atari (Mnih et al., 2015).
Revised: Our model uses a standard DQN model, developed for Atari (Mnih et al., 2015).

Original: See Appendix M.2 for implementation details.
Revised: See Appendix N.2 for implementation details

Original:  In Catcher, we then obtain different tasks by incrementally increasing the pellet velocity a total of 5 times during training.
Revised: [Figure] In Catcher, we then obtain different tasks by incrementally increasing the pellet velocity a total of 5 times during training.

Original: In Flappy Bird, the different tasks are obtained by incrementally reducing the separation between upper and lower pipes a total of 5 times during training.
Revised: In Flappy Bird, the different tasks are obtained by incrementally reducing the separation between upper and lower pipes a total of 5 times during training.

Original: In Figure 4, we show the performance in Catcher when trained sequentially on 6 different tasks for 25k frames each to a maximum of 150k frames, evaluated at each point in time in all 6 tasks.
Revised: In Figure 4, we show the performance in Catcher when trained sequentially on 6 different tasks for 25k frames each to a maximum of 150k frames, evaluated at each point in time in all 6 tasks.

Original: Under these nonstationary conditions, a DQN using MER performs consistently better than the standard DQN with an experience replay buffer (see Appendix M.4 for further comments and ablation results).
Revised: Under these nonstationary conditions, a DQN using MER performs consistently better than the standard DQN with an experience replay buffer (see Appendix N.4 for further comments and ablation results).

Original: If we take as inspiration how humans perform, in the last stages of training we hope that a player that obtains good results in later tasks will also obtain good results in the ﬁrst tasks, as the ﬁrst tasks are subsumed in the latter ones.
Revised: If we take as inspiration how humans perform, in the last stages of training we hope that a player that obtains good results in later tasks will also obtain good results in the ﬁrst tasks, as the ﬁrst tasks are subsumed in the latter ones.

Original: For example, in Catcher, the pellet moves faster in later tasks, and thus we expect to be able to do well on the ﬁrst task.
Revised: For example, in Catcher, the pellet moves faster in later tasks, and thus we expect to be able to do well on the ﬁrst task.

Original: However, DQN forgets signiﬁcantly how to get slowly moving pellets.
Revised: However, DQN forgets signiﬁcantly how to get slowly moving pellets.

Original: In contrast, DQN-MER exhibits minimal or no forgetting after training on the rest of the tasks.
Revised: In contrast, DQN-MER exhibits minimal or no forgetting after training on the rest of the tasks.

Original: This behavior is intuitive because we would expect transfer to happen naturally in this setting.
Revised: This behavior is intuitive because we would expect transfer to happen naturally in this setting.

Original: We see similar behavior for Flappy Bird.
Revised: We see similar behavior for Flappy Bird.

Original: DQN-MER becomes a Platinum player on the ﬁrst task when it is learning the third task.
Revised: DQN-MER becomes a Platinum player on the ﬁrst task when it is learning the third task.

Original: This is a more difﬁcult environment in which the pipe gap is noticeably smaller (see Appendix M.4).
Revised: This is a more difﬁcult environment in which the pipe gap is noticeably smaller (see Appendix N.4).

Original: DQN-MER exhibits the kind of learning patterns expected from humans for these games, while a standard DQN struggles to generalize as the game changes and to retain knowledge over time.
Revised: DQN-MER exhibits the kind of learning patterns expected from humans for these games, while a standard DQN struggles to generalize as the game changes and to retain knowledge over time.

Original:  F URTHER A NALYSIS OF THE A PPROACH
Revised:  F URTHER A NALYSIS OF THE A PPROACH

Original: In this section we would like to dive deeper into how MER works.
Revised: In this section we would like to dive deeper into how MER works.

Original: To do so we run additional detailed experiments across our three MNIST based continual learning benchmarks.
Revised: To do so we run additional detailed experiments across our three MNIST based continual learning benchmarks.

Original: Question 5 Does MER lead to a shift in the distribution of gradient dot products?
Revised: Question 5 Does MER lead to a shift in the distribution of gradient dot products?

Original: We would like to directly verify that MER achieves our motivation in equation 7 and results in signiﬁcant changes in the distribution of gradient dot products between new incoming examples and past examples over the course of learning when compared to experience replay (ER) from algorithm
Revised: We would like to directly verify that MER achieves our motivation in equation 7 and results in signiﬁcant changes in the distribution of gradient dot products between new incoming examples and past examples over the course of learning when compared to experience replay (ER) from algorithm

Original: [Table]
Revised: [Table]

Original: For these experiments, we maintain a history of all examples seen that is totally separate from our notion of memory buffers that only include a partial history of examples.
Revised: For these experiments, we maintain a history of all examples seen that is totally separate from our notion of memory buffers that only include a partial history of examples.

Original: Every time we receive a new example we use the current model to extract a gradient direction and we also randomly sample ﬁve examples from the previous history.
Revised: Every time we receive a new example we use the current model to extract a gradient direction and we also randomly sample ﬁve examples from the previous history.

Original: We save the dot products of the incoming example gradient with these ﬁve past example gradients and consider the mean of the distribution of dot products seen over the course of learning for each model.
Revised: We save the dot products of the incoming example gradient with these ﬁve past example gradients and consider the mean of the distribution of dot products seen over the course of learning for each model.

Original: We run this experiment on the best hyperparamater setting for both ER and MER from algorithm 6 with one batch per example for fair comparison.
Revised: We run this experiment on the best hyperparamater setting for both ER and MER from algorithm 6 with one batch per example for fair comparison.

Original: Each model is evaluated ﬁve times over the course of learning.
Revised: Each model is evaluated ﬁve times over the course of learning.

Original: We report mean and standard deviations of the mean gradient dot product across runs in Table 4.
Revised: We report mean and standard deviations of the mean gradient dot product across runs in Table 4.

Original: We can thus verify that a very signiﬁcant and reproducible difference in the mean gradient encountered is seen for MER in comparison to ER alone.
Revised: We can thus verify that a very signiﬁcant and reproducible difference in the mean gradient encountered is seen for MER in comparison to ER alone.

Original: This difference alters the learning process making incoming examples on average result in slight transfer rather than signiﬁcant interference.
Revised: This difference alters the learning process making incoming examples on average result in slight transfer rather than signiﬁcant interference.

Original: This analysis conﬁrms the desired effect of the objective function in equation 7.
Revised: This analysis conﬁrms the desired effect of the objective function in equation 7.

Original: For these tasks there are enough similarities that our meta-learning generalizes very well into the future.
Revised: For these tasks there are enough similarities that our meta-learning generalizes very well into the future.

Original: We should also expect it to perform well during drastic domain shifts like other meta-learning algorithms driven by SGD alone (Finn & Levine, 2017).
Revised: We should also expect it to perform well during drastic domain shifts like other meta-learning algorithms driven by SGD alone (Finn & Levine, 2017).

Original: Question 6 What components of MER are most important?
Revised: Question 6 What components of MER are most important?

Original: We would like to further analyze our MER model to understand what components add the most value and when.
Revised: We would like to further analyze our MER model to understand what components add the most value and when.

Original: We want to understand how powerful our proposed variants of ER are on their own and how much is added by adding meta-learning to ER.
Revised: We want to understand how powerful our proposed variants of ER are on their own and how much is added by adding meta-learning to ER.

Original: In Appendix L we provide detailed results considering ablated baselines for our experiments on the MNIST lifelong learning benchmarks.
Revised: In Appendix L we provide detailed results considering ablated baselines for our experiments on the MNIST lifelong learning benchmarks.

Original: Our versions of ER consistently provide gains over GEM on their own, but the techniques perform very comparably when we also maintain GEM’s buffer with reservoir sampling or use ER with a GEM style buffer.
Revised: Our versions of ER consistently provide gains over GEM on their own, but the techniques perform very comparably when we also maintain GEM’s buffer with reservoir sampling or use ER with a GEM style buffer.

Original: Additionally, we see that adding meta-learning to ER consistently results in performance gains.
Revised: Additionally, we see that adding meta-learning to ER consistently results in performance gains.

Original: In fact, meta-learning appears to provide increasing value for smaller buffers. 
Revised: In fact, meta-learning appears to provide increasing value for smaller buffers. In

Revised: Appendix M, we provide further validation that our results are reproducible across runs and seeds.

Original: We would also like to compare the variants of MER proposed in algorithms 1, 6, and 7.
Revised: We would also like to compare the variants of MER proposed in algorithms 1, 6, and 7.

Original: Conceptually algorithms 1 and 7 represent different mechanisms of increasing the importance of the current example in algorithm 6.
Revised: Conceptually algorithms 1 and 7 represent different mechanisms of increasing the importance of the current example in algorithm 6.

Original: We ﬁnd that all variants of MER result in signiﬁcant improvements on ER.
Revised: We ﬁnd that all variants of MER result in signiﬁcant improvements on ER.

Original: Meanwhile, the variants that increase the importance of the current example see a further improvement in performance, performing quite comparably to each other.
Revised: Meanwhile, the variants that increase the importance of the current example see a further improvement in performance, performing quite comparably to each other.

Original: Overall, in our MNIST experiments algorithm 7 displays the best tradeoff of computational efﬁciency and performance.
Revised: Overall, in our MNIST experiments algorithm 7 displays the best tradeoff of computational efﬁciency and performance.

Original: Finally, we conducted experiments demonstrating that adaptive optimizers like Adam and RMSProp can not account for the gap between ER and MER.
Revised: Finally, we conducted experiments demonstrating that adaptive optimizers like Adam and RMSProp can not account for the gap between ER and MER.

Original: Particularly for smaller buffer sizes, these approaches overﬁt more on the buffer and actually hurt generalization in comparison to SGD.
Revised: Particularly for smaller buffer sizes, these approaches overﬁt more on the buffer and actually hurt generalization in comparison to SGD.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we have cast a new perspective on the problem of continual learning in terms of a fundamental trade-off between transfer and interference.
Revised: In this paper we have cast a new perspective on the problem of continual learning in terms of a fundamental trade-off between transfer and interference.

Original: Exploiting this perspective, we have in turn developed a new algorithm Meta-Experience Replay (MER) that is well suited for application to general purpose continual learning problems.
Revised: Exploiting this perspective, we have in turn developed a new algorithm Meta-Experience Replay (MER) that is well suited for application to general purpose continual learning problems.

Original: We have demonstrated that MER regularizes the objective of experience replay so that gradients on incoming examples are more likely to have transfer and less likely to have interference with respect to past examples.
Revised: We have demonstrated that MER regularizes the objective of experience replay so that gradients on incoming examples are more likely to have transfer and less likely to have interference with respect to past examples.

Original: The result is a general purpose solution to continual learning problems that outperforms strong baselines for both supervised continual learning benchmarks and continual learning in non-stationary reinforcement learning environments.
Revised: The result is a general purpose solution to continual learning problems that outperforms strong baselines for both supervised continual learning benchmarks and continual learning in non-stationary reinforcement learning environments.

Original: Techniques for continual learning have been largely driven by different conceptualizations of the fundamental problem encountered by neural networks.
Revised: Techniques for continual learning have been largely driven by different conceptualizations of the fundamental problem encountered by neural networks.

Original: We hope that the transfer-interference tradeoff can be a useful problem view for future work to exploit with MER as a ﬁrst successful example.
Revised: We hope that the transfer-interference tradeoff can be a useful problem view for future work to exploit with MER as a ﬁrst successful example.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: We would like to thank Pouya Bashivan, Christopher Potts, Dan Jurafsky and Joshua Greene for their input and support of this work.
Revised: We would like to thank Pouya Bashivan, Christopher Potts, Dan Jurafsky and Joshua Greene for their input and support of this work.

Original: Additionally, we would like to thank Arslan Chaudhry and
Revised: Additionally, we would like to thank Arslan Chaudhry and

Original: Marc’Aurelio Ranzato for their helpful comments and discussions.
Revised: Marc’Aurelio Ranzato for their helpful comments and discussions.

Original: We also thank the three anonymous reviewers for their valuable suggestions.
Revised: We also thank the three anonymous reviewers for their valuable suggestions.

Original: This research was supported by the MIT-IBM Watson
Revised: This research was supported by the MIT-IBM Watson

Original: AI Lab, and is based in part upon work supported by the Stanford Data Science Initiative and by the
Revised: AI Lab, and is based in part upon work supported by the Stanford Data Science Initiative and by the

Original: NSF under Grant No. BCS-1456077 and the NSF Award IIS-1514268.
Revised: NSF under Grant No. BCS-1456077 and the NSF Award IIS-1514268.

Original: A C ONTINUAL L EARNING P ROBLEM F ORMULATION
Revised: A C ONTINUAL L EARNING P ROBLEM F ORMULATION

Original: In the classical ofﬂine supervised learning setting, a learning agent is given a ﬁxed training data set D = { ( x i , y i ) } ni =1 of n samples, each containing an input feature vector x i ∈ X associated with the corresponding output (target, or label)
Revised: In the classical ofﬂine supervised learning setting, a learning agent is given a ﬁxed training data set D = { ( x i , y i ) } ni =1 of n samples, each containing an input feature vector x i ∈ X associated with the corresponding output (target, or label)

Original: y i ∈ Y ; a common assumption is that the training samples are i.i.d.
Revised: y i ∈ Y ; a common assumption is that the training samples are i.i.d.

Original: samples drawn from the same unknown joint probability distribution P ( x , y ) .
Revised: samples drawn from the same unknown joint probability distribution P ( x , y ) .

Original: The learning task is often formulated as a function approximation problem, i.e. ﬁnding a function, or model, f θ ( x ) :
Revised: The learning task is often formulated as a function approximation problem, i.e. ﬁnding a function, or model, f θ ( x ) :

Original: X → Y from a given class of models (e.g., neural networks, decision trees, linear functions, etc.) where θ are the parameters estimated from data.
Revised: X → Y from a given class of models (e.g., neural networks, decision trees, linear functions, etc.) where θ are the parameters estimated from data.

Original: Given a loss function
Revised: Given a loss function

Original: L ( f θ ( x ) , y ) , the parameter estimation is formulated as an empirical risk minimization problem: min
Revised: L ( f θ ( x ) , y ) , the parameter estimation is formulated as an empirical risk minimization problem: min

Original: [Equation]
Revised: [Equation]

Original: On the contrary, the online learning setting does not assume a ﬁxed training dataset but rather a stream of data samples, where unlabeled feature vectors arrive one at a time, or in small minibatches, and the learner must assign labels to those inputs, receive the correct labels, and update the model accordingly, in iterative fashion.
Revised: On the contrary, the online learning setting does not assume a ﬁxed training dataset but rather a stream of data samples, where unlabeled feature vectors arrive one at a time, or in small minibatches, and the learner must assign labels to those inputs, receive the correct labels, and update the model accordingly, in iterative fashion.

Original: While classical online learning assumes i.i.d.
Revised: While classical online learning assumes i.i.d.

Original: samples, continual or lifelong learning does not make such an assumption, and requires a learning agent to handle non-stationary data streams.
Revised: samples, continual or lifelong learning does not make such an assumption, and requires a learning agent to handle non-stationary data streams.

Original: In this work, we deﬁne continual learning as online learning from a non-stationary input data stream, with a speciﬁc type of non-stationarity as deﬁned below.
Revised: In this work, we deﬁne continual learning as online learning from a non-stationary input data stream, with a speciﬁc type of non-stationarity as deﬁned below.

Original: Namely, we follow a commonly used setting to deﬁne non-stationary conditions for continual learning, dubbed locally i.i.d by Lopez-Paz & Ranzato (2017), where the agent learns over a sequence of separate stationary distributions one after another.
Revised: Namely, we follow a commonly used setting to deﬁne non-stationary conditions for continual learning, dubbed locally i.i.d by Lopez-Paz & Ranzato (2017), where the agent learns over a sequence of separate stationary distributions one after another.

Original: We call the individual stationary distributions tasks , where each task t k is an online supervised learning problem associated with its own data probability distribution P k ( x , y ) .
Revised: We call the individual stationary distributions tasks , where each task t k is an online supervised learning problem associated with its own data probability distribution P k ( x , y ) .

Original: Namely, we are given a (potentially inﬁnite) sequence
Revised: Namely, we are given a (potentially inﬁnite) sequence

Original: [Equation]
Revised: [Equation]

Original: While many continual learning methods assume the task descriptors t k are available to a learner, we are interested in developing approaches which do not have to rely on such information and can learn continuously without explicit announcement of the task change.
Revised: While many continual learning methods assume the task descriptors t k are available to a learner, we are interested in developing approaches which do not have to rely on such information and can learn continuously without explicit announcement of the task change.

Original: Borrowing terminology from
Revised: Borrowing terminology from

Original: Chaudhry et al.
Revised: Chaudhry et al.

Original: (2018), we explore the single-headed setting in most of our experiments, which keeps learning a common function f θ across changing tasks.
Revised: (2018), we explore the single-headed setting in most of our experiments, which keeps learning a common function f θ across changing tasks.

Original: In contrast, multi-headed learning, which we consider for our Omniglot experiments, involves a separate ﬁnal classiﬁcation layer for each task.
Revised: In contrast, multi-headed learning, which we consider for our Omniglot experiments, involves a separate ﬁnal classiﬁcation layer for each task.

Original: This makes more sense in case of Omniglot dataset, where the number of classes for each task varies considerably from task to task.
Revised: This makes more sense in case of Omniglot dataset, where the number of classes for each task varies considerably from task to task.

Original: We should also note that for Omniglot we consider a setting that is locally i.i.d.
Revised: We should also note that for Omniglot we consider a setting that is locally i.i.d.

Original: at the class level rather than the task level.
Revised: at the class level rather than the task level.

Original: B R ELATION TO P AST W ORK
Revised: B R ELATION TO P AST W ORK

Original: With regard to the continual learning setting speciﬁcally, other recent work has explored similar operational measures of transfer and interference.
Revised: With regard to the continual learning setting speciﬁcally, other recent work has explored similar operational measures of transfer and interference.

Original: For example, the notions of Forward Transfer and Backward Transfer were explored in Lopez-Paz & Ranzato (2017).
Revised: For example, the notions of Forward Transfer and Backward Transfer were explored in Lopez-Paz & Ranzato (2017).

Original: However, the approach of that work, GEM, was primarily concerned with solving the classic stability-plasticity dilemma (Carpenter & Grossberg, 1987) at a speciﬁc instance of time.
Revised: However, the approach of that work, GEM, was primarily concerned with solving the classic stability-plasticity dilemma (Carpenter & Grossberg, 1987) at a speciﬁc instance of time.

Original: Adjustments to gradients
Revised: Adjustments to gradients

Original: In our work we try to learn a generalizable theory about weight sharing that can learn to inﬂuence the distribution of gradients not just in the past and present, but in the future as well.
Revised: In our work we try to learn a generalizable theory about weight sharing that can learn to inﬂuence the distribution of gradients not just in the past and present, but in the future as well.

Original: Additionally, in Chaudhry et al.
Revised: Additionally, in Chaudhry et al.

Original: (2018) similar ideas were explored with operational measures of intransigence (the inability to learn new data) and forgetting (the loss of previous performance).
Revised: (2018) similar ideas were explored with operational measures of intransigence (the inability to learn new data) and forgetting (the loss of previous performance).

Original: These measures are also intimately related to the stability-plasticity dilemma as intransigence is high when plasticity is low and forgetting is high when stability is low.
Revised: These measures are also intimately related to the stability-plasticity dilemma as intransigence is high when plasticity is low and forgetting is high when stability is low.

Original: The major distinction in the transfer-interference trade-off proposed in this work is that we aim to learn the optimal weight sharing scheme to optimize for the stability-plasticity dilemma with the hope that our learning about weight sharing will improve the stability and efﬁcacy of learning on unseen data as well.
Revised: The major distinction in the transfer-interference trade-off proposed in this work is that we aim to learn the optimal weight sharing scheme to optimize for the stability-plasticity dilemma with the hope that our learning about weight sharing will improve the stability and efﬁcacy of learning on unseen data as well.

Original: With regard to the problem of weight-sharing in neural networks more generally, a host of different strategies have been proposed in the past to deal with the problems of catastrophic forgetting and/or the stability-plasticity dilemma (for review, see French (1999)).
Revised: With regard to the problem of weight-sharing in neural networks more generally, a host of different strategies have been proposed in the past to deal with the problems of catastrophic forgetting and/or the stability-plasticity dilemma (for review, see French (1999)).

Original: For example, one strategy for alleviating catastrophic forgetting is to make distributed representations less distributed – or semi-distributed (French, 1991) – for the case of past learning.
Revised: For example, one strategy for alleviating catastrophic forgetting is to make distributed representations less distributed – or semi-distributed (French, 1991) – for the case of past learning.

Original: Activation sharpening as introduced by French (1991) is a prominent example.
Revised: Activation sharpening as introduced by French (1991) is a prominent example.

Original: A second strategy known as dual network models (McClelland et al., 1995; Ans & Rousset, 1997) is based on the neurobiological ﬁnding that both hippocampal and cortical circuits contributed differentially to memory.
Revised: A second strategy known as dual network models (McClelland et al., 1995; Ans & Rousset, 1997) is based on the neurobiological ﬁnding that both hippocampal and cortical circuits contributed differentially to memory.

Original: The cortical circuits are highly distributed with overlapping representations suitable for task generalization, while the more sparse hippocampal representations tend to be non-overlapping.
Revised: The cortical circuits are highly distributed with overlapping representations suitable for task generalization, while the more sparse hippocampal representations tend to be non-overlapping.

Original: The existence of dual circuits provides an extra degree of freedom for balancing the dual constraints of stability and plasticity.
Revised: The existence of dual circuits provides an extra degree of freedom for balancing the dual constraints of stability and plasticity.

Original: In a similar spirit, models have been proposed that have two classes of weights operating on two different timescales (Hinton & Plaut, 1987).
Revised: In a similar spirit, models have been proposed that have two classes of weights operating on two different timescales (Hinton & Plaut, 1987).

Original: A third strategy also motivated by neurobiological considerations is the use of latent synaptic dynamics (Fusi et al., 2005; Lahiri & Ganguli, 2013).
Revised: A third strategy also motivated by neurobiological considerations is the use of latent synaptic dynamics (Fusi et al., 2005; Lahiri & Ganguli, 2013).

Original: Here the basic idea is that synaptic strength is determined by a multiple of variables, including latent ones not easily observed, operating at different timescales such that their net effect is to provide the system with additional degrees-of-freedom to store past experience without interfering with current learning.
Revised: Here the basic idea is that synaptic strength is determined by a multiple of variables, including latent ones not easily observed, operating at different timescales such that their net effect is to provide the system with additional degrees-of-freedom to store past experience without interfering with current learning.

Original: A fourth strategy is the use of feedback mechanisms to stabilize representations (Carpenter &
Revised: A fourth strategy is the use of feedback mechanisms to stabilize representations (Carpenter &

Original: Grossberg, 1987; Murre, 1992).
Revised: Grossberg, 1987; Murre, 1992).

Original: In this class of models, a previously experienced memory will trigger top down feedback that prevents plasticity, while novel stimuli that experience no such feedback trigger plasticity.
Revised: In this class of models, a previously experienced memory will trigger top down feedback that prevents plasticity, while novel stimuli that experience no such feedback trigger plasticity.

Original: All of these approaches have their own strengths and weaknesses with respect to the stability-plasticity dilemma and, by extension, the transfer-interference trade-off we propose.
Revised: All of these approaches have their own strengths and weaknesses with respect to the stability-plasticity dilemma and, by extension, the transfer-interference trade-off we propose.

Original: Another relevant work is the POWERPLAY framework (Schmidhuber, 2004; 2013) which is a method for asymptotically optimal curriculum learning that by deﬁnition cannot forget previously learned skills.
Revised: Another relevant work is the POWERPLAY framework (Schmidhuber, 2004; 2013) which is a method for asymptotically optimal curriculum learning that by deﬁnition cannot forget previously learned skills.

Original: POWERPLAY also uses environment-independent replay of behavioral traces to avoid forgetting previous skills.
Revised: POWERPLAY also uses environment-independent replay of behavioral traces to avoid forgetting previous skills.

Original: However, POWERPLAY is orthogonal to our work as we consider a different setting where the agent cannot directly control the new tasks that will be encountered in the environment and thus must instead learn to adapt and react to non-stationarity conditions.
Revised: However, POWERPLAY is orthogonal to our work as we consider a different setting where the agent cannot directly control the new tasks that will be encountered in the environment and thus must instead learn to adapt and react to non-stationarity conditions.

Original: In contrast to past work on meta-learning for few shot learning (Santoro et al., 2016; Vinyals et al., 2016; Ravi & Larochelle, 2016; Finn et al., 2017) and reinforcement learning across successive tasks (Al-Shedivat et al., 2018), we are not only trying to improve the speed of learning on new data, but also trying to do it in a way that preserves knowledge of past data and generalizes to future data.
Revised: In contrast to past work on meta-learning for few shot learning (Santoro et al., 2016; Vinyals et al., 2016; Ravi & Larochelle, 2016; Finn et al., 2017) and reinforcement learning across successive tasks (Al-Shedivat et al., 2018), we are not only trying to improve the speed of learning on new data, but also trying to do it in a way that preserves knowledge of past data and generalizes to future data.

Original: While past work has considered learning to inﬂuence gradient angles, so that there is more alignment and thus faster learning within a task, we focus on a setting where we would like to inﬂuence gradient angles from all tasks at all points in time.
Revised: While past work has considered learning to inﬂuence gradient angles, so that there is more alignment and thus faster learning within a task, we focus on a setting where we would like to inﬂuence gradient angles from all tasks at all points in time.

Original: As our model aims to inﬂuence the dynamics of weight sharing, it bears conceptual similarity to mixtures of experts (Jacobs et al., 1991) style models for lifelong and multi-task learning (Misra et al., 2016; Riemer et al., 2016b; Aljundi et al., 2017; Fernando et al., 2017;
Revised: As our model aims to inﬂuence the dynamics of weight sharing, it bears conceptual similarity to mixtures of experts (Jacobs et al., 1991) style models for lifelong and multi-task learning (Misra et al., 2016; Riemer et al., 2016b; Aljundi et al., 2017; Fernando et al., 2017;

Original: Shazeer et al., 2017;
Revised: Shazeer et al., 2017;

Original: Rosenbaum et al., 2018).
Revised: Rosenbaum et al., 2018).

Original: MER implicitly affects the dynamics of weight sharing, but it is possible that combining it with mixtures of experts models could further amplify the ability for the model to control these dynamics.
Revised: MER implicitly affects the dynamics of weight sharing, but it is possible that combining it with mixtures of experts models could further amplify the ability for the model to control these dynamics.

Original: This is potentially an interesting avenue for future work.
Revised: This is potentially an interesting avenue for future work.

Original: The options framework has also been considered as a solution to a similar continual RL setting to the one we explore (Mankowitz et al., 2018).
Revised: The options framework has also been considered as a solution to a similar continual RL setting to the one we explore (Mankowitz et al., 2018).

Original: Options formalize the notion of temporally abstraction actions in RL.
Revised: Options formalize the notion of temporally abstraction actions in RL.

Original: Interestingly, generic architectures designed for shallow (Bacon et al., 2017) or deep (Riemer et al., 2018) hierarchies of options in essence learn very complex patterns of weight sharing over time.
Revised: Interestingly, generic architectures designed for shallow (Bacon et al., 2017) or deep (Riemer et al., 2018) hierarchies of options in essence learn very complex patterns of weight sharing over time.

Original: The option hierarchies constitute an explicit mechanism of controlling the extent of weight sharing for continual learning, allowing for orthogonalization of weights relating to different skills.
Revised: The option hierarchies constitute an explicit mechanism of controlling the extent of weight sharing for continual learning, allowing for orthogonalization of weights relating to different skills.

Original: In contrast, our work explores a method of implicitly optimizing weight sharing for continual learning that improves the efﬁcacy of experience replay.
Revised: In contrast, our work explores a method of implicitly optimizing weight sharing for continual learning that improves the efﬁcacy of experience replay.

Original: MER should be simple to implement in concert with options based methods and combining the two is an interesting direction for future work.
Revised: MER should be simple to implement in concert with options based methods and combining the two is an interesting direction for future work.

Original: C T HE C ONNECTION B ETWEEN W EIGHT S HARING AND THE T RANSFER -I NTERFERENCE T RADE - OFF
Revised: C T HE C ONNECTION B ETWEEN W EIGHT S HARING AND THE T RANSFER -I NTERFERENCE T RADE - OFF

Original: In this section we would like to generalize our interpretation of a large set of different weight sharing schemes including (Riemer et al., 2015; Bengio et al., 2015; Rosenbaum et al., 2018; Serr`a et al., 2018) and how the concept of weight sharing impacts the dynamics of transfer (equation 1) and interference (equation 2).
Revised: In this section we would like to generalize our interpretation of a large set of different weight sharing schemes including (Riemer et al., 2015; Bengio et al., 2015; Rosenbaum et al., 2018; Serr`a et al., 2018) and how the concept of weight sharing impacts the dynamics of transfer (equation 1) and interference (equation 2).

Original: We will assume that we have a total parameter space θ that can be used by our network at any point in time.
Revised: We will assume that we have a total parameter space θ that can be used by our network at any point in time.

Original: However, it is not a requirement that all parameters are actually used at all points in time.
Revised: However, it is not a requirement that all parameters are actually used at all points in time.

Original: So, we can consider two speciﬁc instances in time.
Revised: So, we can consider two speciﬁc instances in time.

Original: One where we receive data point ( x 1 , y 1 ) and leverage parameters θ 1 .
Revised: One where we receive data point ( x 1 , y 1 ) and leverage parameters θ 1 .

Original: Then, at the other instance in time, we receive data point ( x 2 , y 2 ) and leverage parameters θ 2 .
Revised: Then, at the other instance in time, we receive data point ( x 2 , y 2 ) and leverage parameters θ 2 .

Original: θ 1 and θ 2 are both subsets of θ and critically the overlap between these subsets inﬂuences the possible extent of transfer and interference when training on either data point.
Revised: θ 1 and θ 2 are both subsets of θ and critically the overlap between these subsets inﬂuences the possible extent of transfer and interference when training on either data point.

Original: First let us consider two extremes.
Revised: First let us consider two extremes.

Original: In the ﬁrst extreme imagine θ 1 and θ 2 are entirely nonoverlapping.
Revised: In the ﬁrst extreme imagine θ 1 and θ 2 are entirely nonoverlapping.

Original: As such ∂L ( x 1 ,y 1 ) ∂θ · ∂L ( x 2 ,y 2 ) ∂θ = 0 .
Revised: As such ∂L ( x 1 ,y 1 ) ∂θ · ∂L ( x 2 ,y 2 ) ∂θ = 0 .

Original: On the positive side, this means that our solution has no potential for interference between the examples.
Revised: On the positive side, this means that our solution has no potential for interference between the examples.

Original: On the other hand, there is no potential for transfer either.
Revised: On the other hand, there is no potential for transfer either.

Original: On the other extreme, we can imagine that θ 1 = θ 2 .
Revised: On the other extreme, we can imagine that θ 1 = θ 2 .

Original: In this case, the potential for both transfer and interference is maximized as gradients with respect to every parameter have the possibility of a non-zero dot product with each other.
Revised: In this case, the potential for both transfer and interference is maximized as gradients with respect to every parameter have the possibility of a non-zero dot product with each other.

Original: From this discussion it is clear that both the extreme of full weight sharing and the extreme of no weight sharing have value depending on the relationship between data points.
Revised: From this discussion it is clear that both the extreme of full weight sharing and the extreme of no weight sharing have value depending on the relationship between data points.

Original: What we would really like for continual learning is to have a system that learns when to share weights and when not to on its own.
Revised: What we would really like for continual learning is to have a system that learns when to share weights and when not to on its own.

Original: To the extent that our learning about weight sharing generalizes, this should allow us to ﬁnd an optimal solution to the transfer-interference trade-off.
Revised: To the extent that our learning about weight sharing generalizes, this should allow us to ﬁnd an optimal solution to the transfer-interference trade-off.

Original: D F URTHER D ESCRIPTIONS AND C OMPARISONS WITH B ASELINE A LGORITHMS
Revised: D F URTHER D ESCRIPTIONS AND C OMPARISONS WITH B ASELINE A LGORITHMS

Original: Independent: originally reported in (Lopez-Paz & Ranzato, 2017) is the performance of an independent predictor per task which has the same architecture but with less hidden units proportional to the number of tasks.
Revised: Independent: originally reported in (Lopez-Paz & Ranzato, 2017) is the performance of an independent predictor per task which has the same architecture but with less hidden units proportional to the number of tasks.

Original: The independent predictor can be initialized randomly or clone the last trained predictor depending on what leads to better performance.
Revised: The independent predictor can be initialized randomly or clone the last trained predictor depending on what leads to better performance.

Original: EWC: Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) is an algorithm that modiﬁes online learning where the loss is regularized to avoid catastrophic forgetting by considering the importance of parameters in the model as measured by their ﬁsher information.
Revised: EWC: Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) is an algorithm that modiﬁes online learning where the loss is regularized to avoid catastrophic forgetting by considering the importance of parameters in the model as measured by their ﬁsher information.

Original: EWC follows the catastrophic forgetting view of the continual learning problem by promoting less sharing of parameters for new learning that were deemed to be important for performance on old memories .
Revised: EWC follows the catastrophic forgetting view of the continual learning problem by promoting less sharing of parameters for new learning that were deemed to be important for performance on old memories .

Original: We utilize the code provided by Lopez-Paz & Ranzato (2017) in our experiments.
Revised: We utilize the code provided by Lopez-Paz & Ranzato (2017) in our experiments.

Original: The only difference in our setting is that we provide the model one example at a time to test true continual learning rather than providing a batch of 10 examples at a time.
Revised: The only difference in our setting is that we provide the model one example at a time to test true continual learning rather than providing a batch of 10 examples at a time.

Original: GEM:
Revised: GEM:

Original: Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato, 2017) is an algorithm meant to enhance the effectiveness of episodic storage based continual learning techniques by allowing the model to adapt to incoming examples using SGD as long as the gradients do not interfere with examples from each task stored in a memory buffer.
Revised: Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato, 2017) is an algorithm meant to enhance the effectiveness of episodic storage based continual learning techniques by allowing the model to adapt to incoming examples using SGD as long as the gradients do not interfere with examples from each task stored in a memory buffer.

Original: If gradients interfere leading to a decrease in the performance of a past task, a quadratic program is used to solve for the closest gradient to the original that does not have negative gradient dot products with the aggregate memories from any previous tasks.
Revised: If gradients interfere leading to a decrease in the performance of a past task, a quadratic program is used to solve for the closest gradient to the original that does not have negative gradient dot products with the aggregate memories from any previous tasks.

Original: GEM is known to achieve superior performance in comparison to other recently proposed techniques that use episodic storage like Rebufﬁ et al.
Revised: GEM is known to achieve superior performance in comparison to other recently proposed techniques that use episodic storage like Rebufﬁ et al.

Original: (2017), making superior use of small memory buffer sizes.
Revised: (2017), making superior use of small memory buffer sizes.

Original: GEM follows similar motivation to our approach in that it also considers the intelligent use of gradient dot product information to improve the use case of supervised continual learning.
Revised: GEM follows similar motivation to our approach in that it also considers the intelligent use of gradient dot product information to improve the use case of supervised continual learning.

Original: As a result, it is a very strong and interesting baseline to compare with our approach.
Revised: As a result, it is a very strong and interesting baseline to compare with our approach.

Original: We modify the original code and benchmarks provided by Lopez-Paz & Ranzato (2017).
Revised: We modify the original code and benchmarks provided by Lopez-Paz & Ranzato (2017).

Original: Once again the only difference in our setting is that we provide the model one example at a time to test true continual learning rather than providing a batch of 10 examples at a time.
Revised: Once again the only difference in our setting is that we provide the model one example at a time to test true continual learning rather than providing a batch of 10 examples at a time.

Original: We can consider the GEM algorithm as tailored to the stability-plasticity dilemma conceptualization of continual learning in that it looks to preserve performance on past tasks while allowing for maximal plasticity to the new task.
Revised: We can consider the GEM algorithm as tailored to the stability-plasticity dilemma conceptualization of continual learning in that it looks to preserve performance on past tasks while allowing for maximal plasticity to the new task.

Original: To achieve this, GEM solves a quadratic program to ﬁnd an approximate gradient g new that closely matches ∂L ( x new ,y new )
Revised: To achieve this, GEM solves a quadratic program to ﬁnd an approximate gradient g new that closely matches ∂L ( x new ,y new )

Original: ∂θ while ensuring that the following constraint holds:
Revised: ∂θ while ensuring that the following constraint holds:

Original: [Equation]
Revised: [Equation]

Original: E R EPTILE A LGORITHM
Revised: E R EPTILE A LGORITHM

Original: We detail the standard Reptile algorithm from (Nichol & Schulman, 2018) in algorithm 2.
Revised: We detail the standard Reptile algorithm from (Nichol & Schulman, 2018) in algorithm 2.

Original: The sample function randomly samples s batches of size k from dataset D .
Revised: The sample function randomly samples s batches of size k from dataset D .

Original: The SGD function applies min-batch stochastic gradient descent over a batch of data given a set of current parameters and learning rate.
Revised: The SGD function applies min-batch stochastic gradient descent over a batch of data given a set of current parameters and learning rate.

Original: Algorithm 2 Reptile for Stationary Data procedure T RAIN ( D, θ, α, β, s, k ) while not done do // Draw batches from data:
Revised: Algorithm 2 Reptile for Stationary Data procedure T RAIN ( D, θ, α, β, s, k ) while not done do // Draw batches from data:

Original: B 1 , ..., B s ← sample ( D, s, k ) θ 0 ← θ for i = 1 , ..., s do θ i ← SGD ( B i , θ i − 1 , α ) end for //
Revised: B 1 , ..., B s ← sample ( D, s, k ) θ 0 ← θ for i = 1 , ..., s do θ i ← SGD ( B i , θ i − 1 , α ) end for //

Original: Reptile meta-update: θ ← θ 0 + β ( θ s − θ 0 ) end while return θ end procedure
Revised: Reptile meta-update: θ ← θ 0 + β ( θ s − θ 0 ) end while return θ end procedure

Original: F D ETAILS ON R ESERVOIR S AMPLING
Revised: F D ETAILS ON R ESERVOIR S AMPLING

Original: Throughout this paper we refer to updates to our memory M as M ← M ∪ { ( x, y ) } .
Revised: Throughout this paper we refer to updates to our memory M as M ← M ∪ { ( x, y ) } .

Original: We would like to now provide details on how we update our memory buffer using reservoir sampling as outlined in
Revised: We would like to now provide details on how we update our memory buffer using reservoir sampling as outlined in

Original: Vitter (1985) (algorithm 3).
Revised: Vitter (1985) (algorithm 3).

Original: Reservoir sampling solves the problem of keeping some limited number
Revised: Reservoir sampling solves the problem of keeping some limited number

Original: M of N total items seen before with equal probability MN when you don’t know what number N will be in advance.
Revised: M of N total items seen before with equal probability MN when you don’t know what number N will be in advance.

Original: The randomInteger function randomly draws an integer inclusively between the provided minimum and maximum values.
Revised: The randomInteger function randomly draws an integer inclusively between the provided minimum and maximum values.

Original: procedure R ESERVOIR ( M, N, x, y ) if M > N then
Revised: procedure R ESERVOIR ( M, N, x, y ) if M > N then

Original: M [ N ] ← ( x, y ) else j = randomInteger ( min = 0 , max = N ) if j < M then
Revised: M [ N ] ← ( x, y ) else j = randomInteger ( min = 0 , max = N ) if j < M then

Original: M [ j ] ← ( x, y ) end if end if return M end procedure
Revised: M [ j ] ← ( x, y ) end if end if return M end procedure

Original: G E XPERIENCE R EPLAY A LGORITHMS
Revised: G E XPERIENCE R EPLAY A LGORITHMS

Original: We detail the our variant of the experience replay in algorithm 4.
Revised: We detail the our variant of the experience replay in algorithm 4.

Original: This procedure closely follows recent enhancements discussed in Zhang & Sutton (2017);
Revised: This procedure closely follows recent enhancements discussed in Zhang & Sutton (2017);

Original: Riemer et al.
Revised: Riemer et al.

Original: (2017b;a)
Revised: (2017b;a)

Original: The sample function randomly samples k − 1 examples from the memory buffer M and interleaves them with the current example to form a single size k batch.
Revised: The sample function randomly samples k − 1 examples from the memory buffer M and interleaves them with the current example to form a single size k batch.

Original: The SGD function applies mini-batch stochastic gradient descent over a batch of data given a set of current parameters and learning rate.
Revised: The SGD function applies mini-batch stochastic gradient descent over a batch of data given a set of current parameters and learning rate.

Original: procedure T RAIN ( D, θ, α, k )
Revised: procedure T RAIN ( D, θ, α, k )

Original: M ← {} for t = 1 , ..., T do for ( x, y ) in D t do // Draw batch from buffer:
Revised: M ← {} for t = 1 , ..., T do for ( x, y ) in D t do // Draw batch from buffer:

Original: B ← sample ( x, y, k, M ) // Update parameters with mini-batch SGD: θ ← SGD ( B, θ, α ) //
Revised: B ← sample ( x, y, k, M ) // Update parameters with mini-batch SGD: θ ← SGD ( B, θ, α ) //

Original: Reservoir sampling memory update:
Revised: Reservoir sampling memory update:

Original: M ← M ∪ { ( x, y ) } (algorithm 3) end for end for return θ, M end procedure
Revised: M ← M ∪ { ( x, y ) } (algorithm 3) end for end for return θ, M end procedure

Original: Unfortunately, it is not straightforward to implement algorithm 4 in all circumstances.
Revised: Unfortunately, it is not straightforward to implement algorithm 4 in all circumstances.

Original: In particular, it depends whether the neural network architecture is single headed (sharing an output layer and output space among all tasks) or multi-headed (where each task gets its own unique output space).
Revised: In particular, it depends whether the neural network architecture is single headed (sharing an output layer and output space among all tasks) or multi-headed (where each task gets its own unique output space).

Original: In multi-headed settings, it is common to consider the tasks in separate batches and to equally weight the sampled tasks during each update.
Revised: In multi-headed settings, it is common to consider the tasks in separate batches and to equally weight the sampled tasks during each update.

Original: This results in training the parameters evenly for each task and is particularly important so we pay equal attention to each set of task speciﬁc parameters.
Revised: This results in training the parameters evenly for each task and is particularly important so we pay equal attention to each set of task speciﬁc parameters.

Original: We detail an approach that separates tasks into sub-batches for a balanced update in algorithm 5.
Revised: We detail an approach that separates tasks into sub-batches for a balanced update in algorithm 5.

Original: Here
Revised: Here

Original: L is the loss given a set of parameters over a batch of data and SGD applies a mini-batch gradient descent update rule over a loss given a set of parameters and learning rate.
Revised: L is the loss given a set of parameters over a batch of data and SGD applies a mini-batch gradient descent update rule over a loss given a set of parameters and learning rate.

Original: procedure T RAIN ( D, θ, α, k )
Revised: procedure T RAIN ( D, θ, α, k )

Original: M ← {} for t = 1 , ..., T do for ( x, y ) in D t do // Draw batch from buffer:
Revised: M ← {} for t = 1 , ..., T do for ( x, y ) in D t do // Draw batch from buffer:

Original: B ← sample ( x, y, k, M ) //
Revised: B ← sample ( x, y, k, M ) //

Original: Compute balanced loss across tasks loss = 0.
Revised: Compute balanced loss across tasks loss = 0.

Original: for task in B do loss = loss + L ( B [ task ] , θ ) end for //
Revised: for task in B do loss = loss + L ( B [ task ] , θ ) end for //

Original: Update parameters with mini-batch SGD: θ ← SGD ( loss, θ, α ) //
Revised: Update parameters with mini-batch SGD: θ ← SGD ( loss, θ, α ) //

Original: Reservoir sampling memory update:
Revised: Reservoir sampling memory update:

Original: M ← M ∪ { ( x, y ) } (algorithm 3) end for end for return θ, M end procedure
Revised: M ← M ∪ { ( x, y ) } (algorithm 3) end for end for return θ, M end procedure

Original: Our experiments demonstrate that both variants of experience replay are very effective for continual learning.
Revised: Our experiments demonstrate that both variants of experience replay are very effective for continual learning.

Original: Meanwhile, each performs signiﬁcantly better than the other on some datasets and settings.
Revised: Meanwhile, each performs signiﬁcantly better than the other on some datasets and settings.

Original: H T HE V ARIANTS OF MER
Revised: H T HE V ARIANTS OF MER

Original: We detail two additional variants of MER (algorithm 1) in algorithms 6 and 7.
Revised: We detail two additional variants of MER (algorithm 1) in algorithms 6 and 7.

Original: The sample function takes on a slightly different meaning in each variant of the algorithm.
Revised: The sample function takes on a slightly different meaning in each variant of the algorithm.

Original: In algorithm 1 sample is used to produce s batches consisting of k − 1 random examples from the memory buffer and the current example.
Revised: In algorithm 1 sample is used to produce s batches consisting of k − 1 random examples from the memory buffer and the current example.

Original: In algorithm 6 sample is used to produce one batch consisting of sk − s examples from the memory buffer and s copies of the current example.
Revised: In algorithm 6 sample is used to produce one batch consisting of sk − s examples from the memory buffer and s copies of the current example.

Original: In algorithm 7 sample is used to produce one batch consisting of k − 1 examples from the memory buffer.
Revised: In algorithm 7 sample is used to produce one batch consisting of k − 1 examples from the memory buffer.

Original: In algorithm 6, sample places the current example at the end of the batch.
Revised: In algorithm 6, sample places the current example at the end of the batch.

Original: Meanwhile, in algorithm 7, sample places the current example in a random location within the batch.
Revised: Meanwhile, in algorithm 7, sample places the current example in a random location within the batch.

Original: In contrast, the SGD function carries a common meaning across algorithms, applying stochastic gradient descent over a particular input and output given a set of current parameters and learning rate.
Revised: In contrast, the SGD function carries a common meaning across algorithms, applying stochastic gradient descent over a particular input and output given a set of current parameters and learning rate.

Original: Algorithm procedure T RAIN ( D, θ, α, γ, sk )
Revised: Algorithm procedure T RAIN ( D, θ, α, γ, sk )

Original: M ← {} for t = 1 , ..., T do for ( x, y ) in D t do // Draw batch from buffer:
Revised: M ← {} for t = 1 , ..., T do for ( x, y ) in D t do // Draw batch from buffer:

Original: B ← sample ( x, y, s, k, M ) θ 0 ← θ for i = 1 , ..., sk do x c , y c ← B i [ j ] θ i ← SGD ( x c , y c , θ i − 1 , α ) end for //
Revised: B ← sample ( x, y, s, k, M ) θ 0 ← θ for i = 1 , ..., sk do x c , y c ← B i [ j ] θ i ← SGD ( x c , y c , θ i − 1 , α ) end for //

Original: Reptile meta-update: θ ← θ 0 + γ ( θ sk − θ 0 ) //
Revised: Reptile meta-update: θ ← θ 0 + γ ( θ sk − θ 0 ) //

Original: Reservoir sampling memory update:
Revised: Reservoir sampling memory update:

Original: M ← M ∪ { ( x, y ) } (algorithm 3) end for end for return θ, M end procedure procedure T RAIN
Revised: M ← M ∪ { ( x, y ) } (algorithm 3) end for end for return θ, M end procedure procedure T RAIN

Original: ( D, θ, α, γ, s, k )
Revised: ( D, θ, α, γ, s, k )

Original: M ← {} for t = 1 , ..., T do for ( x, y ) in D t do // Draw batch from buffer:
Revised: M ← {} for t = 1 , ..., T do for ( x, y ) in D t do // Draw batch from buffer:

Original: B, index ← sample ( k − 1 , M ) θ 0 ← θ //
Revised: B, index ← sample ( k − 1 , M ) θ 0 ← θ //

Original: SGD on individual samples from batch: for i = 1 , ..., k − 1 do x c , y c ← B i [ j ] if j = index //
Revised: SGD on individual samples from batch: for i = 1 , ..., k − 1 do x c , y c ← B i [ j ] if j = index //

Original: High learning rate SGD on current example:
Revised: High learning rate SGD on current example:

Original: θ k ← SGD ( x, y, θ k − 1 , sα ) else θ i ← SGD ( x c , y c , θ i − 1 , α ) end for //
Revised: θ k ← SGD ( x, y, θ k − 1 , sα ) else θ i ← SGD ( x c , y c , θ i − 1 , α ) end for //

Original: Reptile meta-update: θ ← θ 0 + γ ( θ k − θ 0 ) //
Revised: Reptile meta-update: θ ← θ 0 + γ ( θ k − θ 0 ) //

Original: Reservoir sampling memory update:
Revised: Reservoir sampling memory update:

Original: M ← M ∪ { ( x, y ) } (algorithm 3) end for end for return θ, M end procedure
Revised: M ← M ∪ { ( x, y ) } (algorithm 3) end for end for return θ, M end procedure

Original: I D ERIVING THE E FFECTIVE O BJECTIVE OF MER
Revised: I D ERIVING THE E FFECTIVE O BJECTIVE OF MER

Original: We would like to derive what objective Meta-Experience Replay (algorithm 1) approximates and show that it is approximately the same objective from algorithms 6 and 7.
Revised: We would like to derive what objective Meta-Experience Replay (algorithm 1) approximates and show that it is approximately the same objective from algorithms 6 and 7.

Original: We follow conventions from Nichol & Schulman (2018) and ﬁrst demonstrate what happens to the effective gradients computed by the algorithm in the most trivial case.
Revised: We follow conventions from Nichol & Schulman (2018) and ﬁrst demonstrate what happens to the effective gradients computed by the algorithm in the most trivial case.

Original: As in Nichol & Schulman (2018), this allows us to extrapolate an effective gradient that is a function of the number of steps taken.
Revised: As in Nichol & Schulman (2018), this allows us to extrapolate an effective gradient that is a function of the number of steps taken.

Original: We can then consider the effective loss function that results in this gradient.
Revised: We can then consider the effective loss function that results in this gradient.

Original: Before we begin, let us deﬁne the following terms from Nichol & Schulman (2018):
Revised: Before we begin, let us deﬁne the following terms from Nichol & Schulman (2018):

Original: [Equation]
Revised: [Equation]

Original: In Nichol & Schulman (2018) they consider the effective gradient across one loop of reptile with size k = 2 .
Revised: In Nichol & Schulman (2018) they consider the effective gradient across one loop of reptile with size k = 2 .

Original: As we have both an outer loop of Reptile applied across batches and an inner loop applied within the batch to consider, we start with a setting where the number of batches s = 2 and the number of examples per batch k = 2 .
Revised: As we have both an outer loop of Reptile applied across batches and an inner loop applied within the batch to consider, we start with a setting where the number of batches s = 2 and the number of examples per batch k = 2 .

Original: Let’s recall from the original paper that the gradients of
Revised: Let’s recall from the original paper that the gradients of

Original: Reptile with k = 2 was:
Revised: Reptile with k = 2 was:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: So, we can also consider the gradients of Reptile if we had 4 examples in one big batch (algorithm 6) as opposed to 2 batches of 2 examples:
Revised: So, we can also consider the gradients of Reptile if we had 4 examples in one big batch (algorithm 6) as opposed to 2 batches of 2 examples:

Original: [Equation]
Revised: [Equation]

Original: Now we can consider the case for MER where we deﬁne the parameter values as follows extending algorithm 1 where A stands for across batches and W stands for within batches:
Revised: Now we can consider the case for MER where we deﬁne the parameter values as follows extending algorithm 1 where A stands for across batches and W stands for within batches:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: g MER the gradient of Meta-Experience Replay can thus be deﬁned analogously to the gradient of
Revised: g MER the gradient of Meta-Experience Replay can thus be deﬁned analogously to the gradient of

Original: Reptile as:
Revised: Reptile as:

Original: [Equation]
Revised: [Equation]

Original: By simply applying Reptile from equation 15 we can derive the value of the parameters θ A 1 after updating with Reptile within the ﬁrst batch in terms of the original parameters θ 0 :
Revised: By simply applying Reptile from equation 15 we can derive the value of the parameters θ A 1 after updating with Reptile within the ﬁrst batch in terms of the original parameters θ 0 :

Original: [Equation]
Revised: [Equation]

Original: By subbing equation 26 into equation 23 we can see that:
Revised: By subbing equation 26 into equation 23 we can see that:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We can express g 10 in terms of the initial point, by considering a Taylor expansion following the
Revised: We can express g 10 in terms of the initial point, by considering a Taylor expansion following the

Original: Reptile paper:
Revised: Reptile paper:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then substituting in for θ W 10 we express g 10 in terms of θ 0 :
Revised: Then substituting in for θ W 10 we express g 10 in terms of θ 0 :

Original: [Equation]
Revised: [Equation]

Original: We can then rewrite g 11 by taking a Taylor expansions with respect to θ W 10 :
Revised: We can then rewrite g 11 by taking a Taylor expansions with respect to θ W 10 :

Original: [Equation]
Revised: [Equation]

Original: Taking another Taylor expansion we ﬁnd that we can transform our expression for the Hessian:
Revised: Taking another Taylor expansion we ﬁnd that we can transform our expression for the Hessian:

Original: [Equation]
Revised: [Equation]

Original: We can analogously also transform our expression our expression for g 1011 :
Revised: We can analogously also transform our expression our expression for g 1011 :

Original: [Equation]
Revised: [Equation]

Original: Substituting for θ W 10 in terms of θ
Revised: Substituting for θ W 10 in terms of θ

Original: [Equation]
Revised: [Equation]

Original: We then substitute equation 31, equation 33, and equation 29 into equation 34:
Revised: We then substitute equation 31, equation 33, and equation 29 into equation 34:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Finally, we have all of the terms we need to express θ A 

Revised: Finally, we have all of the terms we need to express θ A

Original: and we can then derive an expression for the
Revised: and we can then derive an expression for the

Original: MER gradient g MER :
Revised: MER gradient g MER :

Original: [Equation]
Revised: [Equation]

Original: This equation is quite interesting and very similar to equation 16.
Revised: This equation is quite interesting and very similar to equation 16.

Original: As we would like to approximate the same objective, we can remove one hyperparameter from our model by setting β = 1 .
Revised: As we would like to approximate the same objective, we can remove one hyperparameter from our model by setting β = 1 .

Original: This yields:
Revised: This yields:

Original: [Equation]
Revised: [Equation]

Original: Indeed, with β set to equal 1, we have shown that the gradient of MER is the same as one loop of Reptile with a number of steps equal to the total number of examples in all batches of MER (algorithm 6) if the current example is mixed in with the same proportion.
Revised: Indeed, with β set to equal 1, we have shown that the gradient of MER is the same as one loop of Reptile with a number of steps equal to the total number of examples in all batches of MER (algorithm 6) if the current example is mixed in with the same proportion.

Original: If we include in the current example for s of sk examples in our meta-replay batch, it gets the same overall priority in both cases which is s times larger than that of a random example drawn from the buffer.
Revised: If we include in the current example for s of sk examples in our meta-replay batch, it gets the same overall priority in both cases which is s times larger than that of a random example drawn from the buffer.

Original: As such, we can also optimize an equivalent gradient using algorithm 7 because it uses a factor s to increase the priority of the gradient given to the current example.
Revised: As such, we can also optimize an equivalent gradient using algorithm 7 because it uses a factor s to increase the priority of the gradient given to the current example.

Original: While β = 1 is an interesting special case of MER in algorithm 1, in general we ﬁnd it can be useful to set β to be a value smaller than 1.
Revised: While β = 1 is an interesting special case of MER in algorithm 1, in general we ﬁnd it can be useful to set β to be a value smaller than 1.

Original: In fact, in our experiments we consider the case when β is smaller than 1 and γ = 1 .
Revised: In fact, in our experiments we consider the case when β is smaller than 1 and γ = 1 .

Original: The success of this approach makes sense because the higher order terms in the Taylor expansion that reﬂect the mismatch between parameters across replay batches disturb the learning process.
Revised: The success of this approach makes sense because the higher order terms in the Taylor expansion that reﬂect the mismatch between parameters across replay batches disturb the learning process.

Original: By setting β to a value below 1 we can reduce our comparative weighting on promoting inter batch gradient similarities rather than intra batch gradient similarities.
Revised: By setting β to a value below 1 we can reduce our comparative weighting on promoting inter batch gradient similarities rather than intra batch gradient similarities.

Original: It was noted in (Nichol & Schulman, 2018) that the following equality holds if the examples and order are random:
Revised: It was noted in (Nichol & Schulman, 2018) that the following equality holds if the examples and order are random:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In our work to make sure this equality holds in an online setting, we must take multiple precautions as noted in the main text.
Revised: In our work to make sure this equality holds in an online setting, we must take multiple precautions as noted in the main text.

Original: The issue is that examples are received in a non-stationary sequence so when applied in a continual learning setting the order is not totally random or arbitrary as in the original Reptile work.
Revised: The issue is that examples are received in a non-stationary sequence so when applied in a continual learning setting the order is not totally random or arbitrary as in the original Reptile work.

Original: We address this by maintaining our buffer using reservoir sampling, which ensures that any example seen before has a probability 1 N of being a particular element in the buffer.
Revised: We address this by maintaining our buffer using reservoir sampling, which ensures that any example seen before has a probability 1 N of being a particular element in the buffer.

Original: We also randomly select over these elements to form a batch.
Revised: We also randomly select over these elements to form a batch.

Original: As this makes the order largely arbitrary to the extent that our buffer includes all examples seen, we are approximating the random ofﬂine setting from the original Reptile paper.
Revised: As this makes the order largely arbitrary to the extent that our buffer includes all examples seen, we are approximating the random ofﬂine setting from the original Reptile paper.

Original: As such we can view the gradients in equation 16 and equation 36 as leading to approximately the following objective function:
Revised: As such we can view the gradients in equation 16 and equation 36 as leading to approximately the following objective function:

Original: [Equation]
Revised: [Equation]

Original: This is precisely equation 7 in the main text.
Revised: This is precisely equation 7 in the main text.

Original: J S UPERVISED C ONTINUAL L IFELONG L EARNING
Revised: J S UPERVISED C ONTINUAL L IFELONG L EARNING

Original: For the supervised continual learning benchmarks leveraging MNIST Rotations and MNIST Permutations, following conventions, we use a two layer MLP architecture for all models with 100 hidden units in each layer.
Revised: For the supervised continual learning benchmarks leveraging MNIST Rotations and MNIST Permutations, following conventions, we use a two layer MLP architecture for all models with 100 hidden units in each layer.

Original: We also model our hyperparameter search after Lopez-Paz & Ranzato (2017) while providing statistics for each model across 5 random seeds.
Revised: We also model our hyperparameter search after Lopez-Paz & Ranzato (2017) while providing statistics for each model across 5 random seeds.

Original: For Omniglot, following Vinyals et al.
Revised: For Omniglot, following Vinyals et al.

Original: we scale the images to 28x28 and use an architecture that consists of a stack of 4 modules before a fully connected softmax layer.
Revised: we scale the images to 28x28 and use an architecture that consists of a stack of 4 modules before a fully connected softmax layer.

Original: Each module includes a 3x
Revised: Each module includes a 3x

Original: J. H YPERPARAMETER S EARCH
Revised: J. H YPERPARAMETER S EARCH

Original: Here we report the hyper-parameter grids that we searched over in our experiments.
Revised: Here we report the hyper-parameter grids that we searched over in our experiments.

Original: The best values forthe MNIST Rotations (ROT) at each buffer size (ROT-5120, ROT-500, ROT-200), MNIST Permutations (PERM) at each buffer size (PERM-5120, PERM-500, PERM-200), Many Permutations (MANY) at each buffer size (MANY-5120, MANY-500), and Omniglot (OMNI) at each buffer size (OMNI-5120, OMNI-500) are noted accordingly in parenthesis.
Revised: We note in parenthesis the best values for MNIST Rotations (ROT) at each buffer size (ROT-5120, ROT-500, ROT-200), MNIST Permutations (PERM) at each buffer size (PERM-5120, PERM-500, PERM200), Many Permutations (MANY) at each buffer size (MANY-5120, MANY-500), and Omniglot (OMNI) at each buffer size (OMNI-5120, OMNI-500).

Original: • Online Learning – learning rate: [0.0001, 0.0003, 0.001 (PERM, ROT), 0.003 (MANY), 0.01, 0.03, 0.
Revised: • Online Learning – learning rate: [0.0001, 0.0003 (ROT), 0.001, 0.003 (PERM, MANY), 0.01, 0.03, 0.

Original: (OMNI)] •  EWC – learning rate: [0.001 (OMNI), 0.003 (MANY), 0.01, 0.03 (ROT, PERM), 0.1, 0.3, 1.0] – regularization: [1, 3 (MANY),  10 (OMNI), 30, 100, 300 (ROT, PERM), 1000, 3000, 10000, 30000] • GEM – learning rate: [0.001, 0.003 (MANY-500), 0.01 (ROT, PERM, OMNI, MANY-5120), 0.03, 0.1, 0.3, 1.0] – memory strength ( γ ): [0.0 (PERM-500, MANY-500), 0.1 (PERM-200, MANY-200), 0.5 (OMNI), 1.0 (ROT-5120, ROT-500, ROT-200, PERM-5120)]
Revised: (OMNI)] • Independent Model Per Task – learning rate: [0.0001, 0.0003, 0.001, 0.003, 0.01 (ROT, PERM, MANY), 0.03, 0.1] • Task Speciﬁc Input Layer – learning rate: [0.0001, 0.0003, 0.001, 0.003, 0.01 (ROT, PERM), 0.03, 0.1] • EWC – learning rate: [0.001 (ROT, OMNI), 0.003 (MANY), 0.01 (PERM), 0.03, 0.1, 0.3, 1.0] – regularization: [1 (MANY), 3, 10 (PERM, OMNI), 30, 100 (ROT), 300, 1000, 3000, 10000, 30000] • GEM – learning rate: [0.001, 0.003 (MANY-500), 0.01 (ROT, PERM, OMNI, MANY-5120), 0.03, 0.1, 0.3, 1.0] – memory strength ( γ ): [0.0 (ROT-500, ROT-200, PERM-200, MANY-5120), 0.

Original:  • Experience Replay (Algorithm 4) –learning rate: [0.00003, 0.0001 (ROT-200), 0.0003, 0.001, 0.003, 0.01, 0.03, 0.
Revised: (MANY-500), 0.5 (OMNI), 1.0 (ROT-5120, PERM-5120, PERM-500)] • Experience Replay (Algorithm 4) –

Original: (ROT-5120, ROT-500, PERM, MANY)] – batch size ( k -1): [5 (ROT-500), 10 (ROT-200, PERM-500, PERM-200), 25 (ROT5120, PERM-5120, MANY), 50, 100, 250] • Experience Replay (Algorithm 5) – learning rate: [0.00003, 0.0001, 0.0003, 0.001, 0.003 (MANY), 0.01 (ROT, PERM), 0.03, 0.1] – batch size ( k -1): [5, 10 (ROT-500, PERM-200), 25 (ROT-5120, PERM-5120, PERM500), 50 (MANY-5120, ROT-200), 100, 250] • Meta-Experience Replay (Algorithm 1) – learning rate ( α ): [0.01 (OMNI-5120), 0.03 (ROT, PERM, MANY), 0.1 (OMNI-500)] – across batch meta-learning rate ( γ ): 1. – within batch meta-learning rate ( β ): [0.03 (ROT-5120, ROT-200, PERM-5120, PERM-200, MANY), 0.1 (ROT-500, PERM-500), 0.3, 1.0 (OMNI)] – batch size ( k -1): [5 (MANY-500, OMNI-500), 10, 25 (PERM-500, PERM-200, OMNI-5120), 50 (PERM-5120, MANY-5120), 100 (ROT)] – number of batches per example: [1, 2 (PERM-500, OMNI-500), 5 (ROT, PERM-200, OMNI-5120, MANY-5120), 10 (PERM-5120, MANY-500)]
Revised: learning rate: [0.00003, 0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1 (ROT, PERM, MANY)] – batch size ( k -1): [5 (ROT-500), 10 (ROT-200, PERM-500, PERM-200), 25 (ROT5120, PERM-5120, MANY), 50, 100, 250] • Experience Replay (Algorithm 5) – learning rate: [0.00003, 0.0001, 0.0003, 0.001, 0.003 (MANY-5120), 0.01 (ROT-500, ROT-200, PERM, MANY-500), 0.03 (ROT-5120), 0.1] – batch size ( k -1): [5 (MANY-500), 10 (PERM-200, MANY-5120), 25 (PERM-5120, PERM-500), 50 (ROT-200), 100 (ROT-5120, ROT-500), 250] • Meta-Experience Replay (Algorithm 1) – learning rate ( α ): [0.01 (OMNI-5120), 0.03 (ROT-5120, PERM, MANY-500), 0. (ROT-500, ROT-200, OMNI-500)] – across batch meta-learning rate ( γ ): 1. – within batch meta-learning rate ( β ): [0.01 (ROT-500, ROT-200, MANY-5120), 0.

Original:  • Meta-Experience Replay (Algorithm 6) – learning rate ( α ): [0.01 (PERM-500), 0.03 (ROT, PERM-5120, PERM-200, MANY), 0.1 ] – meta-learning rate ( γ ): [0.03 (ROT-200), 0.1 (ROT-5120,ROT-500, PERM-5120, MANY), 0.3(PERM-200), 0.6 (PERM-500),  1.0] – batch size ( k -1): [5, 10, 25 (MANY), 50 (ROT-200, PERM-500), 100 (ROT-5120, PERM-200), 250 (ROT-500)] – number of batches per example:• Meta-Experience Replay (Algorithm 7) – learning rate ( α ): [0.01 (PERM-5120, PERM-500), 0.03 (ROT, PERM-200, MANY), 0.1] – within batch meta-learning rate ( γ ): [0.03 (ROT, MANY-5120), 0.1 (PERM, MANY500), 0.3, 1.0]
Revised: (ROT-5120, PERM, MANY-500), 0.1, 0.3, 1.0 (OMNI)] – batch size ( k -1): [5 (MANY, OMNI-500), 10 (ROT-500, ROT-200, PERM-200),(PERM-500, OMNI-5120), 50, 100 (ROT-5120, PERM-5120)] – number of batches per example: [1, 2 (OMNI-500), 5 (ROT-200, OMNI-5120),(ROT-5120, ROT-500, PERM, MANY)] • Meta-Experience Replay (Algorithm 6) – learning rate ( α ): [0.01, 0.03 (ROT-5120, PERM-5120, PERM-500, MANY-5120), 0.1 (ROT-500, ROT-200, PERM-200, MANY-500)] – meta-learning rate ( γ ): [0.03 (ROT-500, ROT-200, PERM-200, MANY-500), 0. (ROT-5120, PERM-5120, MANY-5120), 0.3 (PERM-500), 0.6, 1.0] – batch size ( k -1): [5 (PERM-200, MANY-500), 10 (ROT-500, PERM-500) 25 (ROT200, MANY-5120), 50 (PERM-5120), 100 (ROT-5120), 250]

Original: [Table]
Revised: [Table]

Original: –  batch size ( k -1): [5 (PERM-200, MANY-500), 10, 25 (PERM-500), 50 (ROT-200, ROT-500, MANY-5120), 100 (ROT-5120, PERM-5120)] – current example learning rate multiplier ( s ): [1, 2 (PERM-200, MANY-500), 5 (ROT500, ROT-200), 10 (ROT-5120, PERM-5120, PERM-500, MANY-5120)]
Revised: – number of batches per example:• Meta-Experience Replay (Algorithm 7) – learning rate ( α ): [0.01 (PERM-5120, PERM-500), 0.03 (ROT, PERM-200, MANY), 0.1] – within batch meta-learning rate ( γ ): [0.03 (ROT, MANY), 0.1 (PERM), 0.3, 1.0] – batch size ( k -1): [5 (PERM-200, MANY-500), 10, 25 (PERM-500), 50 (ROT-200, ROT-500, MANY-5120), 100 (ROT-5120, PERM-5120)] – current example learning rate multiplier ( s ): [1, 2 (PERM-200), 5 (ROT), 10 (PERM5120, PERM-500, MANY)]

Original: K F ORWARD T RANSFER AND I NTERFERENCE
Revised: K F ORWARD T RANSFER AND I NTERFERENCE

Original: Forward transfer was a metric deﬁned in Lopez-Paz & Ranzato (2017) based on the average increased performance on a task relative to performance at random initialization before training on that task.
Revised: Forward transfer was a metric deﬁned in Lopez-Paz & Ranzato (2017) based on the average increased performance on a task relative to performance at random initialization before training on that task.

Original: Unfortunately, this metric does not make much sense for tasks like MNIST Permutations where inputs are totally uncorrelated across tasks or Omniglot where outputs are totally uncorrelated across tasks.
Revised: Unfortunately, this metric does not make much sense for tasks like MNIST Permutations where inputs are totally uncorrelated across tasks or Omniglot where outputs are totally uncorrelated across tasks.

Original: As such, we only provide performance for this metric on MNIST Rotations in Table 5.
Revised: As such, we only provide performance for this metric on MNIST Rotations in Table 5.

Original: L A BLATION E XPERIMENTS
Revised: L A BLATION E XPERIMENTS

Original: We plot our detailed ablation results in Table 6.
Revised: We plot our detailed ablation results in Table 6.

Original: In order to consider a version of GEM that uses reservoir sampling, we maintain our buffer the same way that we do for experience replay and
Revised: In order to consider a version of GEM that uses reservoir sampling, we maintain our buffer the same way that we do for experience replay and

Original: MER.
Revised: MER.

Original: We consider everything in the buffer to be old data and solve the GEM quadratic program so that the loss is not increased on this data.
Revised: We consider everything in the buffer to be old data and solve the GEM quadratic program so that the loss is not increased on this data.

Original: We found that considering the task level gradient directions did not lead to improvements.
Revised: We found that considering the task level gradient directions did not lead to improvements.

Original: MC ONTINUAL R EINFORCEMENT L EARNING
Revised: M R EPRODUCIBILITY OF R ESULTS

Original: We detail the application of MER  to deep Q-learning in algorithm 8, using notation from Mnih et al.
Revised: While the results so far have provided substantial evidence of the beneﬁts of MER for continual learning, one potential concern with our experimental protocol in Appendix J.1 is that the larger hyperparameter search space used for MER may artiﬁcially inﬂate improvements given typical run to run variation. To validate that this is not the case, we have run extensive additional experiments in this section to see how the model performs across different random seeds and machines. The codebase presents some inherent stochasticity across runs. As such, in Tables 7, 8, and 9 we report two levels of generalization for a set of hyperparameters beyond the conﬁguration of an individual run.

Revised: In the Same Seeds column, we report the results for the original 5 model seeds (0-4) deployed on different machines.

Revised: In the Different Seeds column, we report the results for a different 25 model seeds (5-29) also deployed on different machines.

Revised: In all cases, we see that there are quantitative differences generalizing across seeds and machines.

Revised: However, new settings do not always result in lower performance.

Revised: Additionally, the differences are not qualitative in nature.

Revised: In fact, in every setting we come to approximately the same qualitative conclusions about how each model performs.

Original: [Table]
Revised: [Table]

Original: with Tasks and Adam
Revised: with Tasks and Adam

Original: [Table]
Revised: [Table]

Original: ER with Tasks and RMSProp
Revised: ER with Tasks and RMSProp

Original: [Table]
Revised: [Table]

Original: Hinton et al., 2012)
Revised: Hinton et al., 2012)

Original: [Table]
Revised: [Table]

Original: ER with Tasks and GEM Style Buffer
Revised: ER with Tasks and GEM Style Buffer

Original: [Table]
Revised: [Table]

Original: Lopez-Paz & Ranzato, 2017)
Revised: Lopez-Paz & Ranzato, 2017)

Original: [Table]
Revised: [Table]

Original: GEM
Revised: GEM

Original: [Table]
Revised: with

Original: Reservoir Sampling
Revised: Reservoir Sampling

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ER with SGD (algorithm 4)

Original: [Table]
Revised: [Table]

Revised: ER with Tasks and SGD (algorithm 5)

Revised: [Table]

Revised: MER (algorithm 1)

Revised: [Table]

Original: procedure DQN-MER( env, frameLimit, θ, α, β, γ, steps, k, E Q ) //
Revised: MER (algorithm 6) [Table] algorithm 7) [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: N

Revised: C ONTINUAL

Revised: R EINFORCEMENT L EARNING

Revised: We detail the application of MER to deep Q-learning in algorithm 8, using notation from Mnih et al.

Revised: procedure DQN-MER( env, frameLimit, θ, α, β, γ, steps, k, E Q ) //

Original: Initialize action-value function Q with parameters θ :
Revised: Initialize action-value function Q with parameters θ :

Original: Q ← Q ( θ ) // Initialize action-value function ˆ Q with the same parameters ˆ θ = θ : ˆ Q ← ˆ Q (ˆ θ )
Revised: Q ← Q ( θ ) // Initialize action-value function ˆ Q with the same parameters ˆ θ = θ : ˆ Q ← ˆ Q (ˆ θ )

Original: =
Revised: =

Original: ˆ Q ( θ ) //
Revised: ˆ Q ( θ ) //

Original: Initialize experience replay buffer:
Revised: Initialize experience replay buffer:

Original: M ← {}
Revised: M ← {}

Original: M.age ←while M.age ≤ frameLimit do // Begin new episode: env.reset () //
Revised: M.age ←while M.age ≤ frameLimit do // Begin new episode: env.reset () //

Original: Initialize the s state with the initial observation: while episode not done do // Select with probability p an action a from set of possible actions: a = (cid:26) random selection of action ˆ a p ≤ (cid:15) arg max a (cid:48) Q ( s t , a (cid:48) ; θ ) p > (cid:15) //
Revised: Initialize the s state with the initial observation: while episode not done do // Select with probability p an action a from set of possible actions: a = (cid:26) random selection of action ˆ a p ≤ (cid:15) arg max a (cid:48) Q ( s t , a (cid:48) ; θ ) p > (cid:15) //

Original: Perform the action a in the environment: s (cid:48) , r t ← env.step ( s, a ) //
Revised: Perform the action a in the environment: s (cid:48) , r t ← env.step ( s, a ) //

Original: Store current transition with reward r :
Revised: Store current transition with reward r :

Original: M ← M ∪ { ( s, a, r, s (cid:48) ) } (algorithm 3)
Revised: M ← M ∪ { ( s, a, r, s (cid:48) ) } (algorithm 3)

Original: B 1 , ..., B steps ← sample ( s, a, r, s (cid:48) , steps, k, M ) //
Revised: B 1 , ..., B steps ← sample ( s, a, r, s (cid:48) , steps, k, M ) //

Original: Store current weights: θ A 0 ← θ for i = 1 , ..., steps do θ Wi, 0 ← θ for j = 1 , ..., k
Revised: Store current weights: θ A 0 ← θ for i = 1 , ..., steps do θ Wi, 0 ← θ for j = 1 , ..., k

Original: do //
Revised: do //

Original: Sample one set of processed sequences, actions, and rewards from M : s, a, r, s (cid:48)
Revised: Sample one set of processed sequences, actions, and rewards from M : s, a, r, s (cid:48)

Original: =
Revised: =

Original: B i [ j ] y = (cid:40)
Revised: B i [ j ] y = (cid:40)

Original: r if ﬁnal frame in episode r + Γ max a ˆ Q ( s (cid:48) , a ; ˆ θ ) otherwise //
Revised: r if ﬁnal frame in episode r + Γ max a ˆ Q ( s (cid:48) , a ; ˆ θ ) otherwise //

Original: Optimize the Huber loss H ( y, Q ( s, a ; θ Wi,j − 1 )) :
Revised: Optimize the Huber loss H ( y, Q ( s, a ; θ Wi,j − 1 )) :

Original: L ← H ( y, Q ( s, a ; θ Wi,j − 1 ))
Revised: L ← H ( y, Q ( s, a ; θ Wi,j − 1 ))

Original: θ Wi,j ← θ Wi,j − 1 − α ∂L
Revised: θ Wi,j ← θ Wi,j − 1 − α ∂L

Original: ∂θ Wi,j −end for //
Revised: ∂θ Wi,j −end for //

Original: Within batch Reptile meta-update: θ θ Wi, [-
-] + β  θ Wi,k − θ Wi, 0 )  i  end for // batch Reptile meta update: 
Revised: Within batch Reptile meta-update: θ θ Wi, [-
-]0 + β ( θ Wi,k − θ Wi, 0 ) A i θ end for // batch Reptile meta update: θ

Revised: A 0 + γ θ Asteps − θ A 0 ) //

Original: // Reset target action value network
Revised: Reset target action value network

Original: Q to Q  E Q number of episodes: ˆ Q  Q end while end while return θ, M end procedure
Revised: Q to Q every E Q number of episodes: ˆ Q = Q end while end while return θ, M end procedure

Original: M. D ESCRIPTION OF C ATCHER AND F LAPPY B IRD
Revised: N.

Revised: D ESCRIPTION OF C ATCHER AND F LAPPY B IRD

Original: In Catcher, the agent controls a segment that lies horizontally in the bottom of the screen, i.e. a basket, and can move right or left, or stay still.
Revised: In Catcher, the agent controls a segment that lies horizontally in the bottom of the screen, i.e. a basket, and can move right or left, or stay still.

Original: The goal is to move the basket to catch as many pellets as possible.
Revised: The goal is to move the basket to catch as many pellets as possible.

Original: Missing a pellet results in losing one of the three available lives.
Revised: Missing a pellet results in losing one of the three available lives.

Original: Pellets emerge one by one from the top of the screen, and have a descending velocity that is ﬁxed for each task.
Revised: Pellets emerge one by one from the top of the screen, and have a descending velocity that is ﬁxed for each task.

Original: The reward and thus y axis in our Catcher experiments refers to the number of fruits caught during the full game span.
Revised: The reward and thus y axis in our Catcher experiments refers to the number of fruits caught during the full game span.

Original: In the case of the very popular game Flappy Bird, the agent has to navigate a bird in an environment full of pipes by deciding whether to ﬂap or not ﬂap its wings.
Revised: In the case of the very popular game Flappy Bird, the agent has to navigate a bird in an environment full of pipes by deciding whether to ﬂap or not ﬂap its wings.

Original: The pipes appear always in pairs, one from the bottom of the screen and one from the top of the screen, and have a gap that allows the bird to pass through them.
Revised: The pipes appear always in pairs, one from the bottom of the screen and one from the top of the screen, and have a gap that allows the bird to pass through them.

Original: Flapping the wings results in the bird ascending, otherwise the bird descends to ground naturally.
Revised: Flapping the wings results in the bird ascending, otherwise the bird descends to ground naturally.

Original: Both ascending and descending velocities are presets by the physics engine of the game.
Revised: Both ascending and descending velocities are presets by the physics engine of the game.

Original: The goal is to pass through many pairs of pipes as possible without hitting a pipe, as this results in losing the game.
Revised: The goal is to pass through many pairs of pipes as possible without hitting a pipe, as this results in losing the game.

Original: The scoring scheme in this game awards a point each time a pipe is crossed.
Revised: The scoring scheme in this game awards a point each time a pipe is crossed.

Original: Despite very simple mechanics, Flappy Bird has proven to be challenging for many humans.
Revised: Despite very simple mechanics, Flappy Bird has proven to be challenging for many humans.

Original: According to the original game scoring scheme, players with a score of 10 receive a Bronze medal; with 20 points, a Silver medal; 30 results in a Gold medal, and any score better than 40 is rewarded with a Platinum medal.
Revised: According to the original game scoring scheme, players with a score of 10 receive a Bronze medal; with 20 points, a Silver medal; 30 results in a Gold medal, and any score better than 40 is rewarded with a Platinum medal.

Original: M. DQN WITH M ETA -E XPERIENCE R EPLAY
Revised: N. DQN WITH M ETA -E XPERIENCE R EPLAY

Original: The DQN used to train on both games follows the classic architecture from Mnih et al.
Revised: The DQN used to train on both games follows the classic architecture from Mnih et al.

Original: (2015): it has a CNN consisting of 3 layers, the ﬁrst with 32 ﬁlters and an 8x8 kernel, the second layer with 64 ﬁlters and a 4x4 kernel, and a ﬁnal layer with 64 ﬁlters and a 3x3 kernel.
Revised: (2015): it has a CNN consisting of 3 layers, the ﬁrst with 32 ﬁlters and an 8x8 kernel, the second layer with 64 ﬁlters and a 4x4 kernel, and a ﬁnal layer with 64 ﬁlters and a 3x3 kernel.

Original: The CNN is followed by two fully connected layers.
Revised: The CNN is followed by two fully connected layers.

Original: A ReLU non-linearity was applied after each layer.
Revised: A ReLU non-linearity was applied after each layer.

Original: We limited the memory buffer size for our models to 50k transitions, which is roughly the proportion of the total memories used in the benchmark setting for our supervised learning tasks.
Revised: We limited the memory buffer size for our models to 50k transitions, which is roughly the proportion of the total memories used in the benchmark setting for our supervised learning tasks.

Original: M. P ARAMETERS FOR C ONTINUAL R EINFORCEMENT L EARNING E XPERIMENTS
Revised: N. P ARAMETERS FOR C ONTINUAL R EINFORCEMENT L EARNING E XPERIMENTS

Original: For the continual reinforcement learning setting we set the parameters using results from the experiments in the supervised setting as guidance.
Revised: For the continual reinforcement learning setting we set the parameters using results from the experiments in the supervised setting as guidance.

Original: Both Catcher and Flappy Bird used the same hyper parameters as detailed below with the obvious exception of the game-dependent parameter that deﬁnes each task.
Revised: Both Catcher and Flappy Bird used the same hyper parameters as detailed below with the obvious exception of the game-dependent parameter that deﬁnes each task.

Original: Models were trained to a maximum of 150k frames and 6 total tasks, switching every 25k frames. used different random seedsfor the initializationas stated in the
Revised: Models were trained to a maximum of 150k frames and 6 total tasks, switching every 25k frames random seeds the initialization

Original: • Game Parameters – Catcher: ∆ : 0 .
Revised: • Game Parameters – Catcher: ∆ : 0 .

Original: 03 (vertical velocity of pellet increased from default 0.608).
Revised: 03 (vertical velocity of pellet increased from default 0.608).

Original: – Flappy Bird: ∆ : − 5 (pipe gap decreased 5 from default 100).
Revised: – Flappy Bird: ∆ : − 5 (pipe gap decreased 5 from default 100).

Original: • Experience Replay – learning rate: 0.
Revised: • Experience Replay – learning rate: 0.

Original: – batch size ( k -1):• Meta-Experience Replay – learning rate ( α ): 0.
Revised: – batch size ( k -1):• Meta-Experience Replay – learning rate ( α ): 0.

Original: – within batch meta-learning rate ( β ):– across batch meta-learning rate ( γ ): 0.
Revised: – within batch meta-learning rate ( β ):– across batch meta-learning rate ( γ ): 0.

Original: – batch size ( k -1):– number of steps:– buffer size:
Revised: – batch size ( k -1):– number of steps:– buffer size:

Original: M. F URTHER C OMMENTS ON C ONTINUAL R EINFORCEMENT L EARNING E VALUATION
Revised: N. F URTHER C OMMENTS ON C ONTINUAL R EINFORCEMENT L EARNING E VALUATION

Original: Performance during training in continual learning for a non-stationary version of Flappy Bird is shown in Figure 5.
Revised: Performance during training in continual learning for a non-stationary version of Flappy Bird is shown in Figure 5.

Original: The graphs show averaged values over three validation episodes across three different seed initializations.
Revised: The graphs show averaged values over three validation episodes across three different seed initializations.

Original: Vertical grid lines on the frames axis indicate task switches.
Revised: Vertical grid lines on the frames axis indicate task switches.

Original: We have also conducted experiments performed using a DQN with reservoir sampling, ﬁnding that it consistently underperforms a DQN with typical recency based sampling.
Revised: We have also conducted experiments performed using a DQN with reservoir sampling, ﬁnding that it consistently underperforms a DQN with typical recency based sampling.

Original: A DQN with MER achieves approximately the asymptotic performance for the single task DQN by the end of training for most tasks.
Revised: A DQN with MER achieves approximately the asymptotic performance for the single task DQN by the end of training for most tasks.

Original: On the other hand, DQN with reservoir sampling achieves worse performance than the standard DQN, so it is clear that, in this particular setting where a later task is subsumed in
Revised: On the other hand, DQN with reservoir sampling achieves worse performance than the standard DQN, so it is clear that, in this particular setting where a later task is subsumed in

Original: [Figure]
Revised: [Figure]

Original: previous tasks, keeping easy experiences alone does not account for the beneﬁt of DQN-MER.
Revised: previous tasks, keeping easy experiences alone does not account for the beneﬁt of DQN-MER.

Original: DQN-MER performs better at training the ﬁrst task and experiences positive forward transfer for the remaining tasks over what is possible just training for 25k steps on a single task.
Revised: DQN-MER performs better at training the ﬁrst task and experiences positive forward transfer for the remaining tasks over what is possible just training for 25k steps on a single task.

Original: In most cases, DQN-MER achieves similar performance to the DQN that takes 1 million steps and achieves asymptotic performance.
Revised: In most cases, DQN-MER achieves similar performance to the DQN that takes 1 million steps and achieves asymptotic performance.

Original: On the ﬁrst three tasks DQN-MER performs better and it performs a bit worse for the later tasks where it has less time to train.
Revised: On the ﬁrst three tasks DQN-MER performs better and it performs a bit worse for the later tasks where it has less time to train.

Original: There does not seem to a price paid by DQNMER in these experiments for not forgetting the easier tasks.
Revised: There does not seem to a price paid by DQNMER in these experiments for not forgetting the easier tasks.

Original: Actually, we ﬁnd that on the ﬁnal tasks, DQN-MER achieves signiﬁcant transfer from easier tasks and achieves better performance than the single task DQN does after even training on those tasks alone for 150k steps.
Revised: Actually, we ﬁnd that on the ﬁnal tasks, DQN-MER achieves signiﬁcant transfer from easier tasks and achieves better performance than the single task DQN does after even training on those tasks alone for 150k steps.