Original: U SING HINDSIGHT TO ANCHOR PAST KNOWLEDGE IN CONTINUAL LEARNING
Revised: U SING HINDSIGHT TO ANCHOR PAST KNOWLEDGE IN CONTINUAL LEARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In continual learning, the learner faces a stream of data whose distribution changes over time.
Revised: In continual learning, the learner faces a stream of data whose distribution changes over time.

Original: Modern neural networks are known to suffer under this setting, as they quickly forget previously acquired knowledge.
Revised: Modern neural networks are known to suffer under this setting, as they quickly forget previously acquired knowledge.

Original: To address such catastrophic forgetting, state-of-the-art continual learning methods implement different types of experience replay , re-learning on past data stored in a small buffer known as episodic memory.
Revised: To address such catastrophic forgetting, many continual learning methods implement different types of experience replay , re-learning on past data stored in a small buffer known as episodic memory.

Original: In this work, we complement experience replay with a metalearning technique that we call “anchoring”: the learner updates its knowledge on the current task, while keeping predictions on some anchor points of past tasks intact.
Revised: In this work, we complement experience replay with a meta-learning technique that we call “anchoring”: the learner updates its knowledge on the current task, while keeping predictions on some anchor points of past tasks intact.

Original: These anchor points are learned using gradient-based optimization as to maximize forgetting of the current task, in hindsight , when the learner is ﬁne-tuned on the episodic memory of past tasks.
Revised: These anchor points are learned using gradient-based optimization as to maximize forgetting of the current task, in hindsight , when the learner is ﬁne-tuned on the episodic memory of past tasks.

Original: Experiments on several supervised learning benchmarks for continual learning demonstrate that our approach improves the state of the art in terms of both accuracy and forgetting metrics and for various sizes of episodic memories.
Revised: Experiments on several supervised learning benchmarks for continual learning demonstrate that our approach improves the standard experience replay in terms of both accuracy and forgetting metrics and for various sizes of episodic memories.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: This manuscript studies the problem of continual learning , where a machine learning model experiences a sequence of tasks.
Revised: This manuscript studies the problem of continual learning , where a machine learning model experiences a sequence of tasks.

Original: Each of these tasks is presented as a stream of input-output pairs, where each pair is drawn from the corresponding task probability distribution.
Revised: Each of these tasks is presented as a stream of input-output pairs, where each pair is drawn from the corresponding task probability distribution.

Original: Since the length of the learning experience is not speciﬁed a-priori, the learner can only assume a single pass over the data , and store nothing but a few examples into a ﬁxed-size episodic memory .
Revised: Since the length of the learning experience is not speciﬁed a-priori, the learner can only assume a single pass over the data , and store nothing but a few examples into a ﬁxed-size episodic memory .

Original: At any time during the lifetime of the model, predictions on examples from all tasks can be requested.
Revised: At all times during the lifetime of the model, predictions on examples from all tasks can be requested.

Original: Addressing continual learning is an important research problem, since it would bypass the long-obsolete assumption of “identically and independently distributed data” impeding progress towards artiﬁcial intelligence, and allow the deployment of models learning in-the-wild.
Revised: Addressing continual learning is an important research problem, since it would waive the long-obsolete assumption of “identically and independently distributed data” impeding progress towards artiﬁcial intelligence, and allow the deployment of models learning in-the-wild.

Original: However, continual learning presents one major challenge, catastrophic forgetting (McCloskey & Cohen, 1989).
Revised: However, continual learning presents one major challenge, catastrophic forgetting (McCloskey & Cohen, 1989).

Original: That is, as the learner experiences new tasks, it quickly forgets previously acquired knowledge.
Revised: That is, as the learner experiences new tasks, it quickly forgets previously acquired knowledge.

Original: This is a hindrance for state-of-the-art deep learning models, where all parameters are updated after observing each example.
Revised: This is a hindrance for state-of-the-art deep learning models, where all parameters are updated after observing each example.

Original: Continual learning has received increasing attention from the scientiﬁc community during the last decade.
Revised: Continual learning has received increasing attention from the scientiﬁc community during the last decade.

Original: The state of the art in algorithms for continual learning fall into three categories.
Revised: The state of the art in algorithms for continual learning fall into three categories.

Original: First, regularization-based approaches reduce forgetting by restricting the updates in parameters that were important for previous tasks (Kirkpatrick et al., 2016;
Revised: First, regularization-based approaches reduce forgetting by restricting the updates in parameters that were important for previous tasks (Kirkpatrick et al., 2016;

Original: Aljundi et al., 2018; Chaudhry et al., 2018; Nguyen et al., 2018).
Revised: Aljundi et al., 2018; Chaudhry et al., 2018; Nguyen et al., 2018).

Original: However, when the number of tasks are large, the regularization of past tasks becomes obsolete, leading to representation drift (Titsias et al., 2019).
Revised: However, when the number of tasks are large, the regularization of past tasks becomes obsolete, leading to representation drift (Titsias et al., 2019).

Original: Second, modular approaches (Rusu et al., 2016; Lee et al., 2017) add new modules to the learner as new tasks are learned.
Revised: Second, modular approaches (Rusu et al., 2016; Lee et al., 2017) add new modules to the learner as new tasks are learned.

Original: While modular architectures overcome forgetting by design, the memory complexity of these approaches scales with the number of tasks.
Revised: While modular architectures overcome forgetting by design, the memory complexity of these approaches scales with the number of tasks.

Original: Third, memory-based methods (Lopez-Paz & Ranzato, 2017; Hayes et al., 2018; Isele & Cosgun, 2018; Riemer et al., 2019; Chaudhry et al., 2019a), store a few examples from past tasks in a so-called episodic memory, to be revisited when training for a new task.
Revised: Third, memory-based methods (Lopez-Paz & Ranzato, 2017; Hayes et al., 2018; Isele & Cosgun, 2018; Riemer et al., 2019; Chaudhry et al., 2019a), store a few examples from past tasks in a so-called episodic memory, to be revisited when training for a new task.

Original: Memory-based methods are the reigning state of the art, but remain a far-cry from the achievable performance by a simple multi-task learning baseline accessing all data at once.
Revised: Memory-based methods are the reigning state of the art, but remain a far-cry from the achievable performance by a simple multi-task learning baseline accessing all data at once.

Original: Despite intense research efforts, such gap in performance renders the problem of continual learning an open research question.
Revised: Despite intense research efforts, such gap in performance renders the problem of continual learning an open research question.

Original: Contribution We propose Hindsight Anchor Learning (HAL), a continual learning approach to improve the performance of memory-based continual learning algorithms.
Revised: Contribution We propose Hindsight Anchor Learning (HAL), a continual learning approach to improve the performance of memory-based continual learning algorithms.

Original: HAL leverages metalearning to regularize the training objective with one representational point per class per task, called anchor .
Revised: HAL leverages metalearning to regularize the training objective with one representational point per class per task, called anchor .

Original: Since it is desirable to preserve the performance of a learner on a task throughout the entire continual learning experience, HAL constructs anchors as to maximize forgetting on the current task throughout the learning experience.
Revised: Since it is desirable to preserve the performance of a learner during its lifetime, HAL constructs anchors as to maximize forgetting on the current task throughout the learning experience.

Original: We estimate the amount of forgetting that the learner would suffer on those anchors when learning future tasks in hindsight : that is, by measuring forgetting on a temporary predictor that has been ﬁne-tuned on the episodic memories of past tasks.
Revised: We estimate the amount of forgetting that the learner would suffer on those anchors when learning future tasks in hindsight : that is, by measuring forgetting on a temporary predictor that has been ﬁne-tuned on the episodic memories of past tasks.

Original: Then, the main parameter update of HAL is to minimize the loss on the currently observed mini-batch, while keeping the predictions at all anchors invariant.
Revised: Then, the main parameter update of HAL is to minimize the loss on the currently observed mini-batch, while keeping the predictions at all anchors invariant.

Original: Results We compare HAL to EWC (Kirkpatrick et al., 2016), AGEM (Chaudhry et al., 2019a), experience replay (Hayes et al., 2018; Riemer et al., 2019), and MER (Riemer et al., 2019), across four standard benchmarks in continual learning (MNIST permutations, MNIST rotations, split CIFAR100, and split miniImageNet).
Revised: Results We compare HAL to EWC (Kirkpatrick et al., 2016), AGEM (Chaudhry et al., 2019a), experience replay (Hayes et al., 2018; Riemer et al., 2019), and MER (Riemer et al., 2019), across four standard benchmarks in continual learning (MNIST permutations, MNIST rotations, split CIFAR100, and split miniImageNet).

Original: In these experiments, HAL achieves state-of-the-art performance, improving accuracy by 5% and reducing forgetting by 20%.
Revised: In these experiments, HAL achieves state-of-the-art performance, improving accuracy by 5% and reducing forgetting by 20%.

Original: We show that these results hold for various sizes of episodic memories (between 1 and 5 examples per class per task).
Revised: We show that these results hold for various sizes of episodic memories (between 1 and 5 examples per class per task).

Original: Finally, we perform an ablation study to show that learning anchors in hindsight is a critical factor to achieve state-of-the-art-performance.
Revised: Finally, we perform an ablation study to show that learning anchors in hindsight is a critical factor to achieve state-of-the-art performance.

Original: We now begin our exposition by reviewing the continual learning setup.
Revised: We now begin our exposition by reviewing the continual learning setup.

Original: The rest of the manuscript then presents our new algorithm HAL (Section 3), showcases its empirical performance (Section 4), surveys the related literature (Section 5), and offers some concluding remarks (Section 6).
Revised: The rest of the manuscript then presents our new algorithm HAL (Section 3), showcases its empirical performance (Section 4), surveys the related literature (Section 5), and offers some concluding remarks (Section 6).

Original:  C ONTINUAL LEARNING SETUP
Revised:  C ONTINUAL LEARNING SETUP

Original: In continual learning, we experience a stream of data triplets ( x i , y i , t i ) , each containing an input x i , a target y i , and a task identiﬁer t i ∈ T = { 1 , . .
Revised: In continual learning, we experience a stream of data triplets ( x i , y i , t i ) , each containing an input x i , a target y i , and a task identiﬁer t i ∈ T = { 1 , . .

Original: , T } .
Revised: , T } .

Original: Each input-target pair ( x i , y i ) ∈
Revised: Each input-target pair ( x i , y i ) ∈

Original: X × Y t i is an identically and independently distributed example drawn from some unknown distribution
Revised: X × Y t i is an identically and independently distributed example drawn from some unknown distribution

Original: P ( X t
Revised: P ( X t

Original: i , Y t i ) , representing the t i -th learning task.
Revised: i , Y t i ) , representing the t i -th learning task.

Original: Without loss of generality, we assume that the tasks are experienced in order ( t i ≤ t j for all i ≤ j ), and that the number of total tasks T is not known a priori.
Revised: Without loss of generality, we assume that the tasks are experienced in order ( t i ≤ t j for all i ≤ j ), and that the number of total tasks T is not known a priori.

Original: Under this setup, our goal is to estimate a predictor f = ( w ◦ φ ) :
Revised: Under this setup, our goal is to estimate a predictor f = ( w ◦ φ ) :

Original: X × T → Y , composed by a featurizer φ : X → H and a classiﬁer w : H → Y , that minimizes the multi-task error
Revised: X × T → Y , composed by a featurizer φ : X → H and a classiﬁer w : H → Y , that minimizes the multi-task error

Original: [Equation]
Revised: [Equation]

Original: where Y = ∪ t ∈T Y t , and (cid:96) :
Revised: where Y = ∪ t ∈T Y t , and (cid:96) :

Original: Y × Y → R is a suitable loss function.
Revised: Y × Y → R is a suitable loss function.

Original: In the sequel, and similarly to prior literature in continual learning (Lopez-Paz & Ranzato, 2017;
Revised: In the sequel, and similarly to prior literature in continual learning (Lopez-Paz & Ranzato, 2017;

Original: Hayes et al., 2018; Riemer et al., 2019; Chaudhry et al., 2019a), we considerinﬁnite streams of data, which are experienced only once .
Revised: Hayes et al., 2018; Riemer et al., 2019; Chaudhry et al., 2019a), we consider streams of data, which are experienced only once .

Original: Therefore, the learner cannot store or revisit any but a small amount of data triplets chosen to be stored in its episodic memory M .
Revised: Therefore, the learner cannot store or revisit any but a small amount of data triplets chosen to be stored in its episodic memory M .

Original: More speciﬁcally, we consider tiny “ring” episodic memories, which contain the last m observed examples per class for each of the experienced tasks, where m ∈ { 1 , 3 , 5 } .
Revised: More speciﬁcally, we consider tiny “ring” episodic memories, which contain the last m observed examples per class for each of the experienced tasks, where m ∈ { 1 , 3 , 5 } .

Original: That is, considering as variables the number of experienced tasks t and examples n , we study continual learning algorithms with a O ( t ) memory footprint.
Revised: That is, considering as variables the number of experienced tasks t and examples n , we study continual learning algorithms with a O ( t ) memory footprint.

Original: Following Lopez-Paz & Ranzato (2017), we monitor two statistics to evaluate the quality of continual learning algorithms: ﬁnal average accuracy , and ﬁnal maximum forgetting .
Revised: Following Lopez-Paz & Ranzato (2017), we monitor two statistics to evaluate the quality of continual learning algorithms: ﬁnal average accuracy , and ﬁnal maximum forgetting .

Original: First, the ﬁnal average accuracy of a predictor is deﬁned as
Revised: First, the ﬁnal average accuracy of a predictor is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where a i,j denotes the test accuracy on task j after the model ﬁnished experiencing task i .
Revised: where a i,j denotes the test accuracy on task j after the model ﬁnished experiencing task i .

Original: That is, the ﬁnal average accuracy measures the test performance of the predictor at each of the tasks after the continual learning experience has ﬁnished.
Revised: That is, the ﬁnal average accuracy measures the test performance of the predictor at each of the tasks after the continual learning experience has ﬁnished.

Original: Second, the ﬁnal maximum forgetting is deﬁned asForgetting
Revised: Second, the ﬁnal maximum forgetting is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: that is, the decrease in performance at each of the tasks between their peak accuracy and their accuracy after the continual learning experience has ﬁnished.
Revised: that is, the decrease in performance at each of the tasks between their peak accuracy and their accuracy after the continual learning experience has ﬁnished.

Original: Finally,and following Chaudhry et al. (2019a), we use the ﬁrst k < T tasks to cross-validate the hyper-parameters of each of the considered continual learning algorithms.
Revised: Finally, following Chaudhry et al. (2019a) , we use the ﬁrst k < T tasks to cross-validate the hyperparameters of each of the considered continual learning algorithms.

Original: These ﬁrst k tasks are not considered when computing the ﬁnal average accuracy and maximum forgetting metrics.
Revised: These ﬁrst k tasks are not considered when computing the ﬁnal average accuracy and maximum forgetting metrics.

Original:  H INSDIGHT A NCHOR L EARNING (HAL)
Revised:  H INSDIGHT A NCHOR L EARNING (HAL)

Original: Our starting point to describe our proposed continual learning algorithm is the current state of the art: methods based on experience replay (Hayes et al., 2018; Riemer et al., 2019; Chaudhry et al., 2019b).
Revised: Our starting point to describe our proposed continual learning algorithm is the current state of the art methods based on experience replay (Hayes et al., 2018; Riemer et al., 2019; Chaudhry et al., 2019b).

Original: These are predictors f θ that, during training, store a small amount of past observed triplets into an episodic memory M = { ( x (cid:48) , y (cid:48) , t (cid:48) ) } .
Revised: These are predictors f θ that, during training, store a small amount of past observed triplets into an episodic memory M = { ( x (cid:48) , y (cid:48) , t (cid:48) ) } .

Original: When facing a new mini-batch of observations B := { ( x, y, t ) } from task t , they employ the rule θ ← θ − α · ∇ 
Revised: When facing a new mini-batch of observations B := { ( x, y, t ) } from task t , they employ the rule θ ← θ − α · ∇ [Equation]

Original: [Equation]
Revised: [Equation]

Original: ( B ∪ B M ) to update their parameters, where
Revised: B M ) to update their parameters, where

Original: [Equation]
Revised: [Equation]

Original: denotes the average loss across a collection of triplets A , and B M is a mini-batch constructed by sampling triplets of previous tasks from M at random.
Revised: denotes the average loss across a collection of triplets A , and B M is a mini-batch constructed by sampling triplets of previous tasks from M at random.

Original: In general, B M is constructed to have the same size as B , but can be smaller if the episodic memory M does not yet contain enough samples.
Revised: In general, B M is constructed to have the same size as B , but can be smaller if the episodic memory M does not yet contain enough samples.

Original: In the previous, the episodic memory M reminds the predictor about how to perform at past tasks using only a very small amount of datum.
Revised: In the previous, the episodic memory M reminds the predictor about how to perform at past tasks using only a very small amount of datum.

Original: As such, the behaviour of the predictor outside the data stored in M is not guaranteed, and subject to worsen.
Revised: As such, the behaviour of the predictor outside the data stored in M is not guaranteed, and subject to worsen.

Original: Because of this reason, we would like to be more conservative, and propose to further ﬁx the behaviour of the predictor at a collection of carefully constructed anchor points e t (cid:48) , one per class per past task t (cid:48) , after a parameter update 1 .
Revised: Because of this reason, we would like to be more conservative, and propose to further ﬁx the behaviour of the predictor at a collection of carefully constructed anchor points e t (cid:48) , one per class per past task t (cid:48) , after a parameter update 1 .

Original: This setup naturally leads to a meta-learning (Finn et al., 2017) formulation.

Original: In particular, we implement  a two-step parameter update rule:
Revised: To implement this, we propose a two-step parameter update rule:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst step computes a temporary parameter vector ˜ θ by minimizing the loss at a minibatch from the current task t , and the episodic memory of past tasks (this is the usual experience replay parameter update).
Revised: The ﬁrst step computes a temporary parameter vector ˜ θ by minimizing the loss at a minibatch from the current task t , and the episodic memory of past tasks (this is the usual experience replay parameter update).

Original: The second step employs meta-learning to perform the ﬁnal update of the parameter θ , which trades-off the minimization of ( a ) the loss value at the current minibatch and the episodic memory, as well as ( b ) the change in predictions at the anchor points for all past tasks.
Revised: The second step employs nested optimization to perform the ﬁnal update of the parameter θ , which trades-off the minimization of ( a ) the loss value at the current minibatch and the episodic memory, as well as ( b ) the change in predictions at the anchor points for all past tasks.

Original: We use L2 over the unnormalized logits as a distance measure in Eq.
Revised: We use L2 over the unnormalized logits as a distance measure in Eq.

Original: 5, as we experimentally ﬁnd it to be superior than cosine similarity and L1 distance.
Revised: 5, as we experimentally ﬁnd it to be superior than cosine similarity or L1 distance.

Revised: This two-step nested optimization bears a similarity to gradient-based meta-learning approach (Finn et al., 2017) where we restrict the inner update to only a single gradient step.

Original: Next, let us discuss how to choose the anchor points.
Revised: Next, let us discuss how to choose the anchor points.

Original: Recall that our objective is to preserve the performance of  current task throughout the  continual learning experience.
Revised: Recall that our objective is to preserve the performance of the current task throughout the entire continual learning experience.

Original: For this,ideally, we would like to choose each e t as a tool to minimize the Forgetting metric (Eq. 3). That is, we are interested in letting e t be the example from task t that would maximize the amount of forgetting about task t throughout the entire learning experience, including past and future tasks.
Revised: For this, we are interested in letting e t be the example from task t on which the loss would increase maximally had training with future tasks ( t (cid:48) > t ) been performed. In other words, such e t would capture instances where the current task is forgotten the most.

Original: Then, we could require predictions to remain invariant at such e t  by using Eq. 
Revised: Then, requiring the predictions to remain invariant at such e t , by using Eq. 5, would minimize the Forgetting metric (Eq. 3) the most.

Original: Mathematically, the desirable e t and its label y t is given by:
Revised: Mathematically, the desirable e t and its label y t is given by:

Original: [Equation]
Revised: [Equation]

Original: 6) where θ t is the parameter vector after training on task t and θ T is the ﬁnal parameter vector after the entire learning experience.
Revised: where θ t is the parameter vector after training on task t and θ T is the ﬁnal parameter vector after the entire learning experience.

Original: Thus, keeping predictions intact on the pair ( e t , y t ) above would maximally preserve the performance of task t .
Revised: Thus, keeping predictions intact on the pair ( e t , y t ) above would maximally preserve the performance of task t .

Original: However, the idealistic Eq. 6 requires access to ( a ) the entire distribution P t as to compute the maximization, and ( b ) access to all future distributions t (cid:48) > t as to compute the ﬁnal parameter vector θ T .
Revised: However, the idealistic Eq. 6 requires access to ( a ) the entire distribution P t as to compute the maximization, and ( b ) access to all future distributions t (cid:48) > t as to compute the ﬁnal parameter vector θ T .

Original: Both are unrealistic assumptions.
Revised: Both are unrealistic assumptions.

Original: To circumvent ( a ), we can recast Eq.
Revised: To circumvent ( a ), we can recast Eq.

Original: 6 as to learn e t by initializing it at random and using k gradient ascent updates for a given label y t :
Revised: 6 as to learn e t by initializing it at random and using k gradient ascent updates for a given label y t :

Original: [Equation]
Revised: [Equation]

Original:  where we recall that φ denotes the feature extractor of the predictor, and φ t is the neural mean embedding (Smola et al., 2007) of all observed examples from task t which regularizes the constructed e t against the outliers.
Revised: [Table] [Equation] where we recall that φ denotes the feature extractor of the predictor, and φ t is the neural mean embedding (Smola et al., 2007) of all observed examples from task t which regularizes the constructed e t against the outliers.

Original: Since the feature extractor is updated after experiencing each data point, the mean embeddings φ t are computed as running averages.
Revised: Since the feature extractor is updated after experiencing each data point, the mean embeddings φ t are computed as running averages.

Original: That is, after observing a minibatch
Revised: That is, after observing a minibatch

Original: B = { ( x, y, t ) }  , we update:
Revised: B = { ( x, y, t ) } of task t , we update:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where φ t is initialized to zero at the beginning of the learning experience.
Revised: where φ t is initialized to zero at the beginning of the learning experience.

Original: In our experiments, we learn one e t per class for each task.
Revised: In our experiments, we learn one e t per class for each task.

Original: We ﬁx the y t to the corresponding class label, and discard φ t after training  task t .
Revised: We ﬁx the y t to the corresponding class label, and discard φ t after training on task t .

Original: Learning e t in this manner circumvents the requirement of storing the entire distribution P t for a task.
Revised: Learning e t in this manner circumvents the requirement of storing the entire distribution P t for a task.

Original: Still, Eq. 7 requires the parameter vector θ T , to be obtained in the distant future after all learning tasks have been experienced.
Revised: Still, Eq. 7 requires the parameter vector θ T , to be obtained in the distant future after all learning tasks have been experienced.

Original: To waive this impossible requirement, we will make an approximation of the future by simulatingit in the past .
Revised: To waive this impossible requirement, we will approximate the future by simulating the past .

Original: That is, instead of measuring the forgetting that would happen after the model is trained at future tasks, we will measure the forgetting that happens when the model is ﬁne-tuned at past tasks.
Revised: That is, instead of measuring the forgetting that would happen after the model is trained at future tasks, we will measure the forgetting that happens when the model is ﬁne-tuned at past tasks.

Original: In this way, we say that forgetting is estimated in hindsight  using past experiences.
Revised: In this way, we say that forgetting is estimated in hindsight , using past experiences.

Original: More concretely, after training on task t and obtaining the parameter vector θ t , we minimize the loss during one epoch on the episodic memory M to obtain the temporary parameter vector θ M , and recast Eq. 7 as:
Revised: More concretely, after training on task t and obtaining the parameter vector θ t , we minimize the loss during one epoch on the episodic memory M to obtain the temporary parameter vector θ M , and recast Eq.as:

Original: [Equation]
Revised: [Equation]

Original: In practice, we ﬁnd the proposed approximation of future by replaying past datato be effective as the optimized e t leads to 20% reduction in  forgetting  compared to a standard experience replay baseline as evidenced in the experiments.
Revised: Our experiments show that the proposed approximation of future by replaying past data in hindsight reduces forgetting by 20% when compared to a standard experience replay baseline.

Original: This completes the description of our proposed algorithm for continual learning, which combines experience replay with anchors learned in hindsight.
Revised: This completes the description of our proposed algorithm for continual learning, which combines experience replay with anchors learned in hindsight.

Original: We call our approach Hindsight Anchor Learning (HAL) and summarize the entire learning process as follows: Hindsight Anchor Learning (HAL)
Revised: We call our approach Hindsight Anchor Learning (HAL) and summarize the entire learning process as follows: [Equation]

Original: • Initialize θ ∼ P ( θ ) and { e t ∼ P ( e ) } Tt =1 from normal distributions P ( θ ) and P ( e ) .
Revised: • Initialize θ ∼ P ( θ ) and { e t ∼ P ( e ) } Tt =1 from normal distributions P ( θ ) and P ( e ) .

Original: • Initialize M = {} • For each task t
Revised: • Initialize M = {} • For each task t

Original: = 1 , . .
Revised: = 1 , . .

Original: , T : – Initialize φ t =–
Revised: , T : – Initialize φ t =–

Original: For each minibatch B from task t : * Sample B M from M * Update θ using Eq.
Revised: For each minibatch B from task t : * Sample B M from M * Update θ using Eq.

Original: * Update φ t using Eq.
Revised: * Update φ t using Eq.

Original: * Update M by adding B in a ﬁrst-in ﬁrst-out (FIFO) ring buffer – Fine-tune on M to obtain θ M – Build e t using Eq.
Revised: * Update M by adding B in a ﬁrst-in ﬁrst-out (FIFO) ring buffer – Fine-tune on M to obtain θ M – Build e t using Eq.

Original: 9 k times – Discard φ t • Return θ .
Revised: 9 k times – Discard φ t • Return θ .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We now evaluate the performance of HAL against a variety of state-of-the-art methods, across four popular continual learning benchmarks that are adapted to continual learning.
Revised: We now evaluate the performance of HAL against a variety of state-of-the-art methods, across four popular continual learning benchmarks.

Original: 4. D ATESETS AND TASKS
Revised: 4. D ATESETS AND TASKS

Original: We perform experiments on four popular benchmarks for continual learning.
Revised: We perform experiments on four popular benchmarks for continual learning.

Original: • Permuted MNIST is a variant of the MNIST dataset of handwritten digits (LeCun, 1998) where each task applies a ﬁxed random pixel permutation to the original dataset.
Revised: • Permuted MNIST is a variant of the MNIST dataset of handwritten digits (LeCun, 1998) where each task applies a ﬁxed random pixel permutation to the original dataset.

Original: This benchmark contains 23 tasks, each with 1000 samples from 10 different classes.
Revised: This benchmark contains 23 tasks, each with 1000 samples from 10 different classes.

Original: • Rotated MNIST is another variant of MNIST, where each task applies a ﬁxed random image rotation (between 0 and 180 degrees) to the original dataset.
Revised: • Rotated MNIST is another variant of MNIST, where each task applies a ﬁxed random image rotation (between 0 and 180 degrees) to the original dataset.

Original: This benchmark containstasks, each with 1000 samples from 10 different classes.
Revised: This benchmark containstasks, each with 1000 samples from 10 different classes.

Original: • Split CIFAR is a variant of the CIFAR-100 dataset (Krizhevsky & Hinton, 2009; Zenke et al., 2017), where each task contains the data pertaining 5 random classes (without replacement) out of the total 100 classes.
Revised: • Split CIFAR is a variant of the CIFAR-100 dataset (Krizhevsky & Hinton, 2009; Zenke et al., 2017), where each task contains the data pertaining 5 random classes (without replacement) out of the total 100 classes.

Original: This benchmark contains 20 tasks, each with 250 samples per each of the 5 classes.
Revised: This benchmark contains 20 tasks, each with 250 samples per each of the 5 classes.

Original: • Split miniImageNet is a variant of the ImageNet dataset (Russakovsky et al., 2015; Vinyals et al., 2016), containing a subset of images and classes from the original dataset.
Revised: • Split miniImageNet is a variant of the ImageNet dataset (Russakovsky et al., 2015; Vinyals et al., 2016), containing a subset of images and classes from the original dataset.

Original: This benchmark contains 20 tasks, each with 250 samples per each of the 5 classes.
Revised: This benchmark contains 20 tasks, each with 250 samples per each of the 5 classes.

Original: For all datasets, the ﬁrst 3 tasks are used for hyper-parameter optimization (grids available in
Revised: For all datasets, the ﬁrst 3 tasks are used for hyper-parameter optimization (grids available in

Original: Appendix B).
Revised: Appendix B) The

Original: The learners can perform multiple epochs on these three initial tasks, that arelater discarded
Revised: learners can perform multiple epochs on these three initial tasks, that are

Original: for
Revised: later

Original: evaluation.
Revised: discarded for evaluation

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: We compare our proposed model HAL to the following baselines.
Revised: We compare our proposed model HAL to the following baselines.

Original: • Finetune is a single model trained continually on the stream of data, without any regularization or episodic memory.
Revised: • Finetune is a single model trained continually on the stream of data, without any regularization or episodic memory.

Original: • EWC (Kirkpatrick et al., 2016) is a continual learning method that limits changes to parameters critical to past tasks, as measured by the Fisher information matrix.
Revised: • EWC (Kirkpatrick et al., 2016) is a continual learning method that limits changes to parameters critical to past tasks, as measured by the Fisher information matrix.

Original: • AGEM (Chaudhry et al., 2019a) is a continual learning method improving on (Lopez-Paz & Ranzato, 2017), which uses an episodic memory of parameter gradients to limit forgetting.
Revised: • AGEM (Chaudhry et al., 2019a) is a continual learning method improving on (Lopez-Paz & Ranzato, 2017), which uses an episodic memory of parameter gradients to limit forgetting.

Original: • MER
Revised: • MER

Original: (Riemer et al., 2019) is a continual learning method that combines episodic memories with meta-learning to limit forgetting.
Revised: (Riemer et al., 2019) is a continual learning method that combines episodic memories with meta-learning to limit forgetting.

Original: • ER-Ring (Chaudhry et al., 2019b) is a continual learning method that uses a ring buffer as episodic memory.
Revised: • ER-Ring (Chaudhry et al., 2019b) is a continual learning method that uses a ring buffer as episodic memory.

Original: • Multitask is an oracle baseline that has access to all data to optimize Eq.
Revised: • Multitask is an oracle baseline that has access to all data to optimize Eq.

Original: 1, useful to estimate an upper bound on the obtainable Accuracy (Eq. 2).
Revised: 1, useful to estimate an upper bound on the obtainable Accuracy (Eq. 2).

Original: • Clone-and-ﬁnetune is an oracle baseline training one independent model per task, where the model for task t (cid:48) is initialized by cloning the parameters of the model for task t (cid:48) − 1 .
Revised: • Clone-and-ﬁnetune is an oracle baseline training one independent model per task, where the model for task t (cid:48) is initialized by cloning the parameters of the model for task t (cid:48) − 1 .

Original: All baselines use the same neural network architectures: a perceptron with two hidden layers of
Revised: All baselines use the same neural network architectures: a perceptron with two hidden layers of

Original: ReLU neurons in the MNIST experiments, and a ResNet18, with three times less feature maps across all layers, similar to Lopez-Paz & Ranzato (2017), in the CIFAR and ImageNet experiments.
Revised: ReLU neurons in the MNIST experiments, and a ResNet18, with three times less feature maps across all layers, similar to Lopez-Paz & Ranzato (2017), their CIFAR and ImageNet experiments.

Original: The task identiﬁers are used to select the output head in the CIFAR and ImageNet experiments, while ignored in the MNIST experiments.
Revised: The task identiﬁers are used to select the output head in the CIFAR and ImageNet experiments, while ignored in the MNIST experiments.

Original: Batch size is set to 10 for both the stream of data and episodic memories, across experiments and models.
Revised: Batch size is set to 10 for both the stream of data and episodic memories, across experiments and models.

Original: The size of episodic memories is set between 1 andexamples per class per task.
Revised: The size of episodic memories is set between 1 andexamples per class per task.

Original: The same uniﬁed code base runs all experiments, and is available at https://bit.ly/2mw8bsE .
Revised: The same uniﬁed code base runs all experiments, and is available at https://bit.ly/2mw8bsE .

Original: [Table]
Revised: [Table]

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: Table 1 summarizes the main results of our experiments.
Revised: Table 1 summarizes the main results of our experiments.

Original: First, our proposed HAL is the method achieving maximum Accuracy (Eq. 2) and minimal Forgetting (Eq. 3) at all benchmarks.
Revised: First, our proposed HAL is the method achieving maximum Accuracy (Eq. 2) and minimal Forgetting (Eq. 3) at all benchmarks.

Original: This does not include oracle baselines Multitask (which has access to all data simultaneously) and Clone-andﬁnetune (which trains a separate model per task).
Revised: This does not include oracle baselines Multitask (which has access to all data simultaneously) and Clone-andﬁnetune (which trains a separate model per task).

Original: Second, the relative gains from the second-best method ER-Ring to HAL are signiﬁcant, conﬁrming that the anchoring objective (Eq. 5) allows experience-replay methods to generalize better from the same amount of episodic memory.
Revised: Second, the relative gains from the second-best method ER-Ring to HAL are signiﬁcant, conﬁrming that the anchoring objective (Eq. 5) allows experience-replay methods to generalize better from the same amount of episodic memory.

Original: Figure 1 shows a more ﬁne grained analysis of average accuracy as new tasks are learned on Permuted
Revised: Figure 1 shows a more ﬁne grained analysis of average accuracy as new tasks are learned on Permuted

Original: MNIST and Split CIFAR.
Revised: MNIST and Split CIFAR.

Original: HAL preserves the performance of a predictor more effectively than other baselines.
Revised: HAL preserves the performance of a predictor more effectively than other baselines.

Original: Table 2 shows the Accuracy of methods that employ an episodic memory, when we allow the size of this memory to increase; 3 or 5 examples per class per task, resulting in a total memory size of 600 or 1000 for MNIST experiments, and 255 or 425 for CIFAR and ImageNet experiments.
Revised: Table 2 shows the Accuracy of methods that employ an episodic memory, when we allow the size of this memory to increase; 3 or 5 examples per class per task, resulting in a total memory size of 600 or 1000 for MNIST experiments, and 255 or 425 for CIFAR and ImageNet experiments.

Original: The corresponding numbers for Forgetting are given in Appendix Table 4.
Revised: The corresponding numbers for Forgetting are given in Appendix Table 4.

Original: HAL outperforms all competitors at all benchmarks.
Revised: HAL outperforms all competitors at all benchmarks.

Original: Figure 2 provides the training time of all the continual learning baselines on MNIST benchmarks.
Revised: Figure 2 provides the training time of all the continual learning baselines on MNIST benchmarks.

Original: Although HAL adds a overhead on top of experience replay baseline, it is signiﬁcantly faster than,
Revised: Although HAL adds a overhead on top of experience replay baseline, it is signiﬁcantly faster than,

Original: MER, another meta-learning approach to reduce forgetting.
Revised: MER, another meta-learning approach to reduce forgetting.

Original: Finally, Table 3 summarizes the results of an ablation study to better understand the impact of anchor selection.
Revised: Finally, Table 3 summarizes the results of an ablation study to better understand the impact of anchor selection.

Original: We compare selecting anchors as random noise (Random-Anchors), as a random example for the associated class and task (Data-Anchors), or our proposed optimization in hindsight procedure (HAL).
Revised: We compare selecting anchors as random noise (Random-Anchors), as a random example for the associated class and task (Data-Anchors), or our proposed optimization in hindsight procedure (HAL).

Original: Our results conﬁrm that optimizing anchors in hindsight is the most effective strategy in terms of both accuracy and forgetting metrics.
Revised: Our results conﬁrm that optimizing anchors in hindsight is the most effective strategy in terms of both accuracy and forgetting metrics.

Original: [Figure]
Revised: [Figure]

Original: Method
Revised: Method

Original: Permuted MNIST
Revised: Permuted MNIST

Original: Rotated MNIST |M| =|M| =|M| =|M| =
Revised: Rotated MNIST |M| =|M| =|M| =|M| =

Original: AGEM 63.2 (±1.47) 64.1 (±0.74) 49.9 (±1.49) 53.0 (±1.52)
Revised: AGEM 63.2 (±1.47) 64.1 (±0.74) 49.9 (±1.49) 53.0 (±1.52)

Original: MER 74.9 (±0.49) 78.3 (±0.19) 76.5 (±0.30) 77.3 (±1.13)
Revised: MER 74.9 (±0.49) 78.3 (±0.19) 76.5 (±0.30) 77.3 (±1.13)

Original: ER-Ring 73.5 (±0.43) 75.8 (±0.24) 74.7 (±0.56) 76.5 (±0.48)
Revised: ER-Ring 73.5 (±0.43) 75.8 (±0.24) 74.7 (±0.56) 76.5 (±0.48)

Original: HAL (ours) 76.2 (±0.52) 78.4 (±0.27) 77.0 (±0.66) 78.7 (±0.97)
Revised: HAL (ours) 76.2 (±0.52) 78.4 (±0.27) 77.0 (±0.66) 78.7 (±0.97)

Original: Method
Revised: Method

Original: Split CIFAR
Revised: Split CIFAR

Original: Split miniImageNet |M| =|M| =|M| =|M| =
Revised: Split miniImageNet |M| =|M| =|M| =|M| =

Original: AGEM 56.9 (±3.45) 59.9 (±2.64) 51.6 (±2.69) 54.3 (±1.56)
Revised: AGEM 56.9 (±3.45) 59.9 (±2.64) 51.6 (±2.69) 54.3 (±1.56)

Original: MER 57.7 (±2.59) 60.6 (±2.09) 49.4 (±3.43) 54.8 (±1.79)
Revised: MER 57.7 (±2.59) 60.6 (±2.09) 49.4 (±3.43) 54.8 (±1.79)

Original: ER-Ring 60.9 (±1.44) 62.6 (±1.77) 53.5 (±1.42) 54.2 (±3.23)
Revised: ER-Ring 60.9 (±1.44) 62.6 (±1.77) 53.5 (±1.42) 54.2 (±3.23)

Original: HAL (ours) 62.9 (±1.49) 64.4 (±2.15) 56.5 (±0.87) 57.2 (±1.54)
Revised: HAL (ours) 62.9 (±1.49) 64.4 (±2.15) 56.5 (±0.87) 57.2 (±1.54)

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: [Figure]
Revised: [Figure]

Original: In continual learning (Ring, 1997), also called lifelong learning (Thrun, 1998), a learner addresses a sequence of changing tasks without storing the complete datasets of these tasks.
Revised: In continual learning (Ring, 1997), also called lifelong learning (Thrun, 1998), a learner addresses a sequence of changing tasks without storing the complete datasets of these tasks.

Original: This is in contrast to multitask learning (Caruana, 1997), where the learner assumes simultaneous access to data from all tasks.
Revised: This is in contrast to multitask learning (Caruana, 1997), where the learner assumes simultaneous access to data from all tasks.

Original: The main challenge in continual learning is to avoid catastrophic interference or forgetting (McCloskey & Cohen, 1989;
Revised: The main challenge in continual learning is to avoid catastrophic interference or forgetting (McCloskey & Cohen, 1989;

Original: McClelland et al., 1995; Goodfellow et al., 2013), that is, the learner forgetting previously acquired knowledge when learning new tasks.
Revised: McClelland et al., 1995; Goodfellow et al., 2013), that is, the learner forgetting previously acquired knowledge when learning new tasks.

Original: The state-of-the art methods in continual learning can be categorized into three classes.
Revised: The state-of-the art methods in continual learning can be categorized into three classes.

Original: First, regularization approaches discourage updating parameters important for past tasks (Kirkpatrick et al., 2016; Aljundi et al., 2018;
Revised: First, regularization approaches discourage updating parameters important for past tasks (Kirkpatrick et al., 2016; Aljundi et al., 2018;

Original: Nguyen et al., 2018; Zenke et al., 2017).
Revised: Nguyen et al., 2018; Zenke et al., 2017).

Original: While efﬁcient in terms of memory and computation, these approaches suffer from brittleness due to feature drift as the number of tasks increases (Titsias et al.,
Revised: While efﬁcient in terms of memory and computation, these approaches suffer from brittleness due to feature drift as the number of tasks increases (Titsias et al.,

Original: [Table]
Revised: [Table]

Original: Additionally, these approaches are only effective when we can perform multiple passes over each dataset (Chaudhry et al., 2019a), a case deemed unrealistic in this work.
Revised: Additionally, these approaches are only effective when we can perform multiple passes over each dataset (Chaudhry et al., 2019a), a case deemed unrealistic in this work.

Original: Second, modular approaches use different parts of the prediction function for each new task (Fernando et al., 2017; Aljundi et al., 2017; Rosenbaum et al., 2018;
Revised: Second, modular approaches use different parts of the prediction function for each new task (Fernando et al., 2017; Aljundi et al., 2017; Rosenbaum et al., 2018;

Original: Chang et al., 2018; Xu & Zhu, 2018;
Revised: Chang et al., 2018; Xu & Zhu, 2018;

Original: Ferran Alet, 2018).
Revised: Ferran Alet, 2018).

Original: Modular approaches do not scale to a large number of tasks, as they require searching over combinatorial space of module architectures.
Revised: Modular approaches do not scale to a large number of tasks, as they require searching over combinatorial space of module architectures.

Original: Another modular approach (Rusu et al., 2016; Lee et al., 2017) adds new parts to the prediction function as new tasks are learned.
Revised: Another modular approach (Rusu et al., 2016; Lee et al., 2017) adds new parts to the prediction function as new tasks are learned.

Original: By construction, modular approaches have zero forgetting, but their memory requirements increase with the number of tasks.
Revised: By construction, modular approaches have zero forgetting, but their memory requirements increase with the number of tasks.

Original: Third, episodic memory approaches maintain and revisit a small episodic memory of datum from past tasks.
Revised: Third, episodic memory approaches maintain and revisit a small episodic memory of datum from past tasks.

Original: In some of these methods (Li & Hoiem, 2016; Rebufﬁ et al., 2017), examples in the episodic memory are replayed and predictions are kept invariant by means of distillation (Hinton et al., 2014).
Revised: In some of these methods (Li & Hoiem, 2016; Rebufﬁ et al., 2017), examples in the episodic memory are replayed and predictions are kept invariant by means of distillation (Hinton et al., 2014).

Original: In other approaches (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019a; Aljundi et al., 2019b)
Revised: In other approaches (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019a; Aljundi et al., 2019b)

Original: the episodic memory is used as an optimization constraint that discourages increases in loss at past tasks.
Revised: the episodic memory is used as an optimization constraint that discourages increases in loss at past tasks.

Original: More recently, several works (Hayes et al., 2018; Riemer et al., 2019; Rolnick et al., 2018;
Revised: More recently, several works (Hayes et al., 2018; Riemer et al., 2019; Rolnick et al., 2018;

Original: Chaudhry et al., 2019b) have shown that directly optimizing the loss on the episodic memory, also known as experience replay, is cheaper than constraint-based approaches and improves prediction performance.
Revised: Chaudhry et al., 2019b) have shown that directly optimizing the loss on the episodic memory, also known as experience replay, is cheaper than constraint-based approaches and improves prediction performance.

Original: Our contribution in this paper has been to improve experience replay methods with task anchors learned in hindsight.
Revised: Our contribution in this paper has been to improve experience replay methods with task anchors learned in hindsight.

Original: There are other deﬁnitions of continual learning, such as the one of task-free continual learning.
Revised: There are other deﬁnitions of continual learning, such as the one of task-free continual learning.

Original: The task-free formulation does not consider the notion of tasks, and instead works on undivided data streams (Aljundi et al., 2019a,b).
Revised: The task-free formulation does not consider the notion of tasks, and instead works on undivided data streams (Aljundi et al., 2019a,b).

Original: We have focused on the task-based deﬁnition of continual learning and, similar to many recent works (Lopez-Paz & Ranzato, 2017; Hayes et al., 2018;
Revised: We have focused on the task-based deﬁnition of continual learning and, similar to many recent works (Lopez-Paz & Ranzato, 2017; Hayes et al., 2018;

Original: Riemer et al., 2019; Chaudhry et al., 2019a), assumed that only a single pass through the data was possible.
Revised: Riemer et al., 2019; Chaudhry et al., 2019a), assumed that only a single pass through the data was possible.

Original: Finally, our gradient-based learning of anchors bears a similarity to (Simonyan et al., 2014) and (Wang et al., 2018).
Revised: Finally, our gradient-based learning of anchors bears a similarity to (Simonyan et al., 2014) and (Wang et al., 2018).

Original: In Simonyan et al.
Revised: In Simonyan et al.

Original: , the authors use gradient ascent on class scores to ﬁnd saliency maps of a classiﬁcation model.
Revised: , the authors use gradient ascent on class scores to ﬁnd saliency maps of a classiﬁcation model.

Original: Contrary to them, our proposed hindsight learning objective optimizes for the forgetting metric, as reducing it is necessary while learning continually.
Revised: Contrary to them, our proposed hindsight learning objective optimizes for the forgetting metric, as reducing it is necessary while learning continually.

Original: Dataset distillation (Wang et al., 2018) proposes to encode the entire dataset in a few synthetic points at a given parameter vector by a gradient-based optimization process.
Revised: Dataset distillation (Wang et al., 2018) proposes to encode the entire dataset in a few synthetic points at a given parameter vector by a gradient-based optimization process.

Original: Their method requires access to the entire dataset of a task for optimization purposes.
Revised: Their method requires access to the entire dataset of a task for optimization purposes.

Original: We, on the other hand, learn anchors in hindsight from the replay buffer of past tasks after training for current task.
Revised: We, instead, learn anchors in hindsight from the replay buffer of past tasks after training for current task.

Original: While Wang et al.
Revised: While Wang et al.

Original: (2018) aim to replicate the performance of the entire dataset from the synthetic points, we focus on reducing forgetting of an already learned task.
Revised: (2018) aim to replicate the performance of the entire dataset from the synthetic points, we focus on reducing forgetting of an already learned task.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduced HAL, a meta-learning objective for continual learning.
Revised: We introduced HAL, a meta-learning objective for continual learning.

Original: In our approach, we learn one “anchor point” per class per task, where predictions are requested to remain invariant.
Revised: In our approach, we learn one “anchor point” per class per task, where predictions are requested to remain invariant.

Original: These anchors are learned using gradient-based optimization, and represent points that would maximize the forgetting of the current task throughout the entire learning experience.
Revised: These anchors are learned using gradient-based optimization, and represent points that would maximize the forgetting of the current task throughout the entire learning experience.

Original: We simulate the forgetting that would happen during the learning of future tasks in hindsight , that is, by taking temporary gradient steps across a small episodic memory of past tasks.
Revised: We simulate the forgetting that would happen during the learning of future tasks in hindsight , that is, by taking temporary gradient steps across a small episodic memory of past tasks.

Original: As shown in our experiments, anchoring in hindsight complements and improves the performance of continual learning methods based on experience replay, achieving a new state of the art on four standard continual learning benchmarks.
Revised: As shown in our experiments, anchoring in hindsight complements and improves the performance of continual learning methods based on experience replay, achieving a new state of the art on four standard continual learning benchmarks.

Original: A PPENDIX
Revised: A PPENDIX

Original: In Section A, we are report the forgetting 3 metric when large episodic memories are used.
Revised: In Section A, we are report the forgetting 3 metric when large episodic memories are used.

Original: Section B provides the grid considered for hyper-parameters.
Revised: Section B provides the grid considered for hyper-parameters.

Original: Section C provides pseudo-code of HAL.
Revised: Section C provides pseudo-code for HAL.

Original: A M ORE R ESULTS
Revised: A M ORE R ESULTS

Original: [Table]
Revised: [Table]

Original: B H YPER - PARAMETER S ELECTION
Revised: B H YPER - PARAMETER S ELECTION

Original: In this section we report the hyper-parameters grid considered for experiments.
Revised: In this section we report the hyper-parameters grid considered for experiments.

Original: The best values for different benchmarks are given in parenthesis.
Revised: The best values for different benchmarks are given in parenthesis.

Original: Multitask – learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0] • Clone-and-ﬁnetune – learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0] • Finetune – learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0] • EWC – learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0] – regularization: [0.1, 1, 10 (MNIST perm, rot, CIFAR, miniImageNet), 100,
Revised: Multitask – learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0] • Clone-and-ﬁnetune – learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0] • Finetune – learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0] • EWC – learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0] – regularization: [0.1, 1, 10 (MNIST perm, rot, CIFAR, miniImageNet), 100,

Original: 1000] • AGEM – learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0] • MER
Revised: 1000] • AGEM – learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0] • MER

Original: – learning rate: [0.003, 0.01, 0.03 (MNIST, CIFAR, miniImageNet), 0.1, 0.3, 1.0] – within batch meta-learning rate: [0.01, 0.03, 0.
Revised: – learning rate: [0.003, 0.01, 0.03 (MNIST, CIFAR, miniImageNet), 0.1, 0.3, 1.0] – within batch meta-learning rate: [0.01, 0.03, 0.

Original: (MNIST, CIFAR, miniImageNet), 0.3, 1.0] – current batch learning rate multiplier: [1, 2, 5 (CIFAR, miniImageNet), 10 (MNIST)]
Revised: (MNIST, CIFAR, miniImageNet), 0.3, 1.0] – current batch learning rate multiplier: [1, 2, 5 (CIFAR, miniImageNet), 10 (MNIST)]

Original: • ER-Ring – learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0] • HAL – learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0] – regularization ( λ ): [0.01, 0.03, 0.1 (MNIST perm, rot), 0.3 (miniImageNet), 1 (CIFAR), 3, 10] – mean embedding strength ( γ ): [0.01, 0.03, 0.1 (MNIST perm, rot, CIFAR, miniImageNet), 0.3, 1, 3, 10]
Revised: • ER-Ring – learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0] • HAL – learning rate: [0.003, 0.01, 0.03 (CIFAR, miniImageNet), 0.1 (MNIST perm, rot), 0.3, 1.0] – regularization ( λ ): [0.01, 0.03, 0.1 (MNIST perm, rot), 0.3 (miniImageNet), 1 (CIFAR), 3, 10] – mean embedding strength ( γ ): [0.01, 0.03, 0.1 (MNIST perm, rot, CIFAR, miniImageNet), 0.3, 1, 3, 10]

Original: C HAL A LGORITHM
Revised: C HAL A LGORITHM

Original: Algorithm 1 provides pseudo-code of HAL.
Revised: Algorithm 1 provides pseudo-code for HAL.

Original: Algorithm1 Training of HAL on sequential data D = {D 1 , · · · , D T } , with total replay buffer size ‘mem sz’, learning rate ‘ α ’, regularization strength ‘ λ ’, mean embedding decay ‘ β ’, mean embedding strength ‘ η ’. [Table] sz
Revised: Algorithm

Original: [Table]
Revised: [Table]

Original: α, λ, β

Original: [Table]

Original: M

Original: [Table]

Original: {} ∗ mem sz

Original: [Table]

Original: [Table]

Original: [Table]

Original: 6: for B ∼ D t do (cid:46) Sampleabatchfromcurrenttask 7:
Revised: 6: for B ∼ D t do (cid:46) Sampleabatchfromcurrenttask 7:

Original: B M ∼ M (cid:46) Sampleabatchfromepisodicmemory 8: ˜ θ ← θ − α · ∇ θ (cid:96) ( B ∪ B M ) (cid:46) Temporaryparameterupdate 9: θ ← θ − α · ∇ θ (cid:16) (cid:96) ( B ∪ B M ) + λ · (cid:80) t (cid:48)
Revised: B M ∼ M (cid:46) Sampleabatchfromepisodicmemory 8: ˜ θ ← θ − α · ∇ θ (cid:96) ( B ∪ B M ) (cid:46) Temporaryparameterupdate 9: θ ← θ − α · ∇ θ (cid:16) (cid:96) ( B ∪ B M ) + λ · (cid:80) t (cid:48)

Original: <t ( f θ ( e t (cid:48) , t (cid:48) )
Revised: <t ( f θ ( e t (cid:48) , t (cid:48) )

Original: − f ˜ θ ( e t (cid:48) , t (cid:48) )) 2 (cid:17) (cid:46) Anchoringobjective(Eq.5) 10: φ t ← β · φ t + (1 − β ) · φ ( B ) (cid:46)
Revised: − f ˜ θ ( e t (cid:48) , t (cid:48) )) 2 (cid:17) (cid:46) Anchoringobjective(Eq.5) 10: φ t ← β · φ t + (1 − β ) · φ ( B ) (cid:46)

Original: Runningaverageofmeanembedding 11:
Revised: Runningaverageofmeanembedding 11:

Original: M ← UpdateMemory ( M , B ) (cid:46) Addsamplestoaringbuffer 12: end for 13: e t , θ ← GetAnchors ( M , θ, φ t , η ) (cid:46) Getanchorsforcurrenttask 14: end for 15: return θ, M 16: end procedure 1: procedure G ET A NCHORS ( M , θ t , φ t , γ ) 2: θ ← θ t 3: for B M ∼ M do 4: θ ← θ − α · ∇ θ (cid:96) ( B M ) (cid:46) Finetune θ t bytakingSGDstepsontheepisodicmemory 5: end for 6: θ M ← θ (cid:46) Storetheupdatedparameter 7: e t ← rand() (cid:46) Initializethetaskanchors 8: for 1 , · · · , k do 9: e t ← e t + α · ∇ e t (cid:0) (cid:96) ( f θ M ( e t , t ) , y t ) − (cid:96) ( f θ t ( e t , 
Revised: M ← UpdateMemory ( M , B ) (cid:46) Addsamplestoaringbuffer 12: end for 13: e t , θ ← GetAnchors ( M , θ, φ t , η ) (cid:46) Getanchorsforcurrenttask 14: end for 15: return θ, M 16: end procedure 1: procedure G ET A NCHORS ( M , θ t , φ t , γ ) 2: θ ← θ t 3: for B M ∼ M do 4: θ ← θ − α · ∇ θ (cid:96) ( B M ) (cid:46) Finetune θ t bytakingSGDstepsontheepisodicmemory 5: end for 6: θ M ← θ (cid:46) Storetheupdatedparameter 7: e t ← rand() (cid:46) Initializethetaskanchors 8: for 1 , · · · , k do 9: e t ← e t + α · ∇ e t (cid:0) (cid:96) ( f θ M ( e t , t ) , y t ) − (cid:96) ( f θ t ( e t , t ) , y t ) − γ ( φ ( e t ) − φ t ) 2 (cid:1) (cid:46) Maximizeforgetting(Eq.9) D A PPROXIMATION OF A NCHORING G RADIENT Here we derive the approximation of anchoring objective (Eq. 5) gradient. In particular, we are more interested in the regularization part of the anchoring objective that involves second-order terms.

Revised: We refer to this gradient as g anch .

Original: [Table]
Revised: We follow similar arguments as (Nichol & Schulman, 2018). Let θ 0 be the parameter vector before the temporary update. We will use the following deﬁnitions: [Equation]

Revised: gradient of

Revised: [Equation]

Revised: [Equation]

Revised: ( Hessian of cross-entropy loss at

Revised: [Equation]

Revised: M

Revised: [Equation]

Revised: ( gradient of L2 loss at initial point on anchors )

Revised: [Equation]

Revised: ( gradient of L2 loss at initial point on anchors )

Original: {+Let U 
+}
Revised: {+Let U 
+}

Revised: = θ 0 − αg 0 be the operator giving temporary update in the two-step process of (Eq. 5), and let θ 1 be the temporary update (note that ˜ θ is used in the main paper).

Revised: [Equation]

Revised: Let’s calculate the ﬁrst order Taylor’s approximation of (cid:96) (cid:48)

Original: [Table]
Revised: L 2 ( θ 1 ) [Equation] Putting Eq. 11 in Eq.

Revised: 10 and simplifying yields:

Revised: [Equation]

Revised: This form is very similar to second-order MAML gradient formulation, Eq.

Revised: in (Nichol & Schulman, 2018).

Revised: Further simpliﬁcation of terms like ( H 1 · g 0 ) would yield inner product between the gradients g 0 and g 1 .

Revised: Unlike MAML (Finn et al., 2017), Reptile (Nichol & Schulman, 2018) or MER (Riemer et al., 2019), in anchoring objective these gradients correspond to different losses.