Original: F ULLY
Revised: F ULLY

Original: O NLINE
Revised: O NLINE

Original: M ETA -L EARNING
Revised: M ETA -L EARNING

Original: W ITHOUT
Revised: W ITHOUT

Original: T ASK
Revised: T ASK

Original: B OUNDARIES
Revised: B OUNDARIES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While deep networks can learn complex functions such as classiﬁers, detectors, and trackers, many applications require models that continually adapt to changing input distributions, changing tasks, and changing environmental conditions.
Revised: While deep networks can learn complex functions such as classifiers, detectors, and trackers, many applications require models that continually adapt to changing input distributions, changing tasks, and changing environmental conditions.

Original: Indeed, this ability to continuously accrue knowledge and use past experience to learn new tasks quickly in continual settings is one of the key properties of an intelligent system.
Revised: Indeed, this ability to continuously accrue knowledge and use past experience to learn new tasks quickly in continual settings is one of the key properties of an intelligent system.

Original: For complex and high-dimensional problems, simply updating the model continually with standard learning algorithms such as gradient descent may result in slow adaptation.
Revised: For complex and high-dimensional problems, simply updating the model continually with standard learning algorithms such as gradient descent may result in slow adaptation.

Original: Meta-learning can provide a powerful tool to accelerate adaptation yet is conventionally studied in batch settings.
Revised: Meta-learning can provide a powerful tool to accelerate adaptation yet is conventionally studied in batch settings.

Original: In this paper, we study how metalearning can be applied to tackle online problems of this nature, simultaneously adapting to changing tasks and input distributions and meta-training the model in order to adapt more quickly in the future.
Revised: In this paper, we study how metalearning can be applied to tackle online problems of this nature, simultaneously adapting to changing tasks and input distributions and meta-training the model in order to adapt more quickly in the future.

Original: Extending meta-learning into the online setting presents its own challenges, and although several prior methods have studied related problems, they generally require a discrete notion of tasks, with known ground-truth task boundaries.
Revised: Extending meta-learning into the online setting presents its own challenges, and although several prior methods have studied related problems, they generally require a discrete notion of tasks, with known ground-truth task boundaries.

Original: Such methods typically adapt to each task in sequence, resetting the model between tasks, rather than adapting continuously across tasks.
Revised: Such methods typically adapt to each task in sequence, resetting the model between tasks, rather than adapting continuously across tasks.

Original: In many real-world settings, such discrete boundaries are unavailable, and may not even exist.
Revised: In many real-world settings, such discrete boundaries are unavailable, and may not even exist.

Original: To address these settings, we propose a Fully Online MetaLearning (FOML) algorithm, which does not require any ground truth knowledge about the task boundaries and stays fully online without resetting back to pretrained weights.
Revised: To address these settings, we propose a Fully Online MetaLearning (FOML) algorithm, which does not require any ground truth knowledge about the task boundaries and stays fully online without resetting back to pretrained weights.

Original: Our experiments show that FOML was able to learn new tasks faster than the state-of-the-art online learning methods on Rainbow-MNIST, and
Revised: Our experiments show that FOML was able to learn new tasks faster than the state-of-the-art online learning methods on Rainbow-MNIST, and

Original: CIFAR100 datasets.
Revised: CIFAR100 datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Flexibility and rapid adaptation are a hallmark of intelligence: humans can not only solve complex problems, but they can also ﬁgure out how to solve them very rapidly, as compared to our current machine learning algorithms.
Revised: Flexibility and rapid adaptation are a hallmark of intelligence: humans can not only solve complex problems, but they can also figure out how to solve them very rapidly, as compared to our current machine learning algorithms.

Original: Such rapid adaptation is crucial for both humans and computers: for humans, it is crucial for survival in changing natural environments, and it is also crucial for agents that classify photographs on the Internet, interpret text, control autonomous vehicles, and generally make accurate predictions with rapidly changing real-world data.
Revised: Such rapid adaptation is crucial for both humans and computers: for humans, it is crucial for survival in changing natural environments, and it is also crucial for agents that classify photographs on the Internet, interpret text, control autonomous vehicles, and generally make accurate predictions with rapidly changing real-world data.

Original: While deep neural networks are remarkably effective for learning and representing accurate models (He et al., 2015; Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Szegedy et al., 2015), they are comparatively unimpressive when it comes to adaptability, due to their computational and data requirements.
Revised: While deep neural networks are remarkably effective for learning and representing accurate models (He et al., 2015; Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; Szegedy et al., 2015), they are comparatively unimpressive when it comes to adaptability, due to their computational and data requirements.

Original: Meta-learning in principle mitigates this problem, by leveraging the generalization power of neural networks to accelerate adaptation to new tasks (Finn et al., 2019; Li et al., 2017; Nichol et al., 2018; Nichol & Schulman, 2018; Park & Oliva, 2019; Antoniou et al., 2018).
Revised: Meta-learning in principle mitigates this problem, by leveraging the generalization power of neural networks to accelerate adaptation to new tasks (Finn et al., 2019; Li et al., 2017; Nichol et al., 2018; Nichol & Schulman, 2018; Park & Oliva, 2019; Antoniou et al., 2018).

Original: However, standard meta-learning algorithms operate in batch mode, making them poorly suited for continuously evolving environments.
Revised: However, standard meta-learning algorithms operate in batch mode, making them poorly suited for continuously evolving environments.

Original: More recently, online meta-learning methods have been proposed with the goal of enabling continual adaptation (Finn et al., 2019;
Revised: More recently, online meta-learning methods have been proposed with the goal of enabling continual adaptation (Finn et al., 2019;

Original: Jerfel et al., 2018; Yao et al., 2020; Nagabandi et al., 2018; Li &
Revised: Jerfel et al., 2018; Yao et al., 2020; Nagabandi et al., 2018; Li &

Original: Hospedales, 2020), where a constant stream of data from distinct tasks is used for both adaptation and meta-training.
Revised: Hospedales, 2020), where a constant stream of data from distinct tasks is used for both adaptation and meta-training.

Original: In this scheme, meta-training is used to accelerate how quickly the network can adapt to each new task it sees, and simultaneously  that data for each new taskis also used for meta-training.
Revised: In this scheme, meta-training is used to accelerate how quickly the network can adapt to each new task it sees, and simultaneously use that data from each new task for meta-training.

Original: This further accelerates how quickly each subsequent task can be acquired.
Revised: This further accelerates how quickly each subsequent task can be acquired.

Original: However, current online meta-learning methods fall short of the goal of creating an effectiveand adaptation system for online data in several ways: (1) they typically require task boundaries in the data stream to be known, making them ill-suited to settings where task boundaries are ill-deﬁned and tasks change or evolve gradually, a common tread in real-world; (2) as a result, they typically re-adapt from the meta-trained model on each task, resulting in a very “discrete” mode of operation, where the model adapts to a task, then resets, then adapts to a new one.
Revised: However, current online meta-learning methods fall short of the goal of creating an effective adaptation system for online data in several ways: (1) they typically require task boundaries in the data stream to be known, making them ill-suited to settings where task boundaries are ill-defined and tasks change or evolve gradually, a common tread in real-world; (2) as a result, they typically re-adapt from the meta-trained model on each task, resulting in a very “discrete” mode of operation, where the model adapts to a task, then resets, then adapts to a new one.

Original: These limitations restrict the applicability of current online meta-learning methods to real-world settings.
Revised: These limitations restrict the applicability of current online meta-learning methods to real-world settings.

Original: We argue that task boundary assumption is somewhat artiﬁcial in online settings, where the stream of incoming data is cleanly partitioned into discrete and well-separated tasks presented in sequence.
Revised: We argue that task boundary assumption is somewhat artificial in online settings, where the stream of incoming data is cleanly partitioned into discrete and well-separated tasks presented in sequence.

Original: In this paper, we instead develop a fully online meta-learning approach, which does not assume knowledge of task boundaries and does not re-adapt for every new task from the meta parameters.
Revised: In this paper, we instead develop a fully online meta-learning approach, which does not assume knowledge of task boundaries and does not re-adapt for every new task from the meta parameters.

Original: Standard meta-learning methods consist of a meta-training phase, typically done with standard SGD, and an “inner loop” adaptation phase, which computes task speciﬁc parameter φ i for the task T i from a support set to make accurate predictions on a query set.
Revised: Standard meta-learning methods consist of a meta-training phase, typically done with standard SGD, and an “inner loop” adaptation phase, which computes task specific parameter ϕ i for the task T i from a support set to make accurate predictions on a query set.

Original: For example, in model-agnostic metalearning (MAML), adaptation consists of taking a few gradient steps on the support set, starting from the meta-trained parameter vector θ , leading to a set of post-adaptation parameters, and meta-training optimizes the meta-trained parameters θ so that these gradient steps lead to good results.
Revised: For example, in model-agnostic metalearning (MAML), adaptation consists of taking a few gradient steps on the support set, starting from the meta-trained parameter vector θ , leading to a set of post-adaptation parameters, and meta-training optimizes the meta-trained parameters θ so that these gradient steps lead to good results.

Original: Previous extensions of such approaches into the online setting typically observe one task at a time, adapt to that task (i.e., compute post-adaptation parameters on it), and then reset φ i back to the meta-trained parameters θ at the beginning of the next task.
Revised: Previous extensions of such approaches into the online setting typically observe one task at a time, adapt to that task (i.e., compute post-adaptation parameters on it), and then reset ϕ i back to the meta-trained parameters θ at the beginning of the next task.

Original: Thus, the algorithm repeatedly adapts, resets at the task boundary, adapts again, and repeats.
Revised: Thus, the algorithm repeatedly adapts, resets at the task boundary, adapts again, and repeats.

Original: This is illustrated in Figure 1 (left).
Revised: This is illustrated in Figure 1 (left).

Original: However, in many realistic settings, the task boundaries are not known, and instead the tasks shift gradually over time.
Revised: However, in many realistic settings, the task boundaries are not known, and instead the tasks shift gradually over time.

Original: The discrete “resetting” procedure is a poor ﬁt in such cases, and we would like to simply continue adapting the weights over time without ever resetting back to the meta-trained parameters, but still beneﬁt from a concurrent meta-training process.
Revised: The discrete “resetting” procedure is a poor fit in such cases, and we would like to simply continue adapting the weights over time without ever resetting back to the meta-trained parameters, but still benefit from a concurrent meta-training process.

Original: For example, a meta-trained image-tagging model on the Internet (e.g., tagging friends in photographs) might gradually adapt to changing patterns and preferences of its users over time, where it would be unnatural to assume discrete shifts in what users want to tag.
Revised: For example, a meta-trained image-tagging model on the Internet (e.g., tagging friends in photographs) might gradually adapt to changing patterns and preferences of its users over time, where it would be unnatural to assume discrete shifts in what users want to tag.

Original: Similarly, a trafﬁc prediction system might adapt to changing trafﬁc patterns, including periodic changes due to seasons, and unexpected changes due to shifting economic conditions, weather, and trafﬁc accidents.
Revised: Similarly, a traffic prediction system might adapt to changing traffic patterns, including periodic changes due to seasons, and unexpected changes due to shifting economic conditions, weather, and traffic accidents.

Original: In this spirit, our method does not require any knowledge on the task boundaries as well as stays fully-online through out the learning.
Revised: In this spirit, our method does not require any knowledge on the task boundaries as well as stays fully-online through out the learning.

Original: The main contribution of our paper is FOML (fully online meta-learning), an online meta-learning algorithm that continually updates its online parameters with each new datapoint or batch of datapoints, while simultaneously performing meta-gradient updates on a separate set of meta-parameters using a buffer of previously seen data.
Revised: The main contribution of our paper is FOML (fully online meta-learning), an online meta-learning algorithm that continually updates its online parameters with each new datapoint or batch of datapoints, while simultaneously performing meta-gradient updates on a separate set of meta-parameters using a buffer of previously seen data.

Original: FOML does not require ground truth knowledge of task boundaries, and does not reset the online parameters back to the meta-parameters between tasks, instead updating the online parameters continually in a fully online fashion.
Revised: FOML does not require ground truth knowledge of task boundaries, and does not reset the online parameters back to the meta-parameters between tasks, instead updating the online parameters continually in a fully online fashion.

Original: We compare FOML empirically to strong baselines and a state-of-the-art prior online meta-learning method, showing that FOML learns to adapt more quickly, and achieves lower error rates, both on a simple sequential image classiﬁcation task from prior work and a more complex benchmark that we propose based on the CIFAR100 dataset, 
Revised: We compare FOML empirically to strong baselines and a state-of-the-art prior online meta-learning method, showing that FOML learns to adapt more quickly, and achieves lower error rates, both on a simple sequential image classification task from prior work and a more complex benchmark that we propose based on the CIFAR100 dataset, with a sequence of 1200 tasks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Online meta-learning brings together ideas from online learning, meta learning, and continual learning, with the aim of adapting quickly to each new task while simultaneously learning how to adapt even more quickly in the future.
Revised: Online meta-learning brings together ideas from online learning, meta learning, and continual learning, with the aim of adapting quickly to each new task while simultaneously learning how to adapt even more quickly in the future.

Original: We discuss these three sets of approaches next.
Revised: We discuss these three sets of approaches next.

Original: Meta Learning:
Revised: Meta Learning:

Original: Meta learning methods try to learn the high-level context of the data, to behave well on new tasks ( Learning to learn ).
Revised: Meta learning methods try to learn the high-level context of the data, to behave well on new tasks ( Learning to learn ).

Original: These methods involve learning a metric space (Koch et al., 2015; Vinyals et al., 2016;
Revised: These methods involve learning a metric space (Koch et al., 2015; Vinyals et al., 2016;

Original: Snell et al., 2017; Yang et al., 2017), gradient based updates (Finn et al., 2017;
Revised: Snell et al., 2017; Yang et al., 2017), gradient based updates (Finn et al., 2017;

Original: Li et al., 2017; Park & Oliva, 2019; Nichol et al., 2018; Nichol & Schulman, 2018), or some speciﬁc architecture designs (Santoro et al., 2016; Munkhdalai & Yu, 2017; Ravi & Larochelle, 2016).
Revised: Li et al., 2017; Park & Oliva, 2019; Nichol et al., 2018; Nichol & Schulman, 2018), or some specific architecture designs (Santoro et al., 2016; Munkhdalai & Yu, 2017; Ravi & Larochelle, 2016).

Original: In this work, we are mainly interested in gradient based meta learning methods for online learning.
Revised: In this work, we are mainly interested in gradient based meta learning methods for online learning.

Original: MAML (Finn et al., 2017) and its variants (Nichol et al., 2018; Nichol & Schulman, 2018; Li et al., 2017; Park & Oliva, 2019; Antoniou et al., 2018) ﬁrst meta train the models in such a way that the meta parameters are close to the optimal task speciﬁc parameters (good initialization).
Revised: MAML (Finn et al., 2017) and its variants (Nichol et al., 2018; Nichol & Schulman, 2018; Li et al., 2017; Park & Oliva, 2019; Antoniou et al., 2018) first meta train the models in such a way that the meta parameters are close to the optimal task specific parameters (good initialization).

Original: This way, adaptation becomes faster when ﬁne tuning from the meta parameters.
Revised: This way, adaptation becomes faster when fine tuning from the meta parameters.

Original: However, directly adapting this approach into an online setting will require more relaxation on online learning assumptions, such as access to task boundaries and resetting back and froth from meta parameters.
Revised: However, directly adapting this approach into an online setting will require more relaxation on online learning assumptions, such as access to task boundaries and resetting back and froth from meta parameters.

Original: Our method does not require knowledge of task boundaries.
Revised: Our method does not require knowledge of task boundaries.

Original: Online Learning: Online learning methods update their models based on the stream of data sequentially.
Revised: Online Learning: Online learning methods update their models based on the stream of data sequentially.

Original: There are various works on online learning using linear models (Cesa-Bianchi & Lugosi, 2006), non-linear models with kernels (Kivinen et al., 2004; Jin et al., 2010), and deep neural networks (Zhou et al., 2012).
Revised: There are various works on online learning using linear models (Cesa-Bianchi & Lugosi, 2006), non-linear models with kernels (Kivinen et al., 2004; Jin et al., 2010), and deep neural networks (Zhou et al., 2012).

Original: Online learning algorithms often simply update the model on the new data, and do not consider the past knowledge of the previously seen data to do this online update more efﬁciently.
Revised: Online learning algorithms often simply update the model on the new data, and do not consider the past knowledge of the previously seen data to do this online update more efficiently.

Original: However, the online meta learning framework, allow us to keep track of previously seen data and with the “meta” knowledge we can update the online weights to the new data more faster and efﬁciently.
Revised: However, the online meta learning framework, allow us to keep track of previously seen data and with the “meta” knowledge we can update the online weights to the new data more faster and efficiently.

Original: Continual Learning:
Revised: Continual Learning:

Original: A number of prior works on continual learning have addressed catastrophic forgetting (McCloskey & Cohen, 1989; Li & Hoiem, 2017; Ratcliff, 1990; Rajasegaran et al., 2019; 2020), removing the need to store all prior data during training.
Revised: A number of prior works on continual learning have addressed catastrophic forgetting (McCloskey & Cohen, 1989; Li & Hoiem, 2017; Ratcliff, 1990; Rajasegaran et al., 2019; 2020), removing the need to store all prior data during training.

Original: Our method does not address catastrophic forgetting for the meta-training phase, because we must still store all data so as to “replay” it for meta-training, though it may be possible to discard or sub-sample old data (which we leave to future work).
Revised: Our method does not address catastrophic forgetting for the meta-training phase, because we must still store all data so as to “replay” it for meta-training, though it may be possible to discard or sub-sample old data (which we leave to future work).

Original: However, our adaptation process is fully online.
Revised: However, our adaptation process is fully online.

Original: A number of works perform meta-learning for better continual learning, i.e. learning good continual learning strategies (AlShedivat et al., 2017;
Revised: A number of works perform meta-learning for better continual learning, i.e. learning good continual learning strategies (AlShedivat et al., 2017;

Original: Nagabandi et al., 2018; Javed & White, 2019; Harrison et al., 2019;
Revised: Nagabandi et al., 2018; Javed & White, 2019; Harrison et al., 2019;

Original: He et al., 2019; Beaulieu et al., 2020).
Revised: He et al., 2019; Beaulieu et al., 2020).

Original: However, these prior methods still perform batch-mode meta-training, while our method also performs the meta-training itself incrementally online, without task boundaries.
Revised: However, these prior methods still perform batch-mode meta-training, while our method also performs the meta-training itself incrementally online, without task boundaries.

Original: The closest work to ours is the follow the meta-leader (FTML) method (Finn et al., 2019) and its variants (Yao et al., 2020).
Revised: The closest work to ours is the follow the meta-leader (FTML) method (Finn et al., 2019) and its variants (Yao et al., 2020).

Original: FTML is a varaint of MAML that ﬁnetunes to each new task in turn, resetting to the meta-trained parameters between every task.
Revised: FTML is a varaint of MAML that finetunes to each new task in turn, resetting to the meta-trained parameters between every task.

Original: While this method effectively accelerates acquisition of new tasks, it requires ground truth knowledge of task boundaries and, as we show in our experiments, our approach outperforms FTML even when FTML has access to task boundaries and our method does not . 
Revised: While this method effectively accelerates acquisition of new tasks, it requires ground truth knowledge of task boundaries and, as we show in our experiments, our approach outperforms FTML even when FTML has access to task boundaries and our method does not . Further, there are some recent works which try to address Online meta learning Harrison et al. Denevi et al. (2019) in a similar setting to ours.

Revised: Oonline-within-Online

Revised: Denevi et al.

Original: F OUNDATIONS
Revised: (2019) tries to learn fast adaptation by learning from old tasks. However, OWO Denevi et al. (2019) requires the knowledge about the task boundaries, while our method does not need any extra information about the task boundaries. MOCA Harrison et al.

Revised: (2020) address a very similar problem as ours, yet it tries to find the task boundaries or change points from past data.

Revised: On the other hand our method can work even without estimating the change points.

Revised:  F OUNDATIONS

Original: Prior to diving into online meta learning, we ﬁrst brieﬂy summarize meta learning, model agnostic meta-learning, and online learning in this section.
Revised: Prior to diving into online meta learning, we first briefly summarize meta learning, model agnostic meta-learning, and online learning in this section.

Original: Meta-learning: Meta-learning address the problem of learning to learn.
Revised: Meta-learning: Meta-learning address the problem of learning to learn.

Original: It uses the knowledge learned from previous tasks to quickly learn new tasks.
Revised: It uses the knowledge learned from previous tasks to quickly learn new tasks.

Original: Meta-learning assumes that the tasks are drawn from a stationary distribution T ∼ P ( T ) .
Revised: Meta-learning assumes that the tasks are drawn from a stationary distribution T ∼ P ( T ) .

Original: During the meta-training phase (outer-loop), N tasks are assumed to be drawn from this distribution to produce the meta-training set, and the model is trained in such a way that, when a new task with its own training and test data T =
Revised: During the meta-training phase (outer-loop), N tasks are assumed to be drawn from this distribution to produce the meta-training set, and the model is trained in such a way that, when a new task with its own training and test data T =

Original: {D tr T , D te T } is presented to it at meta-test time, the model should be able to adapt to this task quickly (inner-loop).
Revised: {D tr T , D te T } is presented to it at meta-test time, the model should be able to adapt to this task quickly (inner-loop).

Original: Using θ to denote the meta-trained parameters, the meta-learning objective is:
Revised: Using θ to denote the meta-trained parameters, the meta-learning objective is:

Original: [Equation]
Revised: [Equation]

Original: where F θ is the meta-learned adaptation process that reads in the training set D trt and outputs task-speciﬁc parameters, prototypes, or features (depending on the method) for the new task T i .
Revised: where F θ is the meta-learned adaptation process that reads in the training set D trt and outputs task-specific parameters, prototypes, or features (depending on the method) for the new task T i .

Original: Model-agnostic meta-learning:
Revised: Model-agnostic meta-learning:

Original: In MAML (Finn et al., 2017), the inner-loop function is (stochastic) gradient decent.
Revised: In MAML (Finn et al., 2017), the inner-loop function is (stochastic) gradient decent.

Original: Hence, during the MAML inner-loop adaptation, F θ ( D tri ) becomes θ − α ∇L θ
Revised: Hence, during the MAML inner-loop adaptation, F θ ( D tri ) becomes θ − α ∇L θ

Original: ( θ, D tri ) (or, more generally, multiple gradient steps).
Revised: ( θ, D tri ) (or, more generally, multiple gradient steps).

Original: Intuitively, what this means is that meta-training with MAML produces a parameter vector θ that can quickly adapt to any task from the meta-training distribution via gradient descent on the task loss.
Revised: Intuitively, what this means is that meta-training with MAML produces a parameter vector θ that can quickly adapt to any task from the meta-training distribution via gradient descent on the task loss.

Original: One of the principle beneﬁts of this is that, when faced with a new task that differs from those seen during meta-training, the algorithm “at worst” adapts with regular gradient descent, and at best is massively accelerated by the meta-training.
Revised: One of the principle benefits of this is that, when faced with a new task that differs from those seen during meta-training, the algorithm “at worst” adapts with regular gradient descent, and at best is massively accelerated by the meta-training.

Original: Online learning: In online learning, the model faces a sequence of loss functions {L t } ∞ t =1 and a sequence of data {D t = { ( x, y ) }} ∞ t =1 for every time step t .
Revised: Online learning: In online learning, the model faces a sequence of loss functions {L t } ∞ t =1 and a sequence of data {D t = { ( x, y ) }} ∞ t =1 for every time step t .

Original: The function f : x → ˆ y maps inputs x to predictions ˆ y .
Revised: The function f : x → ˆ y maps inputs x to predictions ˆ y .

Original: The goal of an online learning algorithm is to ﬁnd a set of parameters for each time step { φ } ∞ t =1 , such that the overall loss between the predictions ˆ y and the ground truth labels y is minimized over the sequence.
Revised: The goal of an online learning algorithm is to find a set of parameters for each time step { ϕ } ∞ t =1 , such that the overall loss between the predictions ˆ y and the ground truth labels y is minimized over the sequence.

Original: This is typically quantiﬁed in terms of regret:
Revised: This is typically quantified in terms of regret:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: ) where, φ t = argmin φ L t ( φ, D t ) .
Revised: where, ϕ t = argmin ϕ L t ( ϕ, D t ) .

Original: The ﬁrst term measures the loss from the online model, and the second term measures the loss of the best possible model on that task.
Revised: The first term measures the loss from the online model, and the second term measures the loss of the best possible model on that task.

Original: Various online algorithms tries to minimize the regret as much as possible when introducing new tasks.
Revised: Various online algorithms try to minimize the regret as much as possible when introducing new tasks.

Original:  O NLINE M ETA -L EARNING : P ROBLEM S TATEMENT AND M ETHODS
Revised:  O NLINE M ETA -L EARNING : P ROBLEM S TATEMENT AND M ETHODS

Original: In an online meta-learning setting (Finn et al., 2019), the model f φ observes datapoints one at a time from a online data stream S .
Revised: In an online meta-learning setting (Finn et al., 2019), the model f ϕ observes datapoints one at a time from a online data stream S .

Original: Each datapoint consists of an input x tm , where t is the task index and m is the index of the datapoint within that task, and a label y tm .
Revised: Each datapoint consists of an input x tm , where t is the task index and m is the index of the datapoint within that task, and a label y tm .

Original: The task changes over time and the model should be able to update the parameters φ to minimize the loss at each time step.
Revised: The task changes over time and the model should be able to update the parameters ϕ to minimize the loss at each time step.

Original: The goal of online meta-learning is to quickly learn each new task T t and perform well as soon as possible according to the speciﬁed loss function .
Revised: The goal of online meta-learning is to quickly learn each new task T t and perform well as soon as possible according to the specified loss function .

Original: A simple baseline solution would be to just train the model on the current task T t .
Revised: A simple baseline solution would be to just train the model on the current task T t .

Original: We denote this baseline as TFS (Train from Scratch).
Revised: We denote this baseline as TFS (Train from Scratch).

Original: For every new task, the model simply trains a new set of parameters using all of the data from the current task T t that has been seen so far:
Revised: For every new task, the model simply trains a new set of parameters using all of the data from the current task T t that has been seen so far:

Original: [Equation]
Revised: [Equation]

Original: TFS has two issues.
Revised: TFS has two issues.

Original: First, it requires the task boundaries to be known, which can make it difﬁcult to apply to settings where this information is not available.
Revised: First, it requires the task boundaries to be known, which can make it difficult to apply to settings where this information is not available.

Original: Second, it does not utilize knowledge from other tasks, which greatly limits its performance even when task boundaries are available.
Revised: Second, it does not utilize knowledge from other tasks, which greatly limits its performance even when task boundaries are available.

Original: A straightforward way to utilize knowledge from other tasks of the online data stream is to store all the seen tasks in a large buffer B , and simply keep training the model on all of the seen tasks.
Revised: A straightforward way to utilize knowledge from other tasks of the online data stream is to store all the seen tasks in a large buffer B , and simply keep training the model on all of the seen tasks.

Original: We will refer to this baseline method as TOE (Train on Everything):
Revised: We will refer to this baseline method as TOE (Train on Everything):

Original: [Equation]
Revised: [Equation]

Original: TOE learns a function that ﬁts all of the previously seen samples.
Revised: TOE learns a function that fits all of the previously seen samples.

Original: However, this function may be far from optimal for the task at hand, because the different tasks may be mutually exclusive.
Revised: However, this function may be far from optimal for the task at hand, because the different tasks may be mutually exclusive.

Original: Therefore, ﬁtting a single model on all of the previously seen tasks might not provide a good task-speciﬁc model for the current task.
Revised: Therefore, fitting a single model on all of the previously seen tasks might not provide a good task-specific model for the current task.

Original: A more sophisticated baseline, which we refer to as FTL (Follow the Leader), pre-trains a model on all of the previous tasks, and then ﬁne-tunes it only on the data from the current task.
Revised: A more sophisticated baseline, which we refer to as FTL (Follow the Leader), pre-trains a model on all of the previous tasks, and then fine-tunes it only on the data from the current task.

Original: Note that this is subtly different from FTL in the classic online learning setting, due to the difference in problem formulation.
Revised: Note that this is subtly different from FTL in the classic online learning setting, due to the difference in problem formulation.

Original: This can be achieved by initializing φ witha pretrained weights up to the previous task φ t − 1 TOE .
Revised: This can be achieved by initializing ϕ with pretrained weights up to the previous task ϕ t − 1 TOE .

Original: [Equation]
Revised: [Equation]

Original: Here, for the task t , we take a model that is pre-trained on all previously seen tasks ( f φ t − 1 TOE ) and ﬁne-tune on the current task data.
Revised: Here, for the task t , we take a model that is pre-trained on all previously seen tasks ( f ϕ t − 1 TOE ) and fine-tune on the current task data.

Original: In this way, FTL can use the past knowledge to more quickly adapt to the new task.
Revised: In this way, FTL can use the past knowledge to more quickly adapt to the new task.

Original: However, pre-training on past tasks may not necessarily results in an initialization that is conducive to fast adaptation (Finn et al., 2017; Nichol et al., 2018; Nichol & Schulman, 2018;
Revised: However, pre-training on past tasks may not necessarily result in an initialization that is conducive to fast adaptation (Finn et al., 2017; Nichol et al., 2018; Nichol & Schulman, 2018;

Original: Li et al., 2017).
Revised: Li et al., 2017).

Original: Finn et al.
Revised: Finn et al.

Original: (Finn et al., 2019) proposed a MAML-based online meta-learning approach, where MAML is used to meta-train a “meta-leader” model on all previously seen tasks, which is then adapted on all data from the current task.
Revised: (Finn et al., 2019) proposed a MAML-based online meta-learning approach, where MAML is used to meta-train a “meta-leader” model on all previously seen tasks, which is then adapted on all data from the current task.

Original: This way, the meta-leader parameters will be much closer to new task optimal parameters, and because of this it is much faster to adapt to new tasks from the online data.
Revised: This way, the meta-leader parameters will be much closer to new task optimal parameters, and because of this it is much faster to adapt to new tasks from the online data.

Original: [Equation]
Revised: [Equation]

Original: where, φ
Revised: where, ϕ

Original: [Equation]
Revised: [Equation]

Original: MAML = argmin
Revised: = argmin

Original: [Equation]
Revised: [Equation]

Original: FTL and FTML try to efﬁciently use knowledge from past tasks to quickly adapt to the new task.
Revised: FTL and FTML try to efficiently use knowledge from past tasks to quickly adapt to the new task.

Original: However, pre-trained weights from FTL do not guarantee fast adaptation, and both methods require ground-truth knowledge of task boundaries.
Revised: However, pre-trained weights from FTL do not guarantee fast adaptation, and both methods require ground-truth knowledge of task boundaries.

Original: This knowledge may not be realistic in many real-world settings, where the tasks may change gradually and no external information is available to indicate task transitions.
Revised: This knowledge may not be realistic in many real-world settings, where the tasks may change gradually and no external information is available to indicate task transitions.

Original: Although FTML can enable fast adaptation, the model needs to be “reset” at each task, essentially creating a “branch” on each task and maintaining two independent learning processes: an adaptation process, whose result is discarded completely at the end of the task, and a meta-training process, which does not inﬂuence the current task at all, and is only used for forward transfer into future tasks.
Revised: Although FTML can enable fast adaptation, the model needs to be “reset” at each task, essentially creating a “branch” on each task and maintaining two independent learning processes: an adaptation process, whose result is discarded completely at the end of the task, and a meta-training process, which does not influence the current task at all, and is only used for forward transfer into future tasks.

Original: For this reason, we argue that FTL and FTML are not fully online.
Revised: For this reason, we argue that FTL and FTML are not fully online.

Original: In this work, our aim is to develop a fully online meta-learning method that continually performs both “fast” updates and “slow” meta-updates, does not need to periodically “reset” the adapted parameters back to the meta-parameters, and does not require any ground truth knowledge of task boundaries.
Revised: In this work, our aim is to develop a fully online meta-learning method that continually performs both “fast” updates and “slow” meta-updates, does not need to periodically “reset” the adapted parameters back to the meta-parameters, and does not require any ground truth knowledge of task boundaries.

Original:  F ULLY O NLINE M ETA -L EARNING W ITHOUT T ASK B OUNDARIES
Revised:  F ULLY O NLINE M ETA -L EARNING W ITHOUT T ASK B OUNDARIES

Original: We ﬁrst discuss the intuition behind how our approach handles online meta-learning without task boundaries.
Revised: We first discuss the intuition behind how our approach handles online meta-learning without task boundaries.

Original: In many real-world tasks, we might expect the tasks in the online data stream to change gradually.
Revised: In many real-world tasks, we might expect the tasks in the online data stream to change gradually.

Original: This makes it very hard to draw a clear boundary between the tasks.
Revised: This makes it very hard to draw a clear boundary between the tasks.

Original: Therefore, it is necessary to relax task boundary assumption if we want a robust online learner that can work on a real-world data stream.
Revised: Therefore, it is necessary to relax task boundary assumption if we want a robust online learner that can work on a real-world data stream.

Original: Additionally, since nearby data points are most likely to belong to the same or similar task, we would expect adaptation to each new data point to be much faster from a model that has already been adapted to other recent data points.
Revised: Additionally, since nearby data points are most likely to belong to the same or similar task, we would expect adaptation to each new data point to be much faster from a model that has already been adapted to other recent data points.

Original: FOML maintains two separate parameter vectors for the online updates ( φ ) and the meta updates ( θ ).
Revised: FOML maintains two separate parameter vectors for the online updates ( ϕ ) and the meta updates ( θ ).

Original: Both parameterize the same architecture, such that f φ and f θ represent the same neural network, but with different weights.
Revised: Both parameterize the same architecture, such that f ϕ and f θ represent the same neural network, but with different weights.

Original: The online model continuously reads in the latest datapoints from the online data stream, and updates the parameters φ in online fashion, without any boundaries or resets.
Revised: The online model continuously reads in the latest datapoints from the online data stream, and updates the parameters ϕ in online fashion, without any boundaries or resets.

Original: However, simply updating the online model on each data point naïvely will not meta-train it to adapt more quickly, and may even result in drift, where the model forgets prior data.
Revised: However, simply updating the online model on each data point naïvely will not meta-train it to adapt more quickly, and may even result in drift, where the model forgets prior data.

Original: Therefore, we also incorporate a regularizer into the online update that is determined by a concurrent meta-learning process (see Fig. 2).
Revised: Therefore, we also incorporate a regularizer into the online update that is determined by a concurrent meta-learning process (see Fig. 2).

Original: Note that FOML only incorporates the meta-parameters into the online updates via the meta-learned regularization term, without ever resetting the online parameters back to the meta-parameters (in contrast, e.g., to FTML
Revised: Note that FOML only incorporates the meta-parameters into the online updates via the meta-learned regularization term, without ever resetting the online parameters back to the meta-parameters (in contrast, e.g., to FTML

Original: (Finn et al., 2019)).
Revised: (Finn et al., 2019)).

Original: The meta-updates of previous MAML-based online meta-learning approaches involve sampling data from all of the tasks seen so far, and then updating the meta-parameters θ based on the derivatives of the MAML objective.
Revised: The meta-updates of previous MAML-based online meta-learning approaches involve sampling data from all of the tasks seen so far, and then updating the meta-parameters θ based on the derivatives of the MAML objective.

Original: This provides a diverse sampling of tasks for the meta update, though it requires storing all of the seen data (Finn et al., 2019).
Revised: This provides a diverse sampling of tasks for the meta update, though it requires storing all of the seen data (Finn et al., 2019).

Original: We also use a MAML-style update for the meta parameters, and also require storing the previously seen data.
Revised: We also use a MAML-style update for the meta parameters, and also require storing the previously seen data.

Original: To this end, we will  B to denote a buffer containing all of the data seen so far.
Revised: To this end, we will use B to denote a buffer containing all of the data seen so far.

Original: Each new datapoint is added to B once the label is observed.
Revised: Each new datapoint is added to B once the label is observed.

Original: However, since we do not assume knowledge of task boundaries, we cannot sample entire tasks from B , but instead must sample individual datapoints.
Revised: However, since we do not assume knowledge of task boundaries, we cannot sample entire tasks from B , but instead must sample individual datapoints.

Original: We therefore adopt a different strategy, which we describe in Section 5.2: as shown in Fig 2, instead of aiming to sample in complete tasks from the data buffer, we simply sample random past datapoints, and meta-train the regularizer so that the online updates retain good performance on all past data.
Revised: We therefore adopt a different strategy, which we describe in Section 5.2: as shown in Fig 2, instead of aiming to sample in complete tasks from the data buffer, we simply sample random past datapoints, and meta-train the regularizer so that the online updates retain good performance on all past data.

Original: We ﬁnd that this strategy is effective at accelerating acquisition of future tasks in online meta-learning settings where the tasks are not mutual exclusive.
Revised: We find that this strategy is effective at accelerating acquisition of future tasks in online meta-learning settings where the tasks are not mutual exclusive.

Original: We deﬁne both types of updates in detail in the next sections.
Revised: We define both types of updates in detail in the next sections.

Original: 5. F ULLY O NLINE A DAPTATION
Revised: 5. F ULLY O NLINE A DAPTATION

Original: At each time step, FOML observes a data point x t , predicts its label ˆ y t , then receives the true label y t and updates the online parameters.
Revised: At each time step, FOML observes a data point x t , predicts its label ˆ y t , then receives the true label y t and updates the online parameters.

Original: In practice, we make updates after observing N new datapoints ( N = 10 in our experiments), so as to reduce the variance of the gradient updates.
Revised: In practice, we make updates after observing N new datapoints ( N = 10 in our experiments), so as to reduce the variance of the gradient updates.

Original: We create a small dataset D jtr with these N datapoints for the time step j .
Revised: We create a small dataset D jtr with these N datapoints for the time step j .

Original: The true label for these datapoints can be from class labels, annotations, rewards, or even self-supervision, though we focus on the supervised classiﬁcation setting in our experiments.
Revised: The true label for these datapoints can be from class labels, annotations, rewards, or even self-supervision, though we focus on the supervised classification setting in our experiments.

Original: However, the online updates are based only on the most recent samples, and do not make use of any past data.
Revised: However, the online updates are based only on the most recent samples, and do not make use of any past data.

Original: Therefore, we need some mechanism for the (slower) meta-training process to “transfer” the knowledge it is distilling from the prior tasks into this online parameter vector.
Revised: Therefore, we need some mechanism for the (slower) meta-training process to “transfer” the knowledge it is distilling from the prior tasks into this online parameter vector.

Original: We can instantiate such a mechanism by introducing a regularizer into the online parameter update that depends on the meta-parameters θ , which we denote as R ( φ, θ ) .
Revised: We can instantiate such a mechanism by introducing a regularizer into the online parameter update that depends on the meta-parameters θ , which we denote as R ( ϕ, θ ) .

Original: While a variety of parameterizations could be used for R ( φ, θ ) , we opt for a simple squared error term of the form R ( φ, θ ) =
Revised: While a variety of parameterizations could be used for R ( ϕ, θ ) , we opt for a simple squared error term of the form R ( ϕ, θ ) =

Original: ( φ − θ ) 2 , resulting in the following online update at each step j :
Revised: ( ϕ − θ ) 2 , resulting in the following online update at each step j :

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: In the case of classiﬁcation, L is the cross-entropy loss.
Revised: In the case of classification, L is the cross-entropy loss.

Original: α 1 , β 1 are hyperparameters.
Revised: α 1 , β 1 are hyperparameters.

Original: Next, we discuss how these meta-parameters are trained so as to maximize the effectiveness of this regularizer at accelerating adaption to new tasks.
Revised: Next, we discuss how these meta-parameters are trained so as to maximize the effectiveness of this regularizer at accelerating adaption to new tasks.

Original: 5. M ETA -L EARNING W ITHOUT T ASK B OUNDARIES
Revised: 5. M ETA -L EARNING W ITHOUT T ASK B OUNDARIES

Original: As discussed in the previous section, the online updates to φ j include a regularizer R that transfers knowledge from the meta-parameters θ into each online update.
Revised: As discussed in the previous section, the online updates to ϕ j include a regularizer R that transfers knowledge from the meta-parameters θ into each online update.

Original: Additionally, our method maintains a buffer B containing all data seen so far, which is used for the meta-update.
Revised: Additionally, our method maintains a buffer B containing all data seen so far, which is used for the meta-update.

Original: In contrast to prior methods, which explicitly draw a training and validation set from the buffer (i.e., a query and support set) and then perform a separate “inner loop” update on this training set (Finn et al., 2019), our meta-updates recycle the inner loop adaptation that is already performed via the online updates, and therefore we only draw a validation set D m val from the buffer B .
Revised: In contrast to prior methods, which explicitly draw a training and validation set from the buffer (i.e., a query and support set) and then perform a separate “inner loop” update on this training set (Finn et al., 2019), our meta-updates recycle the inner loop adaptation that is already performed via the online updates, and therefore we only draw a validation set D mval from the buffer B .

Original: Speciﬁcally, we sample a set of N datapoints at random from B to form D mval .
Revised: Specifically, we sample a set of N datapoints at random from B to form D mval .

Original: We then update the meta-parameters θ using the gradient of the loss on D mval and the regularizer R after the last K updates on φ .
Revised: We then update the meta-parameters θ using the gradient of the loss on D mval and the regularizer R after the last K updates on ϕ .

Original: In other words, we adjust the meta-parameters in such a way that, if an online update is regularized with this meta-weights, then the loss on the online update will be minimized.
Revised: In other words, we adjust the meta-parameters in such a way that, if an online update is regularized with this meta-weights, then the loss on the online update will be minimized.

Original: This can be expressed via following meta update:
Revised: This can be expressed via following meta update:

Original: [Equation]
Revised: [Equation]

Original: Here, θ and φ j are only related via regularization term, unlike FTML (Finn et al., 2019), which sets φ 

Revised: Here, θ and ϕ j are only related via regularization term, unlike FTML (Finn et al., 2019), which sets ϕ

Original: = θ at every task boundary.
Revised: = θ at every task boundary.

Original: The choice of sampling D m val at random from B has several interpretations.
Revised: The choice of sampling D mval at random from B has several interpretations.

Original: We can interpret this as regularizing φ to prevent the online parameters from drifting away from solutions that also work well on the entire data buffer.
Revised: We can interpret this as regularizing ϕ to prevent the online parameters from drifting away from solutions that also work well on the entire data buffer.

Original: However, this interpretation is incomplete, since the meta-update doesn’t simply keep φ close to a single parameter vector that works well on D mval , but rather changes the regularizer so that gradient updates with that regularizer maximally improve performance on
Revised: However, this interpretation is incomplete, since the meta-update doesn’t simply keep ϕ close to a single parameter vector that works well on D mval , but rather changes the regularizer so that gradient updates with that regularizer maximally improve performance on

Original: D mval .
Revised: D mval .

Original: This has the effect of actually accelerating how quickly φ can adapt to new tasks using this regularizer, so long as past tasks are reasonably representative of prior tasks.
Revised: This has the effect of actually accelerating how quickly ϕ can adapt to new tasks using this regularizer, so long as past tasks are reasonably representative of prior tasks.

Original: We experimentally verify this claim in our experiments.
Revised: We experimentally verify this claim in our experiments.

Original: Note, however, that this scheme does assume that the tasks are not mutually exclusive.
Revised: Note, however, that this scheme does assume that the tasks are not mutually exclusive.

Original: In future work, it would also be interesting to explore more sophisticated strategies for sampling D mval , for example by leveraging temporal coherence and drawing N sequential points, which are more likely to belong to the same task.
Revised: In future work, it would also be interesting to explore more sophisticated strategies for sampling D mval , for example by leveraging temporal coherence and drawing N sequential points, which are more likely to belong to the same task.

Original: We summarize the complete algorithm in
Revised: We summarize the complete algorithm in

Original: Algorithm 1.
Revised: Algorithm 1.

Original: 1: procedure M ETA TRAINING 2: Require: θ, B , φ, Buffer B , Data stream S 3: φ 0 ← φ 4: while Data stream S available do 5:
Revised: 1: procedure M ETA TRAINING 2: Require: θ, B , ϕ, Buffer B , Data stream S 3: ϕ 0 ← ϕ 4: while Data stream S available do 5:

Original: D j ← S (cid:46) get new data from online data-stream 6:
Revised: D j ← S ▷ get new data from online data-stream 6:

Original: B ← B + D (cid:46) add new data to the buffer 7:
Revised: B ← B + D ▷ add new data to the buffer 7:

Original: D jtr , D jval ← D (cid:46) partition the data into train and validation splits 8: ˆ y tr ← f φ j − 1 ( D jtr ) (cid:46) make predictions on the train set 9: φ j ← φ j − 1 − α 1 ∇ φ j − 1 {L task ( φ j − 1 ; D jtr ) + β 1 R ( φ j − 1 , θ ) } 10: ˆ y val ← φ j ( D jval )
Revised: D jtr , D jval ← D ▷ partition the data into train and validation splits 8: ˆ y tr ← f ϕ j − 1 ( D jtr ) ▷ make predictions on the train set 9: ϕ j ← ϕ j − 1 − α 1 ∇ ϕ j − 1 {L task ( ϕ j − 1 ; D jtr ) + β 1 R ( ϕ j − 1 , θ ) } 10: ˆ y val ← ϕ j ( D jval )

Original: (cid:46) Evaluate the updated model on the validation set 11:
Revised: ▷ Evaluate the updated model on the validation set 11:

Original: D mval ∼ random-sample ( B ) (cid:46) sample random batch from buffer 12: θ ← θ − α 2 ∇ θ {L task ( φ j ; D mval ) + β 2 (cid:80)
Revised: D mval ∼ random-sample ( B ) ▷ sample random batch from buffer 12: θ ← θ − α 2 ∇ θ {L task ( ϕ j ; D mval ) + β 2 (cid:80)

Original: Kk =1 R ( φ j − k , θ ) } 13: j ← j +
Revised: Kk =1 R ( ϕ j − k , θ ) } 13: j ← j +

Original:  E XPERIMENTAL E VALUATION
Revised:  E XPERIMENTAL E VALUATION

Original: Our experiments focus on online meta-learning tasks based on supervised classiﬁcation.
Revised: Our experiments focus on online meta-learning tasks based on supervised classification.

Original: In these settings, an effective algorithm should adapt to changings tasks as quickly as possible, rapidly identifying when the task has changed and adjusting its parameters accordingly.
Revised: In these settings, an effective algorithm should adapt to changing tasks as quickly as possible, rapidly identifying when the task has changed and adjusting its parameters accordingly.

Original: Furthermore, a successful algorithm should make use of past task shifts to meta-learn effectively, thus accelerating the speed with which it adapts to future task changes.
Revised: Furthermore, a successful algorithm should make use of past task shifts to meta-learn effectively, thus accelerating the speed with which it adapts to future task changes.

Original: In all cases, the algorithm receives one data point at a time, and the task changes periodically.
Revised: In all cases, the algorithm receives one data point at a time, and the task changes periodically.

Original: In order to allow for a comparison with prior methods, which generally assume known task boundaries, the data stream is partitioned into discrete tasks, but our algorithm is not aware of which datapoint belongs to which tasks or where the boundaries occur.
Revised: In order to allow for a comparison with prior methods, which generally assume known task boundaries, the data stream is partitioned into discrete tasks, but our algorithm is not aware of which datapoint belongs to which tasks or where the boundaries occur.

Original: The prior methods, in contrast, are provided with this information, thus giving them an advantage.
Revised: The prior methods, in contrast, are provided with this information, thus giving them an advantage.

Original: We ﬁrst describe the speciﬁc task formulations, and then the prior methods that we compare to.
Revised: We first describe the specific task formulations, and then the prior methods that we compare to.

Original: Online meta-learning algorithms should adapt to each task as quickly as possible, and also use data from past tasks to accelerate acquisition of future tasks.
Revised: Online meta-learning algorithms should adapt to each task as quickly as possible, and also use data from past tasks to accelerate acquisition of future tasks.

Original: Therefore, we report our results as a learning curve, with one axis corresponding to the number of seen tasks, and the other axis corresponding to the cumulative error rate on that task.
Revised: Therefore, we report our results as a learning curve, with one axis corresponding to the number of seen tasks, and the other axis corresponding to the cumulative error rate on that task.

Original: This error rate is computed using a held-out validation data for each task after the adaptation that task.
Revised: This error rate is computed using a held-out validation data for each task after the adaptation that task.

Original: We evaluate prior online meta-learning methods and baselines on two different datasets (Rainbow-MNIST and CIFAR100).
Revised: We evaluate prior online meta-learning methods and baselines on two different datasets (Rainbow-MNIST and CIFAR100).

Original: TOE (train on everything), TFS (train from scratch), FTL (follow the leader) and FTML (follow the meta-leader)
Revised: TOE (train on everything), TFS (train from scratch), FTL (follow the leader) and FTML (follow the meta-leader)

Original: (Finn et al., 2019) are the baseline methods we compare against our method FOML.
Revised: (Finn et al., 2019) are the baseline methods we compare against our method FOML.

Original: See Section 4 for more detailed description of these methods.
Revised: See Section 4 for more detailed description of these methods.

Original: Datasets: We compare TOE, TFS, FTL, FTML and FOML on two different datasets.
Revised: Datasets: We compare TOE, TFS, FTL, FTML and FOML on two different datasets.

Original: RainbowMNIST (Finn et al., 2019) was created by changing the background color, scale and rotation of the
Revised: RainbowMNIST (Finn et al., 2019) was created by changing the background color, scale and rotation of the

Original: MNIST dataset.
Revised: MNIST dataset.

Original: It includes 7 different background colors, 2 scales (full and half) and 4 different rotations.
Revised: It includes 7 different background colors, 2 scales (full and half) and 4 different rotations.

Original: This leads to a total of 56 number of tasks.
Revised: This leads to a total of 56 number of tasks.

Original: Each individual task is to classify the images into 10 classes.
Revised: Each individual task is to classify the images into 10 classes.

Original: We use the same partition with 900 samples per each task, as in prior work (Finn et al., 2019).
Revised: We use the same partition with 900 samples per each task, as in prior work (Finn et al., 2019).

Original: However, this task contains relatively simple images, and only 56 tasks.
Revised: However, this task contains relatively simple images, and only 56 tasks.

Original: To create a much longer task sequence with signiﬁcantly more realistic images, we modiﬁed the CIFARdataset to create an online meta-learning benchmark, which we call online-CIFAR100.
Revised: To create a much longer task sequence with significantly more realistic images, we modified the CIFARdataset to create an online meta-learning benchmark, which we call online-CIFAR100.

Original: Every task is a randomly sampled set of classes, and the goal is to classify whether two images in this set belongs to same class or not.{+50 Number of tasks
 E rr o r r a t e+}
Revised: Every task is a randomly sampled set of classes, and the goal is to classify whether two images in this set belongs {+50 Number of tasks
 E rr o r r a t e+}

Revised: Rainbow-MNIST

Original: Speciﬁcally, each task corresponds to 5 classes from CIFAR-100, and every datapoint consists of a pair{+tasks 1020 2535 
 T e s t E rr o r Online-CIFAR TFSTOEFTLFTMLFOML     50 Number+} of images, each corresponding to one of the 5 classes for that task.
Revised: TFSTOEFTLFTMLFOML 0 200 400 600 800 1000 1200 Number of {+tasks 1020 2535 
 T e s t E rr o r Online-CIFAR TFSTOEFTLFTMLFOML     50 Number+} of tasks

Revised: E rr

Revised: o r r a t e

Revised: Rainbow-MNIST

Revised: FOMLMOCAICARLLWF 0 20 40 60 80 100 120 140 Number of tasks

Revised: T e s t E rr o r

Revised: Online-CELEBA

Revised: TFSTOEFTLFTMLFOML to same class or not.

Revised: Specifically, each task corresponds to 5 classes from CIFAR-100, and every datapoint consists of a pair of images, each corresponding to one of the 5 classes for that task.

Original: The goal is to predict whether the two images belong to the same class or not.
Revised: The goal is to predict whether the two images belong to the same class or not.

Original: Note that different tasks are not mutually exclusive, which in principle should favor a TOE-style method, since meta-learning is[Figure] TFSTOEFTLFTMLOML known to underperform with non-mutually-exclusive tasks (Yin et al., 2019).
Revised: Note that different tasks are not mutually exclusive, which in principle should favor a TOE-style method, since meta-learning is known to underperform with non-mutually-exclusive tasks (Yin et al., 2019).

Original: To make sure the data distribution changes smoothly over tasks, we only change a subset of the classes between consecutive tasks.
Revised: To make sure the data distribution changes smoothly over tasks, we only change a subset of the classes between consecutive tasks.

Original: This allow us to create a very large number of tasks (1200), and evaluate our method and the baselines on much longer and more realistic task sequences.
Revised: This allow us to create a very large number of tasks (1200), and evaluate our method and the baselines on much longer and more realistic task sequences.

Original: Implementation Details: We use a simple 4 layer convolutional neural network with 8,16,32, ﬁlters at each layer for both experiments.
Revised: Implementation Details: We use a simple 4 layer convolutional neural network with 8,16,32, filters at each layer for both experiments.

Original: However, for the CIFAR-100 experiments, a Siamese version of the same network is used.
Revised: However, for the CIFAR-100 experiments, a Siamese version of the same network is used.

Original: All the methods were trained via a cross-entropy loss, with their best performing hyper-parameters, on a single NVIDA-2080 GPU machine.
Revised: All the methods were trained via a cross-entropy loss, with their best performing hyper-parameters, on a single NVIDA-2080 GPU machine.

Original: Please see Appendix ?? for more details on hyper-parameters and network architecture.
Revised: Please see Appendix A. for more details on hyper-parameters and network architecture.

Original: Results on Rainbow-MNIST: As shown in Fig 3, FOML attains the lowest error rate on most tasks in Rainbow-MNIST, except a small segment in the very beginning.
Revised: Results on Rainbow-MNIST: As shown in Fig 3, FOML attains the lowest error rate on most tasks in Rainbow-MNIST, except a small segment in the very beginning.

Original: The performance of TFS is similar performance across all the tasks, and does not improve over time.
Revised: The performance of TFS is similar performance across all the tasks, and does not improve over time.

Original: This is because it resets its weights every time it encounters a new task, and therefore cannot not gain any advantage from past data.
Revised: This is because it resets its weights every time it encounters a new task, and therefore cannot not gain any advantage from past data.

Original: TOE has larger error rates at the start, but as we add more data into the buffer, TOE is able to improve.
Revised: TOE has larger error rates at the start, but as we add more data into the buffer, TOE is able to improve.

Original: On the other hand, both FTL and FTML start with similar performance, but FTML achieve much lower error rates at the end of the sequence compared to FTL, consistently with prior work (Finn et al., 2019).
Revised: On the other hand, both FTL and FTML start with similar performance, but FTML achieve much lower error rates at the end of the sequence compared to FTL, consistently with prior work (Finn et al., 2019).

Original: The ﬁnal error rates of FOML are around 10%, and it reaches this performance signiﬁcantly faster than FTML, after less than 20 tasks.
Revised: The final error rates of FOML are around 10%, and it reaches this performance significantly faster than FTML, after less than 20 tasks.

Original: Note that FTML also has access to task boundaries, while
Revised: Note that FTML also has access to task boundaries, while

Original: FOML does not.
Revised: FOML does not.

Original: Results on Online-CIFAR100: We use a Siamese network for this experiment, where each image is fed into a 7-layer convolutional network, and each branch outputs a 128 dimensional embedding vector.
Revised: Results on Online-CIFAR100: We use a Siamese network for this experiment, where each image is fed into a 7-layer convolutional network, and each branch outputs a 128 dimensional embedding vector.

Original: A difference of these vectors are fed into a fully connected layer for the ﬁnal classiﬁcation.
Revised: A difference of these vectors are fed into a fully connected layer for the final classification.

Original: Each task contains data from 5 classes of CIFAR-100, and each new task introduces three new classes, and retains two of the classes from the previous task, providing a degree of temporal coherence while still requiring each algorithm to handle persistent shift in the task.
Revised: Each task contains data from 5 classes of CIFAR-100, and each new task introduces three new classes, and retains two of the classes from the previous task, providing a degree of temporal coherence while still requiring each algorithm to handle persistent shift in the task.

Original: Fig 3 shows the error rates of various online learning methods, where each method is trained over a sequence of 1200 tasks.
Revised: Fig 3 shows the error rates of various online learning methods, where each method is trained over a sequence of 1200 tasks.

Original: All the methods start with initial error rates of 50%.
Revised: All the methods start with initial error rates of 50%.

Original: The tasks are not mutually exclusive, so in principle TOE can attain good performance, but it makes the slowest progress among all the methods, suggesting that simple pretraining is not sufﬁcient to accelerate learning.
Revised: The tasks are not mutually exclusive, so in principle TOE can attain good performance, but it makes the slowest progress among all the methods, suggesting that simple pretraining is not sufficient to accelerate learning.

Original: FTL uses a similar pre-training strategy as TOE.
Revised: FTL uses a similar pre-training strategy as TOE.

Original: However it has an adaptation stage where the model is ﬁne-tuned on the new task.
Revised: However it has an adaptation stage where the model is fine-tuned on the new task.

Original: This allows it to make slower progress.
Revised: This allows it to make slower progress.

Original: As expected from prior work (Finn et al., 2019), the meta-learning procedure used by FTML allows it to make faster progress than FTL.
Revised: As expected from prior work (Finn et al., 2019), the meta-learning procedure used by FTML allows it to make faster progress than FTL.

Original: However, FOML makes faster progress on average, and achieves the lowest ﬁnal error rate ( ∼ 15% ) afterthe full sequence oftasks.
Revised: However, FOML makes faster progress on average, and achieves the lowest final error rate ( ∼ 15% ) after sequence of 1200 tasks.

Original: 6. A BLATION S TUDIES
Revised: 6. A BLATION S TUDIES

Original: We perform various ablations by varying the number of online parameters used for the meta-update
Revised: We perform various ablations by varying the number of online parameters used for the meta-update

Original: K , importance of meta-modeland sensitive of the regularization term to analysis the properties of our method.
Revised: K , importance of meta-model to analysis the properties of our method.

Original: [Figure]
Revised: [Figure]

Revised: tasks

Original: E rr
Revised: E rr

Original: o r r a t e
Revised: o r r a t e

Original: Online Meta Learning      [Figure] [Figure]
Revised: Online Meta Learning      50 Number of tasks

Original: E rr
Revised: E rr

Original: o r r a t e
Revised: o r r a t e

Original: [Figure]
Revised: [Figure]

Original: Number of online parameters used for the meta-update: Our method periodically updates the online weights and meta weights.
Revised: Number of online parameters used for the meta-update: Our method periodically updates the online weights and meta weights.

Original: The meta-updates involves taking K recent online parameters and updating the meta model via MAML gradient.
Revised: The meta-updates involves taking K recent online parameters and updating the meta model via MAML gradient.

Original: Therefore, meta-updates depends on the trajectory of the online parameters.
Revised: Therefore, meta-updates depend on the trajectory of the online parameters.

Original: In this experiment, we investigate how the performance of FOML changes as we vary the number of parameters used for the meta-update ( K in Algorithm 1).
Revised: In this experiment, we investigate how the performance of FOML changes as we vary the number of parameters used for the meta-update ( K in Algorithm 1).

Original: Fig 4 shows the performance of our method with various values of K : K =
Revised: Fig 4 shows the performance of our method with various values of K : K =

Original: We can see that the performance improves when we update the meta parameters over longer trajectory of online parameters (larger K ).
Revised: We can see that the performance improves when we update the meta parameters over longer trajectory of online parameters (larger K ).

Original: We speculate that this is due to the longer sequences providing a clearer signal for how the meta-parameters inﬂuence online updates over many steps.
Revised: We speculate that this is due to the longer sequences providing a clearer signal for how the meta-parameters influence online updates over many steps.

Original: Importance of meta update: FOML keeps track of separate online parameters and meta-parameters, and each of them is updated via corresponding updates.
Revised: Importance of meta update: FOML keeps track of separate online parameters and meta-parameters, and each of them is updated via corresponding updates.

Original: However, only the online parameters φ are used for evaluation, while the meta-parameters θ only inﬂuence them via the regularizer, and have no direct effect on evaluation performance.
Revised: However, only the online parameters ϕ are used for evaluation, while the meta-parameters θ only influence them via the regularizer, and have no direct effect on evaluation performance.

Original: This might raise the question: how important is the contribution of the meta-parameters to the performance of the algorithm during online training?
Revised: This might raise the question: how important is the contribution of the meta-parameters to the performance of the algorithm during online training?

Original: We train a model with and without meta-updates, and the performance is shown in Fig 4.
Revised: We train a model with and without meta-updates, and the performance is shown in Fig 4.

Original: None that, the model without meta-updates is identical to our method, except that the meta-updates themselves are not performed.
Revised: None that, the model without meta-updates is identical to our method, except that the meta-updates themselves are not performed.

Original: We can clearly see that the model trained with meta-updates preforms much better than a model trained without meta-updates.
Revised: We can clearly see that the model trained with meta-updates preforms much better than a model trained without meta-updates.

Original: The model trained without meta-updates generally does not improve signiﬁcantly as more tasks are seen, while the model trained with meta-updates improves with each task, and reaches signiﬁcantly lower ﬁnal error.
Revised: The model trained without meta-updates generally does not improve significantly as more tasks are seen, while the model trained with meta-updates improves with each task, and reaches significantly lower final error.

Original: This shows that, even though θ and φ are decoupled and only connected via a regularization, the meta-learning component of our method really is critical for its good performance.
Revised: This shows that, even though θ and ϕ are decoupled and only connected via a regularization, the meta-learning component of our method really is critical for its good performance.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented FOML, a MAML-based algorithm for online meta-learning that does not require ground truth knowledge of task boundaries, and does not require resetting the parameter vector back to the meta-learned parameters for every task.
Revised: We presented FOML, a MAML-based algorithm for online meta-learning that does not require ground truth knowledge of task boundaries, and does not require resetting the parameter vector back to the meta-learned parameters for every task.

Original: FOML is conceptually simple, maintaining just two parameter vectors over the entire online adaptation process: a vector of online parameters φ , which are updated continually on each new batch of datapoints, and a vector of meta-parameters θ , which are updated correspondingly with meta-updates to accelerate the online adaptation process, and inﬂuence the online updates via a regularizer.
Revised: FOML is conceptually simple, maintaining just two parameter vectors over the entire online adaptation process: a vector of online parameters ϕ , which are updated continually on each new batch of datapoints, and a vector of meta-parameters θ , which are updated correspondingly with meta-updates to accelerate the online adaptation process, and influence the online updates via a regularizer.

Original: We ﬁnd that even a relatively simple task sampling scheme that selects datapoints at random from a buffer of all seen data enables effective meta-training that accelerates the speed with which FOML can adapt to each new task, and we ﬁnd that FOML reaches a ﬁnal performance that is comparable to or better than baselines and prior methods, while learning to adapt quickly to new tasks signiﬁcantly faster.
Revised: We find that even a relatively simple task sampling scheme that selects datapoints at random from a buffer of all seen data enables effective meta-training that accelerates the speed with which FOML can adapt to each new task, and we find that FOML reaches a final performance that is comparable to or better than baselines and prior methods, while learning to adapt quickly to new tasks significantly faster.

Original: While our work focuses on supervised classiﬁcation problems, a particularly exciting direction for future work is to extend such online meta-learning methods to other types of online supervision that may be more readily available, including self-supervision and prediction, so that models equipped with online meta-learning can continually improve as they see more of the world. 
Revised: While our work focuses on supervised classification problems, a particularly exciting direction for future work is to extend such online meta-learning methods to other types of online supervision that may be more readily available, including self-supervision and prediction, so that models equipped with online meta-learning can continually improve as they see more of the world. A A PPENDIX A. I MPLEMENTATION D ETAILS -R AINBOW MNIST

Revised: All the images from Rainbow-MNIST are in 28 × 28 pixel resolution and with 3 channels.

Revised: We used a 4 layer convolutional neural network for these experiments, with each layer having [32,32,64,64] number of filters.

Revised: After every convolution ReLU activation and max pooling is applied to the features.

Revised: At the last layer we take an average pooling.

Revised: Finally, 64 dimensional features are passed through fully connected layer with a sigmoid activation.

Revised: Since the Rainbow-MNIST uses MNIST data to classify, the final layer has 10 neurons, and the objective is to correctly classify the digits of different scales and colors into 10 classes (actual numbers irrespective of the rotation and background color).

Revised: All the models were trained with 50 gradient updates.

Revised: A. I MPLEMENTATION D ETAILS -ONLINE-CIFAR

Revised: All the images from CIFAR100 are in 32 × 32 pixel resolution and with 3 channels.

Revised: We used a similar 7 layer convolutional neural network in a siamese network architecture for these experiments, with each layer having [32,32,32,64,64,64,128] number of filters.

Revised: After every convolution ReLU activation and batchnorm is applied to the features.

Revised: Every other layers have max-pooling.

Revised: At the last layer we take an average pooling.

Revised: The L2 distance between two images are calculated and a fully connected layer is used to classify the two images as same class or not.

Revised: A. B ASELINE M ETHODS

Revised: TFS:

Revised: Every time this model sees a new task (This needs to know the task boundary), the model resets to new sets of random weights.

Revised: After that the model is trained only using the data from the new task.

Revised: At the same time, we also rest the optimizer state to remove any effect of momentum from previous updates.

Revised: The model is trained with Adam optimizer with a learning rate of 0.001.

Revised: TOE:

Revised: Although TOE optimize the parameters on the all available task, training the model using all the available data is practically impossible.

Revised: Therefore, we sample datapoints from the buffer and update the model parameters.

Revised: However, since the number of data is increasing over time, the model also needs be updated on a good representative data of the online stream.

Revised: Therefore, we increase the number of gradient updates over time.

Revised: For ONLINE-CIFAR100 experiment, the number of gradients updates are increased by 10 for every 100 tasks.

Revised: The model is trained with Adam optimizer with a learning rate of 0.001.

Revised: FTL: This method, first pretrains a model on the past data, and then fine-tune it on the very recent samples.

Revised: Similar to TOE, we first train a set of weights using the datapoints in the data buffer.

Revised: After that, the model is fine-tuned on the correct task data.

Revised: However, after this adaptation and evaluation the fine-tuned weights are simply discarded and the pretraining starts from the initial weights of the adaptation process.

Revised: Adam optimizer with same configuration is used to train this model.

Revised: FTML:

Revised: The FTML updates the meta-parameters using MAML style update.

Revised: For that, we trained the FTML with 5 inner-loop updates (each inner-loop have task-specific data for the inner-loop adaptation).

Revised: After the meta-update, the meta-parameters are used as the initialization for the innerloop adaptation, and similar to FTL the adapted weights are simply discarded after evaluation.

Revised: The inner-loop is trained with SGD with a learning rate of 0.001 and the outer-loop is trained with a learning rate of 0.0005.

Revised: FOML: Our method also have two sets of parameter vectors, thus two different optimizers.

Revised: We use Adam for both online and meta updates with a learning rate of 0.001 for both cases.

Revised: Since, our online updates share the meta-knowledge via the regularization term, the β 1 controls how much pull is applied to the meta parameters by the online parameters.

Revised: We use 0.01 for β 1 .

Revised: Similarly the pull on meta-parameters by the online parameters is controlled by β 2 , and we use 0.001 in our experiments.