Original: V ARIABLE -S HOT A DAPTATION FOR I NCREMENTAL M ETA -L EARNING
Revised: V ARIABLE -S HOT A DAPTATION FOR I NCREMENTAL M ETA -L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Few-shot meta-learning methods consider the problem of learning new tasks from a small, ﬁxed number of examples, by meta-learning across static data from a set of previous tasks.
Revised: Few-shot meta-learning methods consider the problem of learning new tasks from a small, ﬁxed number of examples, by meta-learning across static data from a set of previous tasks.

Original: However, in many real world settings, it is more natural to view the problem as one of minimizing the total amount of supervision — both the number of examples needed to learn a new task and the amount of data needed for metalearning.
Revised: However, in many real world settings, it is more natural to view the problem as one of minimizing the total amount of supervision — both the number of examples needed to learn a new task and the amount of data needed for metalearning.

Original: Such a formulation can be studied in a sequential learning setting, where tasks are presented in sequence.
Revised: Such a formulation can be studied in a sequential learning setting, where tasks are presented in sequence.

Original: When studying meta-learning in this online setting, a critical question arises: can meta-learning improve over the sample complexity and regret of standard empirical risk minimization methods, when considering both meta-training and adaptation together?
Revised: When studying meta-learning in this online setting, a critical question arises: can meta-learning improve over the sample complexity and regret of standard empirical risk minimization methods, when considering both meta-training and adaptation together?

Original: The answer is particularly non-obvious for meta-learning algorithms with complex bi-level optimizations that may demand large amounts of meta-training data.
Revised: The answer is particularly non-obvious for meta-learning algorithms with complex bi-level optimizations that may demand large amounts of meta-training data.

Original: To answer this question, we extend previous meta-learning algorithms to handle the variable-shot settings that naturally arise in sequential learning: from many-shot learning at the start, to zero-shot learning towards the end.
Revised: To answer this question, we extend previous meta-learning algorithms to handle the variable-shot settings that naturally arise in sequential learning: from many-shot learning at the start, to zero-shot learning towards the end.

Original: On sequential learning problems, we ﬁnd that meta-learning solves the full task set with fewer overall labels and achieves greater cumulative performance, compared to standard supervised methods.
Revised: On sequential learning problems, we ﬁnd that meta-learning solves the full task set with fewer overall labels and achieves greater cumulative performance, compared to standard supervised methods.

Original: These results suggest that meta-learning is an important ingredient for building learning systems that continuously learn and improve over a sequence of problems.
Revised: These results suggest that meta-learning is an important ingredient for building learning systems that continuously learn and improve over a sequence of problems.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Standard machine learning methods typically consider a static training set, with a discrete training phase and test phase.
Revised: Standard machine learning methods typically consider a static training set, with a discrete training phase and test phase.

Original: However, in the real world, this process is almost always cyclical: machine learning systems might be improved with the acquisition of new data, repurposed for new tasks via ﬁnetuning, or might simply need to be adjusted to suit the needs of a changing, non-stationary world.
Revised: However, in the real world, this process is almost always cyclical: machine learning systems might be improved with the acquisition of new data, repurposed for new tasks via ﬁnetuning, or might simply need to be adjusted to suit the needs of a changing, non-stationary world.

Original: Indeed, the real world is arguably so complex that, for all practical purposes, learning is never truly ﬁnished, and any real system in open-world settings will need to improve and ﬁnetune perpetually (Chen & Asch, 2017; Zhao et al., 2019).
Revised: Indeed, the real world is arguably so complex that, for all practical purposes, learning is never truly ﬁnished, and any real system in open-world settings will need to improve and ﬁnetune perpetually (Chen & Asch, 2017; Zhao et al., 2019).

Original: In this continual learning process, metalearning provides the appealing prospect of accelerating how quickly new tasks can be acquired using past experience, which in principle should make the learning system more and more efﬁcient over the course of its lifetime.
Revised: In this continual learning process, metalearning provides the appealing prospect of accelerating how quickly new tasks can be acquired using past experience, which in principle should make the learning system more and more efﬁcient over the course of its lifetime.

Original: However, current meta-learning methods are typically concerned with asymptotic few-shot performance (Finn et al., 2017; Snell et al., 2017).
Revised: However, current meta-learning methods are typically concerned with asymptotic few-shot performance (Finn et al., 2017; Snell et al., 2017).

Original: For a continual learning system of this sort, we instead need a method that can minimize both the number of examples per task, and the number of tasks needed to accelerate the learning process.
Revised: For a continual learning system of this sort, we instead need a method that can minimize both the number of examples per task, and the number of tasks needed to accelerate the learning process.

Original: Few-shot meta-learning algorithms aim to learn the structure that underlies data coming from a set of related tasks, and use this structure to learn new tasks with only a few datapoints.
Revised: Few-shot meta-learning algorithms aim to learn the structure that underlies data coming from a set of related tasks, and use this structure to learn new tasks with only a few datapoints.

Original: While these algorithms enable efﬁcient learning for new tasks at test time, it is not clear if these efﬁciency gains persist in online learning settings, where the efﬁciency of both meta-training and few-shot adaptation is critical.
Revised: While these algorithms enable efﬁcient learning for new tasks at test time, it is not clear if these efﬁciency gains persist in online learning settings, where the efﬁciency of both meta-training and few-shot adaptation is critical.

Original: Indeed, simply training a model on all received data, i.e. standard supervised learning with empirical risk minimization, is a strong competitor since supervised learning methods are known to generalize well to in-distribution tasks in a zero-shot manner.
Revised: Indeed, simply training a model on all received data, i.e. standard supervised learning with empirical risk minimization, is a strong competitor since supervised learning methods are known to generalize well to in-distribution tasks in a zero-shot manner.

Original: Moreover, it’s not clear that meta-learning algorithms can improve over such methods by leveraging shared task structure in online learning settings.
Revised: Moreover, it’s not clear that meta-learning algorithms can improve over such methods by leveraging shared task structure in online learning settings.

Original: Provided that it is possible for a single model to fully master all of the tasks with enough data, both meta-learning and standard empirical risk minimization approaches should produce a model of equal competence.
Revised: Provided that it is possible for a single model to fully master all of the tasks with enough data, both meta-learning and standard empirical risk minimization approaches should produce a model of equal competence.

Original: However, the key hypothesis of this work is that meta-learned models will become more accurate more quickly in the middle of the online learning process, while data is still being collected, resulting in lower overall regret in realistic problem settings.
Revised: However, the key hypothesis of this work is that meta-learned models will become more accurate more quickly in the middle of the online learning process, while data is still being collected, resulting in lower overall regret in realistic problem settings.

Original: To test this hypothesis, we consider a practical online learning problem setting, which we refer to as online incremental learning, where the algorithm must learn a sequence of tasks, and datapoints from each task are received sequentially.
Revised: To test this hypothesis, we consider a practical online learning problem setting, which we refer to as online incremental learning, where the algorithm must learn a sequence of tasks, and datapoints from each task are received sequentially.

Original: Once a model reaches a certain level of proﬁciency, the algorithm may move on to training the model on the next task.
Revised: Once a model reaches a certain level of proﬁciency, the algorithm may move on to training the model on the next task.

Original: This problem is crucial to solve in the real world, especially in settings where online data collection and supervision signals are costly to obtain.
Revised: This problem is crucial to solve in the real world, especially in settings where online data collection and supervision signals are costly to obtain.

Original: One example of such a problem deﬁnition is a setting where a company receives requests for object classiﬁers, sequentially at different points in time.
Revised: One example of such a problem deﬁnition is a setting where a company receives requests for object classiﬁers, sequentially at different points in time.

Original: Data collection and labels are expensive, and the company wants to spend the smallest amount of money on acquiring a good classiﬁer for each request.
Revised: Data collection and labels are expensive, and the company wants to spend the smallest amount of money on acquiring a good classiﬁer for each request.

Original: A major challenge for meta-learning that arises in this problem setting is to design a meta-learning algorithm that can generalize with variable shots: As data from new tasks is incrementally introduced, at any given point in time, the model may have access to zero, a few, or many datapoints for a provided task.
Revised: A major challenge for meta-learning that arises in this problem setting is to design a meta-learning algorithm that can generalize with variable shots: As data from new tasks is incrementally introduced, at any given point in time, the model may have access to zero, a few, or many datapoints for a provided task.

Original: The goal of this work is to achieve variable-shot adaptation while minimizing the total amount of supervision in terms of number of shots required for each added task.
Revised: The goal of this work is to achieve variable-shot adaptation while minimizing the total amount of supervision in terms of number of shots required for each added task.

Original: The desired incremental meta-learning algorithm is expected to generalize to a new task with decreasing number of shots over the course of training.
Revised: The desired incremental meta-learning algorithm is expected to generalize to a new task with decreasing number of shots over the course of training.

Original: We visualize our problem setting in Figure 1.
Revised: We visualize our problem setting in Figure 1.

Original: The key contributions of this work are (a) a new meta-learning algorithm that can adapt to variable amounts of data, and (b) an online version of this algorithm that addresses the above problem setting of online incremental learning.
Revised: The key contributions of this work are (a) a new meta-learning algorithm that can adapt to variable amounts of data, and (b) an online version of this algorithm that addresses the above problem setting of online incremental learning.

Original: We theoretically derive our variable-shot meta-learning algorithm and combine it with deep neural networks for effective online learning on challenging sequential problem settings.
Revised: We theoretically derive our variable-shot meta-learning algorithm and combine it with deep neural networks for effective online learning on challenging sequential problem settings.

Original: Perhaps surprisingly, we ﬁnd that our approach can outperform empirical risk minimization and a previous online meta-learning method (Finn et al., 2019) on two online image classiﬁcation problems consisting of sequences of classiﬁcation tasks and one online regression problem.
Revised: Perhaps surprisingly, we ﬁnd that our approach can outperform empirical risk minimization and a previous online meta-learning method (Finn et al., 2019) on two online image classiﬁcation problems consisting of sequences of classiﬁcation tasks and one online regression problem.

Original: Further, we ﬁnd that, in the ofﬂine setting, our approach performs comparably to previous state-of-the-art algorithms in few-shot learning, and provides considerable gains in the variable-shot setting.
Revised: Further, we ﬁnd that, in the ofﬂine setting, our approach performs comparably to previous state-of-the-art algorithms in few-shot learning, and provides considerable gains in the variable-shot setting.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Meta-learning algorithms optimize for efﬁcient adaptation to new tasks.
Revised: Meta-learning algorithms optimize for efﬁcient adaptation to new tasks.

Original: To deﬁne the meta-learning problem, let p ( T ) denote a task distribution , where each task T i ∼ p ( T ) consists of a dataset
Revised: To deﬁne the meta-learning problem, let p ( T ) denote a task distribution , where each task T i ∼ p ( T ) consists of a dataset

Original: D i :=
Revised: D i :=

Original: { xi , yi } with i.i.d.
Revised: { x , y } with i.i.d.

Original: input and output pairs.
Revised: input and output pairs.

Original: If we have a predictive model h ( x ; θ ) with some parameter θ and a loss function (cid:96) , such as the cross entropy between the predicted label distribution and the true label distribution in a classiﬁcation problem, the risk of T i , f i , can be computed as f i ( θ ) =
Revised: If we have a predictive model h ( x ; θ ) with some parameter θ and a loss function (cid:96) , such as the cross entropy between the predicted label distribution and the true label distribution in a classiﬁcation problem, the risk of T i , f i , can be computed as f i ( θ ) =

Original: E ( xi , yi )
Revised: E ( x , y )

Original: ∼D i [ (cid:96) ( h ( xi ; θ ) , yi )] .
Revised: ∼D i [ (cid:96) ( h ( x ; θ ) , y )] .

Original: At meta-training time, N tasks {T i } Ni =1 are sampled from p ( T ) and meta-learning algorithms aim to learn how to quickly learn these tasks such that, at meta-test time, the population risk f j ( θ ) of the unseen tasks T j ∼ p ( T ) is minimized quickly.
Revised: At meta-training time, N tasks {T i } Ni =1 are sampled from p ( T ) and meta-learning algorithms aim to learn how to quickly learn these tasks such that, at meta-test time, the population risk f j ( θ ) of the unseen tasks T j ∼ p ( T ) is minimized quickly.

Original: In this work, we build on the model-agnostic meta-learning (MAML) algorithm
Revised: In this work, we build on the model-agnostic meta-learning (MAML) algorithm

Original: (Finn et al., 2017).
Revised: (Finn et al., 2017).

Original: MAML achieves fast adaptation to new tasks by optimizing a set of initial parameters θ MAML that can be quickly adapted to the meta-training tasks {T i } Ni =1 .
Revised: MAML achieves fast adaptation to new tasks by optimizing a set of initial parameters θ MAML that can be quickly adapted to the meta-training tasks {T i } Ni =1 .

Original: Thus, at meta-test time, after a small number of gradient steps on θ MAML with K datapoints from D j , the model can minimize f j for the new task
Revised: Thus, at meta-test time, after a small number of gradient steps on θ MAML with K datapoints from D j , the model can minimize f j for the new task

Original: T j .
Revised: T j .

Original: Note that K is a small and ﬁxed number across all tasks.
Revised: Note that K is a small and ﬁxed number across all tasks.

Original: Formally, MAML achieves such an initialization by optimizing the following objective: min θ MAML 1 N (cid:80) Ni =
Revised: Formally, MAML achieves such an initialization by optimizing the following objective: min θ MAML 1 N (cid:80) Ni =

Original: 1 f i ( U i ( θ MAML , α, K i )) where α is the inner gradient step size, U i ( θ, α, K i )
Revised: 1 f i ( U i ( θ MAML , α, K i )) where α is the inner gradient step size, U i ( θ, α, K i )

Original: =
Revised: =

Original: θ − α ∇ θ ˆ f Ki ( θ ) denotes the gradient update in the inner loop for task i , and ˆ f Ki ( θ ) = 1 K (cid:80) Kj
Revised: θ − α ∇ θ ˆ f Ki ( θ ) denotes the gradient update in the inner loop for task i , and ˆ f Ki ( θ ) = 1 K (cid:80) Kj

Original: =1 (cid:96) ( h ( x j ; θ ) , y j ) is the empirical risk of T i based on a minibatch with K datapoints sampled from D i .
Revised: =1 (cid:96) ( h ( x j ; θ ) , y j ) is the empirical risk of T i based on a minibatch with K datapoints sampled from D i .

Original: Hence, MAML can be viewed as optimizing for
Revised: Hence, MAML can be viewed as optimizing for

Original: K-shot generalization.
Revised: K-shot generalization.

Original: In the online setting, prior work (Finn et al., 2019) has proposed the follow-the-meta-leader (FTML) algorithm, which adapts MAML to an online meta-learning setting where the meta-learner learns to adapt to tasks that arrive in a sequence.
Revised: In the online setting, prior work (Finn et al., 2019) has proposed the follow-the-meta-leader (FTML) algorithm, which adapts MAML to an online meta-learning setting where the meta-learner learns to adapt to tasks that arrive in a sequence.

Original: In this online meta-learning setting, the meta-learner faces a sequence of loss functions { (cid:96) t } ∞ t =1 in each round t , where each of the tasks share some common structure that could be captured by the meta-learner.
Revised: In this online meta-learning setting, the meta-learner faces a sequence of loss functions { (cid:96) t } ∞ t =1 in each round t , where each of the tasks share some common structure that could be captured by the meta-learner.

Original: The goal of the meta-learner is to ﬁnd a sequence of models parametrized by { θ t } ∞ t =1 that performs well on the sequence of loss functions after making some task-speciﬁc update procedures U t ( θ t , α, K ) : θ t ∈ Θ , K ∈ N → θ (cid:48)
Revised: The goal of the meta-learner is to ﬁnd a sequence of models parametrized by { θ t } ∞ t =1 that performs well on the sequence of loss functions after making some task-speciﬁc update procedures U t ( θ t , α, K ) : θ t ∈ Θ , K ∈ N → θ (cid:48)

Original: ∈ Θ where K is a ﬁxed number of datapoints used to perform the update.
Revised: ∈ Θ where K is a ﬁxed number of datapoints used to perform the update.

Original: In FTML, which adapts MAML to the online setting,
Revised: In FTML, which adapts MAML to the online setting,

Original: U t ( θ t , α, K ) , as deﬁned in the paragraph above, is the inner gradient update w.r.t.
Revised: U t ( θ t , α, K ) , as deﬁned in the paragraph above, is the inner gradient update w.r.t.

Original: the empirical risk with K -shot data ˆ f Kt at round t .
Revised: the empirical risk with K -shot data ˆ f Kt at round t .

Original: The performance of FTML is typically measured using the regret that compares the sum of population risks to the cumulative population risks with some ﬁxed model
Revised: The performance of FTML is typically measured using the regret that compares the sum of population risks to the cumulative population risks with some ﬁxed model

Original: [Table]
Revised: [Table]

Original: Analogously to the standard follow-the-leader (FTL) algorithm
Revised: Analogously to the standard follow-the-leader (FTL) algorithm

Original: (Hannan, 1957; Kalai & Vempala, 2005), FTML optimizes θ t as θ t +1 = arg min θ (cid:80) tj =1 f j ( U j ( θ j , α, K )) .
Revised: (Hannan, 1957; Kalai & Vempala, 2005), FTML optimizes θ t as θ t +1 = arg min θ (cid:80) tj =1 f j ( U j ( θ, α, K )) .

Original: As mentioned above, the number of datapoints used for computing updates, K , is always ﬁxed in both ofﬂine and online meta-learning settings.
Revised: As mentioned above, the number of datapoints used for computing updates, K , is always ﬁxed in both ofﬂine and online meta-learning settings.

Original: However, in the sequential learning setting, where datapoints for each task arrive incrementally, the model is required to handle varying numbers of datapoints from different tasks, and hence needs to generalize to new tasks quickly using any amount of data.
Revised: However, in the sequential learning setting, where datapoints for each task arrive incrementally, the model is required to handle varying numbers of datapoints from different tasks, and hence needs to generalize to new tasks quickly using any amount of data.

Original: Our method is designed to tackle this problem setting, which we will discuss in the next section.
Revised: Our method is designed to tackle this problem setting, which we will discuss in the next section.

Original:  I NCREMENTAL M ETA -L EARNING P ROBLEM S TATEMENT
Revised:  I NCREMENTAL M ETA -L EARNING P ROBLEM S TATEMENT

Original: The incremental meta-learning problem statement extends the online meta-learning setting (Finn et al., 2019), where the tasks arrive one at a time.
Revised: The incremental meta-learning problem statement extends the online meta-learning setting (Finn et al., 2019), where the tasks arrive one at a time.

Original: In our proposed incremental setting, the data within each task also arrives one data point at a time, and the goal of the model is to minimize the cumulative regret summed over all of the tasks.
Revised: In our proposed incremental setting, the data within each task also arrives one data point at a time, and the goal of the model is to minimize the cumulative regret summed over all of the tasks.

Original: A model that can adapt to a new task more quickly in fewer shots should attain lower regret.
Revised: A model that can adapt to a new task more quickly in fewer shots should attain lower regret.

Original: A model that can do so while having seen as few previous tasks as possible will be even better.
Revised: A model that can do so while having seen as few previous tasks as possible will be even better.

Original: Formally, the learning process is divided into “tasks” and “shots”.
Revised: Formally, the learning process is divided into “tasks” and “shots”.

Original: Each time step, the learner either receives one more “shot” (data point) for the current task, or transitions to a new task and receives the ﬁrst shot for that task.
Revised: Each time step, the learner either receives one more “shot” (data point) for the current task, or transitions to a new task and receives the ﬁrst shot for that task.

Revised: In practice, instead of generating 1 datapoint every time step, the learner receives a small batch of datapoints every few time steps.

Original: Let t denote the current task counter, and s the current shot counter.
Revised: Let t denote the current task counter, and s the current shot counter.

Original: At each step s for task t , the model has access to ˆ D t ( s ) , the dataset for the current task, which presently contains s shots.
Revised: At each step s for task t , the model has access to ˆ D t ( s ) , the dataset for the current task, which presently contains s shots.

Original: The model can adapt to the current task using ˆ D t ( s ) , and must then attain the lowest possible risk on that task.
Revised: The model can adapt to the current task using ˆ D t ( s ) , and must then attain the lowest possible risk on that task.

Original: To realize this setting, we extend the update procedure deﬁned in Section 2 to the variable-shot setting, formally U t ( θ t , α, s ) .
Revised: To realize this setting, we extend the update procedure deﬁned in Section 2 to the variable-shot setting, formally U t ( θ t , α, s ) .

Original: The risk in round t with s shots in this incremental setting is deﬁned as f t ( U t ( θ t , α, min { s, M } )) where M is some large scalar representing the maximum number of shots per task.
Revised: The risk in round t with s shots in this incremental setting is deﬁned as f t ( U t ( θ t , α, min { s, M } )) where M is some large scalar representing the maximum number of shots per task.

Original: The risk deﬁned above suggests that the performance of the model is only evaluated using the s -shot risk f t ( U t ( θ t , α, s )) , when there are s < =
Revised: The risk deﬁned above suggests that the performance of the model is only evaluated using the s -shot risk f t ( U t ( θ t , α, s )) , when there are s < =

Original: M datapoints in the current datasets, and always use M -shot risk when there are more than M datapoints.
Revised: M datapoints in the current datasets, and always use M -shot risk when there are more than M datapoints.

Original: Now that we have deﬁned our measure of incremental risk, we will discuss when the model can transition from one task to the next.
Revised: Now that we have deﬁned our measure of incremental risk, we will discuss when the model can transition from one task to the next.

Original: Intuitively, we want the model to be able to “complete” a task and transition to a new task as soon as it has achieved a requisite accuracy on that task.
Revised: Intuitively, we want the model to be able to “complete” a task and transition to a new task as soon as it has achieved a requisite accuracy on that task.

Original: For instance, in the example in Section 1 of the company that receives requests for object classiﬁers, the company might choose to stop collecting training data for a given task once accuracy on that task is good enough, and move on to the next customer.
Revised: For instance, in the example in Section 1 of the company that receives requests for object classiﬁers, the company might choose to stop collecting training data for a given task once accuracy on that task is good enough, and move on to the next customer.

Original: We formalize this by introducing a threshold value C , such that when f t ( U t ( θ t , α, s )) ≤ C at step s , the learner switches to the next task immediately.
Revised: We formalize this by introducing a threshold value C , such that when f t ( U t ( θ t , α, s )) ≤ C at step s , the learner switches to the next task immediately.

Original: In this way, a very efﬁcient model that can adapt and master a task in just a few data points will attain lower overall regret and solve the full sequence of tasks with fewer overall datapoints, since it will need fewer shots for each task.
Revised: In this way, a very efﬁcient model that can adapt and master a task in just a few data points will attain lower overall regret and solve the full sequence of tasks with fewer overall datapoints, since it will need fewer shots for each task.

Original: To summarize, the overall incremental meta-learning process looks like this: (1) At the start of round t , the meta-learner selects a model parameterized by parameter θ t ; (2) The world chooses a task T t ∼ p ( T ) ; (3) At each step s = 0 , . .
Revised: To summarize, the overall incremental meta-learning process looks like this: (1) At the start of round t , the meta-learner selects a model parameterized by parameter θ t ; (2) The world chooses a task T t ∼ p ( T ) ; (3) At each step s = 0 , . .

Original: , the dataset ˆ D t ( s ) receives one datapoint; (4) At step s = 0 , . .
Revised: , the dataset ˆ D t ( s ) receives one datapoint; (4) At step s = 0 , . .

Original: . , the model updates θ t as θ (cid:48)
Revised: . , the model updates θ t as θ (cid:48)

Original: t =
Revised: t =

Original: U t ( θ t , α, s ) ; (5) At step s = 0 , . . .
Revised: U t ( θ t , α, s ) ; (5) At step s = 0 , . . .

Original: , the model is evaluated with f t ( U t ( θ t , α, s )) and if f t ( U t ( θ t , α, s )) ≤ C , the agent will advance to the next round.
Revised: , the model is evaluated with f t ( U t ( θ t , α, s )) and if f t ( U t ( θ t , α, s )) ≤ C , the agent will advance to the next round.

Original: When the advancement happens, denote S t =
Revised: When the advancement happens, denote S t =

Original: s .
Revised: s .

Original: The objective of incremental meta-learning is to minimize the regret over all the rounds as follows: 
Revised: The objective of incremental meta-learning is to minimize the regret over all the rounds, resembling the objective of online learning (Hannan, 1957; Kalai & Vempala, 2005), which we state as follows: Regret T =

Original: [Equation]
Revised: [Equation]

Original:  where { θ t } Tt =1 is a sequence of models at round t = 1 , . .
Revised: M } )) [Equation] where { θ t } Tt =1 is a sequence of models at round t = 1 , . .

Original: , T .
Revised: , T .

Original: Similar to FTML, we aim to obtain a meta-learner that learns each task in the data-incremental setting, while attaining minimal regret.
Revised: Similar to FTML, we aim to obtain a meta-learner that learns each task in the data-incremental setting, while attaining minimal regret.

Original:  V ARIABLE -S HOT M ODEL -A GNOSTIC M ETA -L EARNING
Revised:  V ARIABLE -S HOT M ODEL -A GNOSTIC M ETA -L EARNING

Original: The ﬁrst step toward designing an incremental meta-learning algorithm that can address this problem statement is to design a method that can handle variable numbers of shots, since in the incremental setting, the model will need to accommodate settings ranging from zero-shot at the beginning of each task to many-shot at the end.
Revised: The ﬁrst step toward designing an incremental meta-learning algorithm that can address this problem statement is to design a method that can handle variable numbers of shots, since in the incremental setting, the model will need to accommodate settings ranging from zero-shot at the beginning of each task to many-shot at the end.

Original: Attaining the lowest regret therefore depends crucially on maximally utilizing whatever task data may be available.
Revised: Attaining the lowest regret therefore depends crucially on maximally utilizing whatever task data may be available.

Original: We will therefore ﬁrst design a variable-shot generalization of MAML in the standard ofﬂine setting, and then apply it to the online setting to provide a viable approach for incremental meta-learning.
Revised: We will therefore ﬁrst design a variable-shot generalization of MAML in the standard ofﬂine setting, and then apply it to the online setting to provide a viable approach for incremental meta-learning.

Original: 4. A N A ¨ IVE S OLUTION
Revised: 4. A N A ¨ IVE S OLUTION

Original: A naive solution would simply involve applying MAML with varying numbers of datapoints used to compute the inner graduate updates.
Revised: A naive solution would simply involve applying MAML with varying numbers of datapoints used to compute the inner graduate updates.

Original: During meta-training, the number of datapoints could be drawn uniformly from { 0 , . . .
Revised: During meta-training, the number of datapoints could be drawn uniformly from { 0 , . . .

Original: , M } for some M ∈ Z + .
Revised: , M } for some M ∈ Z + .

Original: The corresponding meta-training objective is then given as min θ 1 N (cid:80) Ni =
Revised: The corresponding meta-training objective is then given as min θ 1 N (cid:80) Ni =

Original: 1 E s ∼ Unif (0 ,M ) (cid:104)
Revised: 1 E s ∼ Unif (0 ,M ) (cid:104)

Original: f
Revised: f

Original: i ( θ − α ∇ θ ˆ f si ( θ )) (cid:105) .
Revised: i ( θ − α ∇ θ ˆ f si ( θ )) (cid:105) .

Original: Unfortunately, as we will see in Section 7, this approach does not perform well when the number of shots varies, since the adaptation process is not aware of the number of shots that are available.
Revised: Unfortunately, as we will see in Section 7, this approach does not perform well when the number of shots varies, since the adaptation process is not aware of the number of shots that are available.

Original: Intuitively, when more data is available, the model should be able to deviate further from the prior parameter vector θ , since the larger dataset provides us with more information about the task.
Revised: Intuitively, when more data is available, the model should be able to deviate further from the prior parameter vector θ , since the larger dataset provides us with more information about the task.

Original: Mathematically, this can be captured by the variance of the inner gradient.
Revised: Mathematically, this can be captured by the variance of the inner gradient.

Original: The variance of the inner gradient depends on s as follows:
Revised: The variance of the inner gradient depends on s as follows:

Original: Var ( ∇ θ ˆ f si ( θ )) =
Revised: Var ( ∇ θ ˆ f si ( θ )) =

Original: Var (cid:16) ∇ θ 1 s (cid:80) sj =1 (cid:96) ( h ( x j ; θ ) , y j ) (cid:17)
Revised: Var (cid:16) ∇ θ 1 s (cid:80) sj =1 (cid:96) ( h ( x j ; θ ) , y j ) (cid:17)

Original: = 1 s Var ( ∇ θ (cid:96) ( h ( x ; θ ) , y )) where the last equality follows from the property of the variance of the sum of i.i.d.
Revised: = 1 s Var ( ∇ θ (cid:96) ( h ( x ; θ ) , y )) where the last equality follows from the property of the variance of the sum of i.i.d.

Original: random variables.
Revised: random variables.

Original: The above equation indicates that the inner gradient variance inversely scales with the number of shots, but the model is unaware of this fact, since it is only updated via the average gradient.
Revised: The above equation indicates that the inner gradient variance inversely scales with the number of shots, but the model is unaware of this fact, since it is only updated via the average gradient.

Original: Therefore, the model has no way to know that it should “trust” the gradient more when more shots are available, leading to poor performance in the variable shot setting.
Revised: Therefore, the model has no way to know that it should “trust” the gradient more when more shots are available, leading to poor performance in the variable shot setting.

Original: This is also shown empirically in Section 7.1.
Revised: This is also shown empirically in Section 7.1.

Original: 4. T HE L EARNING R ATE S CALING M ETHOD
Revised: 4. T HE L EARNING R ATE S CALING M ETHOD

Original: Intuitively, if we have a lower variance gradient, we can take larger steps along that gradient.
Revised: Intuitively, if we have a lower variance gradient, we can take larger steps along that gradient.

Original: Therefore, a natural way to vary the model update based on the number of datapoints is to change the inner learning rate.
Revised: Therefore, a natural way to vary the model update based on the number of datapoints is to change the inner learning rate.

Original: It is well known in standard supervised learning that the optimal learning rate scales with batch size (Smith & Le, 2018).
Revised: It is well known in standard supervised learning that the optimal learning rate scales with batch size (Smith & Le, 2018).

Original: One way to incorporate this observation into a variableshot MAML algorithm would be to meta-learn different learning rates for each number of shots.
Revised: One way to incorporate this observation into a variable-shot MAML algorithm would be to meta-learn different learning rates for each number of shots.

Original: As we will see in Section 7.1, this approach, which we call MAML-VL, can indeed improve over standard MAML.
Revised: As we will see in Section 7.1, this approach, which we call MAML-VL (variable-shot and learning), can indeed improve over standard MAML.

Original: However, this requires meta-learning an additional parameter for every possible number of shots, which can become impractical as the maximum number of shots increases.
Revised: However, this requires meta-learning an additional parameter for every possible number of shots, which can become impractical as the maximum number of shots increases.

Original: Instead, we can derive a simple rule for the optimal learning, which we will show can work even better than meta-learning separate rates for each number of shots.
Revised: Instead, we can derive a simple rule for the optimal learning, which we will show can work even better than meta-learning separate rates for each number of shots.

Original: First, let us deﬁne U t
Revised: First, let us deﬁne U t

Original: ( θ, α )
Revised: ( θ, α )

Original: =
Revised: =

Original: θ − α ∇ θ f t ( θ ) to be the ideal update rule given the average gradient.
Revised: θ − α ∇ θ f t ( θ ) to be the ideal update rule given the average gradient.

Original: We can view this as the update rule for the limiting case of inﬁnite shots, since U t ( θ, α ) =
Revised: We can view this as the update rule for the limiting case of inﬁnite shots, since U t ( θ, α ) =

Original: lim s →∞
Revised: lim s →∞

Original: U t ( θ, α, s ) in distribution by the law of large numbers.
Revised: U t ( θ, α, s ) in distribution by the law of large numbers.

Original: Let β ∗ be the optimal learning rate such that U t ( θ, β ∗ ) attains optimal performance, i.e. β ∗ =
Revised: Let β ∗ be the optimal learning rate such that U t ( θ, β ∗ ) attains optimal performance, i.e. β ∗ =

Original: arg min β E T t [ f t ( U t ( θ, β ))] .
Revised: arg min β E T t [ f t ( U t ( θ, β ))] .

Original: This can be interpreted as the optimal inner learning rate when s → ∞ .
Revised: This can be interpreted as the optimal inner learning rate when s → ∞ .

Original: We want to ﬁnd an optimal s -shot learning rate α ∗ s that imitates this behavior for a ﬁnite s .
Revised: We want to ﬁnd an optimal s -shot learning rate α ∗ s that imitates this behavior for a ﬁnite s .

Original: Speciﬁcally, we solve for the value of α ∗ s that minimizes the mean square error between the adapted parameters:α [Equation]
Revised: Speciﬁcally, we solve for the value of α ∗ s that minimizes the mean square error between the adapted parameters:

Original:  s = [Equation] || U t ( [Equation]
Revised: α ∗ s = arg min α E T t (cid:2) ||

Original: [Equation] [Equation] [Equation] U 
Revised: U t ( θ, α, s ) − U t

Original: [Equation] [Equation] [Equation]
Revised: ( θ, β ∗ )

Original: ) || 2 2 (cid:3) .
Revised: || 22 (cid:3) .

Original: Theorem 1.
Revised: Theorem 1.

Original: The optimal learning rate for the s -shot case can be calculated as α ∗ s 
Revised: The optimal learning rate for the s -shot case can be calculated as α ∗ s = (cid:18)

Original: [Equation]
Revised: [Equation]

Original: β ∗ where C=
Revised: β ∗ where C

Original: E T t [ Var( ∇ θ (cid:96) ( h ( x ; θ ) , y ))] and C= E T t (cid:2) ||
Revised: [Equation] Var

Original: E D t ∇ θ (cid:96) ( h ( x ; θ ) , y ) || 22 (cid:3) .
Revised: [Equation] [Equation] and C= [Equation]

Original: We provide the proof in Appendix B.
Revised: We provide the proof in Appendix B.

Original: The quantities C 1 , C 2 and β ∗ are all functions of the parameters θ , and thus require re-estimation each time we update θ during meta-training.
Revised: The quantities C 1 , C 2 and β ∗ are all functions of the parameters θ , and thus require re-estimation each time we update θ during meta-training.

Original: In practice, we denote β := β ∗ as the learned learning rate and η :=
Revised: In practice, we denote β := β ∗ as the learned learning rate and η :=

Original: C 2 C 1 as the learned scaling factor, and treat them as parameters that are meta-learned jointly with θ .
Revised: C 2 C 1 as the learned scaling factor, and treat them as parameters that are meta-learned jointly with θ .

Original: Denote α s ( β, η ) = (cid:16) 1 − 1 1+ ηs (cid:17) β as the learnable scaled learning rate with parameters β and η .
Revised: Denote α s ( β, η ) = (cid:16) 1 − [Equation] β as the learnable scaled learning rate with parameters β and η .

Original: In contrast to MAML-VL, which must meta-learn a number of learning rates that scales with the maximum number of shots, here we only meta-learn two additional scalar-valued parameters.
Revised: In contrast to MAML-VL, which must meta-learn a number of learning rates that scales with the maximum number of shots, here we only meta-learn two additional scalar-valued parameters.

Original: This leads to our proposed model-agnostic meta-learning with variable-shot and scaling (MAML-VS) algorithm, whose meta-training objective is given below:
Revised: This leads to our proposed model-agnostic meta-learning with variable-shot and scaling (MAML-VS) algorithm, whose meta-training objective is given below:

Original: [Equation]
Revised: [Equation]

Original:  O NLINE I NCREMENTAL M ETA -L EARNING WITH V ARIABLE -S HOT MAML
Revised:  O NLINE I NCREMENTAL M ETA -L EARNING WITH V ARIABLE -S HOT MAML

Original: With MAML-VS deﬁned in the section above, we are ready to present the algorithm for the incremental meta-learning setting.
Revised: With MAML-VS deﬁned in the section above, we are ready to present the algorithm for the incremental meta-learning setting.

Original: In following two subsections, we will discuss an online incremental meta-learning algorithm capable of variable-shot generalizations (Section 5.1) and its practical instantiation (Section 5.2).
Revised: In following two subsections, we will discuss an online incremental meta-learning algorithm capable of variable-shot generalizations (Section 5.1) and its practical instantiation (Section 5.2).

Original: 5. F OLLOW THE M ETA L EADER WITH V ARIABLE -S HOT AND S CALING
Revised: 5. F OLLOW THE M ETA L EADER WITH V ARIABLE -S HOT AND S CALING

Original: We extend FTML
Revised: We extend FTML

Original: (Finn et al., 2019) to the incremental meta-learning setting by enabling the meta-learner to handle variable-shot data with the scaling rule in MAML-VS.
Revised: (Finn et al., 2019) to the incremental meta-learning setting by enabling the meta-learner to handle variable-shot data with the scaling rule in MAML-VS.

Original: Speciﬁcally, in the incremental meta-learning setting, the model parameter θ and the scaled learning rate α s ( β, η ) deﬁned in Section 4.2 are updated in the following way:
Revised: Speciﬁcally, in the incremental meta-learning setting, the model parameter θ and the scaled learning rate α s ( β, η ) deﬁned in Section 4.2 are updated in the following way:

Original: [Equation]
Revised: [Equation]

Original: We term our incremental meta-learning algorithm FTML-VS.
Revised: We term our incremental meta-learning algorithm FTML-VS.

Original: Intuitively, for the current round t + 1 ,
Revised: Intuitively, for the current round t + 1 ,

Original: FTML-VS plays the best variable-shot meta-learner in hindsight after each round based on all the meta-learned models attained in previous rounds j = 1 . . .
Revised: FTML-VS plays the best variable-shot meta-learner in hindsight after each round based on all the meta-learned models attained in previous rounds j = 1 . . .

Original: t .
Revised: t .

Original: In practice, we of course cannot calculate the exact loss f j , and therefore must approximate it using stochastic gradient descent methods, which will be presented in the next subsection.
Revised: In practice, we of course cannot calculate the exact loss f j , and therefore must approximate it using stochastic gradient descent methods, which will be presented in the next subsection.

Original: 5. FTML-VS IN P RACTICE
Revised: 5. FTML-VS IN P RACTICE

Original: Optimizing Equation 2 is not feasible in practice as it has no closed form solution.
Revised: Optimizing Equation 2 is not feasible in practice as it has no closed form solution.

Original: We only have access to a portion of T t in round t , and thus the population risk f t is impossible to compute exactly, as discussed by Finn et al.
Revised: We only have access to a portion of T t in round t , and thus the population risk f t is impossible to compute exactly, as discussed by Finn et al.

Original: We adopt online stochastic gradient descent algorithms to resolve these issues.
Revised: We adopt online stochastic gradient descent algorithms to resolve these issues.

Original: At step s in round t , we compute gradients of our model parameters θ t and scale learning rate α K ( β, η ) where K is the number of shots sampled in a minibatch using the following procedure:
Revised: At step s in round t , we compute gradients of our model parameters θ t and scale learning rate α K ( β, η ) where K is the number of shots sampled in a minibatch using the following procedure:

Original: Draw a task T j : j ∼ κ ( t ) (or a minibatch of tasks) uniformly from the set of tasks the agent has seen so far {T i } ti =1 , where κ ( t ) is the uniform distribution on { 1 , . .
Revised: Draw a task T j : j ∼ κ ( t ) (or a minibatch of tasks) uniformly from the set of tasks the agent has seen so far {T i } ti =1 , where κ ( t ) is the uniform distribution on { 1 , . .

Original: . , t } .
Revised: . , t } .

Original: Let ˆ D j = ˆ D j ( s ) if j = s and otherwise ˆ D j = ˆ D j ( S j ) , which are all the data accumulated for task T j in the previous round.
Revised: Let ˆ D j = ˆ D j ( s ) if j = s and otherwise ˆ D j = ˆ D j ( S j ) , which are all the data accumulated for task T j in the previous round.

Original: Sample K ∼ ν ( j ) , where ν ( j ) is uniformly distributed on { 0 , . .
Revised: Sample K ∼ ν ( j ) , where ν ( j ) is uniformly distributed on { 0 , . .

Original: , M ( j ) } with M ( j )
Revised: , M ( j ) } with M ( j )

Original: =
Revised: =

Original: min { M, | ˆ D j |} . 3.
Revised: min { M, | ˆ D j |} . 3.

Original: Sample a minibatch D tr j ⊂ ˆ D t ( s ) of size K to compute the inner gradient updates using the derived scaled learning rate α K ( β, η ) .
Revised: Sample a minibatch D tr j ⊂ ˆ D t ( s ) of size K to compute the inner gradient updates using the derived scaled learning rate α K ( β, η ) .

Original: Sample another minibatch D val j ⊂ ˆ D t ( s ) to compute the gradients g θ t , g β , g η for updating the model parameter θ t along with parameters of scaled learning rate, β and η respectively.

Original: where we denote J =
Revised: where we denote J =

Original: E j ∼ κ ( t ) ,K ∼ ν ( j ) (cid:2) L ( D val j , U t
Revised: E j ∼ κ ( t ) ,K ∼ ν ( j ) (cid:2) L ( D val j , U t

Original: ( θ t , α K ( β, η ) , K )) (cid:3) , U t ( θ t , α, K )
Revised: ( θ t , α K ( β, η ) , K )) (cid:3) , U t ( θ t , α, K )

Original: = θ t − α K ( β, η ) ∇ θ t
Revised: = θ t − α K ( β, η ) ∇ θ t

Original: ( L ( D tr t , θ t )) and g θ = ∇ θ t J , g β = ∇ β J and g η = ∇ η J .
Revised: ( L ( D tr t , θ t )) and g θ = ∇ θ t J , g β = ∇ β J and g η = ∇ η J .

Original: Note that in the case of zero-shot adaptation, i.e. K = 0 , U t ( θ t , α K ( β, η ) , K ) corresponds to the parameter at the previous round, i.e. θ t − 1 .
Revised: Note that in the case of zero-shot adaptation, i.e. K = 0 , U t ( θ t , α K ( β, η ) , K ) corresponds to the parameter at the previous round, i.e. θ t − 1 .

Original: We outline the complete algorithm in Algorithm 1 in Appendix A with some implementation details in Appendix C. The algorithm learns θ , β , and η .
Revised: We outline the complete algorithm in Algorithm 1 in Appendix A with some implementation details in Appendix C. The algorithm learns θ , β , and η .

Original: We also include the training procedure and the evaluation protocol in Appendix A.
Revised: We also include the training procedure and the evaluation protocol in Appendix A.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Prior work in meta-learning has studied how to acquire learning rules (Schmidhuber, 1987; Bengio et al., 1992; Hochreiter et al., 2001), accelerate optimization (Andrychowicz et al., 2016), and acquire priors suitable for few-shot learning (Finn et al., 2017).
Revised: Prior work in meta-learning has studied how to acquire learning rules (Schmidhuber, 1987; Bengio et al., 1992; Hochreiter et al., 2001), accelerate optimization (Andrychowicz et al., 2016), and acquire priors suitable for few-shot learning (Finn et al., 2017).

Original: Modern meta-learning algorithms can broadly be classiﬁed into three high-level approaches: black-box meta-learners that parameterize a learning procedure using a neural network (Santoro et al., 2016; Ravi & Larochelle, 2017; Munkhdalai & Yu, 2017; Duan et al., 2016;
Revised: Modern meta-learning algorithms can broadly be classiﬁed into three high-level approaches: black-box meta-learners that parameterize a learning procedure using a neural network (Santoro et al., 2016; Ravi & Larochelle, 2017; Munkhdalai & Yu, 2017; Duan et al., 2016;

Original: Wang et al., 2016; Mishra et al., 2017), non-parametric meta-learners (Koch, 2015; Vinyals et al., 2016;
Revised: Wang et al., 2016; Mishra et al., 2017), non-parametric meta-learners (Koch, 2015; Vinyals et al., 2016;

Original: Snell et al., 2017), and optimization-based meta-learners that embed an optimization procedure into the meta-learner (Finn et al., 2017; Li et al., 2017; Rusu et al., 2018;
Revised: Snell et al., 2017), and optimization-based meta-learners that embed an optimization procedure into the meta-learner (Finn et al., 2017; Li et al., 2017b; Rusu et al., 2018;

Original: Zintgraf et al., 2018; Bertinetto et al., 2018; Rajeswaran et al., 2019; Lee et al., 2019; Yoon et al., 2018; Finn et al., 2018).
Revised: Zintgraf et al., 2018; Bertinetto et al., 2018; Rajeswaran et al., 2019; Lee et al., 2019; Yoon et al., 2018; Finn et al., 2018).

Original: We choose to build upon the latter class of optimization-based meta-learners, since they produce a well-formed optimization process that tends to be robust to out-of-distribution tasks, a useful characteristic in online meta-learning settings.
Revised: We choose to build upon the latter class of optimization-based meta-learners, since they produce a well-formed optimization process that tends to be robust to out-of-distribution tasks, a useful characteristic in online meta-learning settings.

Original: While some works have evaluated meta-learning algorithms for K -shot N -way classiﬁcation for a breadth of K and N (Snell et al., 2017; Triantaﬁllou et al., 2019; Allen et al., 2019b;a; Hsu et al., 2018; Cao et al., 2020), these works typically train different networks for each value of N and K , with some speciﬁcally noting poor generalization to values of K not used in training (Snell et al., 2017; Hsu et al., 2018).
Revised: While some works have evaluated meta-learning algorithms for K -shot N -way classiﬁcation for a breadth of K and N (Snell et al., 2017; Triantaﬁllou et al., 2019; Allen et al., 2019b;a; Hsu et al., 2018; Cao et al., 2020), these works typically train different networks for each value of N and K , with some speciﬁcally noting poor generalization to values of K not used in training (Snell et al., 2017; Hsu et al., 2018).

Original: Unlike these works, we consider the setting where a single meta-learned model must handle multiple different values of K .
Revised: Unlike these works, we consider the setting where a single meta-learned model must handle multiple different values of K .

Original: This setting has been considered in black-box methods that receive datapoints and make predictions incrementally (Santoro et al., 2016; Woodward & Finn, 2017), but this approach is known to underperform the setting where the model is trained for a speciﬁc value of K (Mishra et al., 2017).
Revised: This setting has been considered in black-box methods that receive datapoints and make predictions incrementally (Santoro et al., 2016; Woodward & Finn, 2017), but this approach is known to underperform the setting where the model is trained for a speciﬁc value of K (Mishra et al., 2017).

Original: Recent works have proposed non-parametric metalearners that work effectively with variable numbers of shots (Allen et al., 2019b; Cao et al., 2020).
Revised: Recent works have proposed non-parametric metalearners that work effectively with variable numbers of shots (Allen et al., 2019b; Cao et al., 2020).

Original: Non-parametric methods such as these prior works require at least one shot per class, whereas we consider a setting where there is even fewer datapoints available.
Revised: Non-parametric methods such as these prior works require at least one shot per class, whereas we consider a setting where there is even fewer datapoints available.

Original: This is possible in settings with non-mutually-exclusive tasks, such that the meta-learned model can provide a sufﬁciently strong prior about the class labels.
Revised: This is possible in settings with non-mutually-exclusive tasks, such that the meta-learned model can provide a sufﬁciently strong prior about the class labels.

Original: Our variable-shot meta-learning setting is speciﬁcally motivated by a problem setting that resembles online learning (Shalev-Shwartz, 2012) and continual learning (Thrun, 1998).
Revised: Our variable-shot meta-learning setting is speciﬁcally motivated by a problem setting that resembles online learning (Shalev-Shwartz, 2012) and continual learning (Thrun, 1998).

Original: It is well known that algorithms such as follow the leader (FTL) are computationally-expensive, and many works in online learning and continual learning focus on developing more computationally-efﬁcient methods (Kirkpatrick et al., 2017; Rebufﬁ et al., 2017;
Revised: It is well known that algorithms such as follow the leader (FTL) are computationally-expensive, and many works in online learning and continual learning focus on developing more computationally-efﬁcient methods (Kirkpatrick et al., 2017; Rebufﬁ et al., 2017;

Original: Lopez-Paz et al., 2017; French, 1999;  Zenke et al., 2017).
Revised: Lopez-Paz et al., 2017; French, 1999; Chaudhry et al., 2018; Zenke et al., 2017).

Original: Following multiple prior methods (Finn et al., 2019; Tessler et al., 2016; Rolnick et al., 2019), we instead focus on settings where it is practical to maintain a replay buffer of data, and focus on the problem of effective forward transfer when presented data from a sequence of tasks.
Revised: Following multiple prior methods (Finn et al., 2019; Tessler et al., 2016; Rolnick et al., 2019), we instead focus on settings where it is practical to maintain a replay buffer of data, and focus on the problem of effective forward transfer when presented data from a sequence of tasks.

Original: We do so by combining employing meta-learning in the online setting.
Revised: We do so by combining employing meta-learning in the online setting.

Original: Unlike prior online meta-learning works (Finn et al., 2019; Grant et al., 2019; Khodak et al., 2019; Zhuang et al., 2019), we develop an algorithm that is speciﬁcally tailored to leverage variable amounts of data, allowing it to outperform a prior state-of-the-art online meta-learning approach (Finn et al., 2019).
Revised: Unlike prior online meta-learning works (Finn et al., 2019; Grant et al., 2019; Khodak et al., 2019; Zhuang et al., 2019), we develop an algorithm that is speciﬁcally tailored to leverage variable amounts of data, allowing it to outperform a prior state-of-the-art online meta-learning approach (Finn et al., 2019).

Original: Finally, our work considers a problem setting that is distinct from works that a study online or continual learning in the inner loop of meta-learning (Al-Shedivat et al., 2017; Nagabandi et al., 2018; Javed & White, 2019; Denevi et al., 2019;
Revised: Finally, our work considers a problem setting that is distinct from works that a study online or continual learning in the inner loop of meta-learning (Al-Shedivat et al., 2017; Nagabandi et al., 2018; Javed & White, 2019; Denevi et al., 2019;

Original: He et al., 2019; Harrison et al., 2019).
Revised: He et al., 2019; Harrison et al., 2019).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our experiments aim to address the following questions, in both ofﬂine and online problem settings.
Revised: Our experiments aim to address the following questions, in both ofﬂine and online problem settings.

Original: In the more standard ofﬂine meta-learning setting, we aim to answer: (1) How does our variable-shot meta-learning method compare to prior methods, when evaluated with variable shots?
Revised: In the more standard ofﬂine meta-learning setting, we aim to answer: (1) How does our variable-shot meta-learning method compare to prior methods, when evaluated with variable shots in the ofﬂine setting?

Original: (2) Does our theoretically motivated learning rate rule match the performance of learning per-shot learning rates?
Revised: (2) Does our theoretically motivated learning rate rule match the performance of learning per-shot learning rates?

Original: [Table]

Original: We then integrate our variable-shot method into the online setting, to study: (3) Does variable-shot learning improve the regret in the incremental meta-learning setting?
Revised: We then integrate our variable-shot method into the online setting, to study: (3) Does variable-shot learning improve the regret in the incremental meta-learning setting?

Original: (4) Does our incremental meta-learning algorithm attain better cumulative regret than standard empirical risk minimization?
Revised: (4) Does our incremental meta-learning algorithm attain better cumulative regret than standard empirical risk minimization?

Original: Details regarding the architecture of the models and the training setup are presented in Appendix D.
Revised: Details regarding the architecture of the models and the training setup are presented in Appendix D.

Original: While our method can accommodate any standard meta-learning problem, we intentionally focus our evaluation on meta-learning problems with non-exclusive tasks, since this setting is particularly relevant in the online setting, which we will discuss in the subsection below and since this setting known to be especially difﬁcult (Yin et al., 2019).
Revised: While our method can accommodate any standard meta-learning problem, we intentionally focus our evaluation on meta-learning problems with non-exclusive tasks, since this setting is particularly relevant in the online setting, which we will discuss in the subsection below and since this setting known to be especially difﬁcult (Yin et al., 2019).

Original: With non-exclusive tasks, it is technically possible for the model to learn to solve all tasks in zero shot, which provides for a fair comparison to non-metalearning online methods, such as the “follow the learder” method that trains on all data seen so far (indicated as “TOE” in Section 7.2).
Revised: With non-exclusive tasks, it is technically possible for the model to learn to solve all tasks in zero shot, which provides for a fair comparison to non-metalearning online methods, such as the “follow the learder” method that trains on all data seen so far (indicated as “TOE” in Section 7.2).

Original: However, in order to attain the best possible regret, an effective meta-trained model must still learn to adapt, so as to solve new tasks quickly before it has learned to solve all tasks in zero shot.
Revised: However, in order to attain the best possible regret, an effective meta-trained model must still learn to adapt, so as to solve new tasks quickly before it has learned to solve all tasks in zero shot.

Original: Standard meta-learning methods often perform poorly in this regime, due to a memorization problem (Yin et al., 2019), where they learn to only solve the meta-training tasks in zero shot, failing to either solve or adapt to the new task.
Revised: Standard meta-learning methods often perform poorly in this regime, due to a memorization problem (Yin et al., 2019), where they learn to only solve the meta-training tasks in zero shot, failing to either solve or adapt to the new task.

Original: We use three non-mutually exclusive datasets for our ofﬂine and online meta-learning experiments respectively and one mutually-exclusive dataset for the ofﬂine setting only, which are listed as follows: (1) (Non-mutually exclusive)
Revised: We use three non-mutually exclusive datasets for our ofﬂine and online meta-learning experiments respectively and one mutually-exclusive dataset for the ofﬂine setting only, which are listed as follows: (1) (Non-mutually exclusive)

Original: Rainbow MNIST (Finn et al., 2019).
Revised: Rainbow MNIST (Finn et al., 2019).

Original: Based on the MNIST digit recognition dataset, this dataset consists of digits transformed in different ways: 7 backgrounds with different colors, 2 scales (half and original size), and 4 rotations of 90 degree intervals, which leads to 56 total tasks in combination.
Revised: Based on the MNIST digit recognition dataset, this dataset consists of digits transformed in different ways: 7 backgrounds with different colors, 2 scales (half and original size), and 4 rotations of 90 degree intervals, which leads to 56 total tasks in combination.

Original: Each task contains 900 images applied with one of the transformation listed above (See Appendix D for visualization); (2) (Non-mutually exclusive)
Revised: Each task contains 900 images applied with one of the transformation listed above (See Appendix D for visualization); (2) (Non-mutually exclusive)

Original: Contextual MiniImagenet.
Revised: Contextual MiniImagenet.

Original: This dataset, adapted from MiniImagenet (Ravi & Larochelle, 2016), is meant to provided a non-mutually-exclusive variant of the MiniImagenet few-shot recognition task, such that non-meta-learning baselines can feasibly solve this task.
Revised: This dataset, adapted from MiniImagenet (Ravi & Larochelle, 2016), is meant to provided a non-mutually-exclusive variant of the MiniImagenet few-shot recognition task, such that non-meta-learning baselines can feasibly solve this task.

Original: We use the same images and classes as MiniImagenet, but formulate each class as a separate binary classiﬁcation problem.
Revised: We use the same images and classes as MiniImagenet, but formulate each class as a separate binary classiﬁcation problem.

Original: The model receives two images as input: a reference image that determines the current task, and a query image, and the goal is to output a binary label indicating whether the query image belongs to the same class as the reference image, or not.
Revised: The model receives two images as input: a reference image that determines the current task, and a query image, and the goal is to output a binary label indicating whether the query image belongs to the same class as the reference image, or not.

Original: In this case, each class in the MiniImagenet constitutes a task, which makes the task space large and the problem challenging.
Revised: In this case, each class in the MiniImagenet constitutes a task, which makes the task space large and the problem challenging.

Original: We include a visualization of the architecture for solving this dataset in Figure 3 in Appendix D.3; (3) (Non-mutually exclusive) Pose
Revised: We include a visualization of the architecture for solving this dataset in Figure 3 in Appendix D.3; (3) (Non-mutually exclusive) Pose

Original: Prediction (Finn et al., 2019; Yin et al., 2019).
Revised: Prediction (Finn et al., 2019; Yin et al., 2019).

Original: In this setting, we construct a multi-task dataset with 65 tasks based on the Pascal 3D data (Xiang et al., 2014) where each task corresponds to a different object with a random camera angle.
Revised: In this setting, we construct a multi-task dataset with 65 tasks based on the Pascal 3D data (Xiang et al., 2014) where each task corresponds to a different object with a random camera angle.

Original: We render the image each object using MuJoCo (Todorov et al., 2012) with a random orientation.
Revised: We render the image each object using MuJoCo (Todorov et al., 2012) with a random orientation.

Original: We only use this dataset in the online setting; (4) (Mutually exclusive) Omniglot
Revised: We only use this dataset in the online setting; (4) (Mutually exclusive) Omniglot

Original: (Lake et al., 2011).
Revised: (Lake et al., 2011).

Original: This dataset has 20 instances of 1623 characters fromdifferent alphabets where each instance is drawn by a different person.
Revised: This dataset has 20 instances of 1623 characters fromdifferent alphabets where each instance is drawn by a different person.

Original: We only use this dataset in the ofﬂine setting and include the results in Appendix F.
Revised: We only use this dataset in the ofﬂine setting and include the results in Appendix F.

Original: 7. O FFLINE M ETA -L EARNING E XPERIMENTS
Revised: 7. O FFLINE M ETA -L EARNING E XPERIMENTS

Original: We ﬁrst evaluate our variable-shot method, MAML-VS, in the ofﬂine setting, in comparison to MAML (Finn et al., 2017) and MANN (Santoro et al., 2016), which uses a non-gradient based recurrent meta-learner.
Revised: We ﬁrst evaluate our variable-shot methods, MAML-VS and MAML-VL, in the ofﬂine setting, in comparison to MAML (Finn et al., 2017) and MANN (Santoro et al., 2016), which uses a nongradient based recurrent meta-learner.

Original: We also compare to MAML-VL in Table 3 in Appendix D. Experiment setup details, including descriptions of baselines, the dataset setup, and model architectures can be found in Appendix D. Note that as discussed in Section 6, non-parametric meta-learning methods require at least one shot for each class, making them inapplicable to non-mutually exclusive tasks where the total number of support datapoints is often much less than the total number of classes.
Revised: The ofﬂine experiments mainly serve as a sanity check of our online incremental experiments, where variable-shot learning is practically important. Experiment setup details, including descriptions of baselines, the dataset setup, and model architectures can be found in Appendix D. Note that as discussed in Section 6, non-parametric meta-learning methods require at least one shot for each class, making them inapplicable to non-mutually exclusive tasks where the total number of support datapoints is often much less than the total number of classes.

Original: Hence, 
Revised: Hence, we choose not to compare to those approaches. As shown in Table 1, our method MAML-VL achieves the best performance in 5 out of 8 total settings in both datasets while MAML-VS achieves the best performance in the 20-shot classiﬁcation accuracy on the Contextual MiniImagenet, which

Original: [Table]
Revised: [Table]

Original: we choose not to compare to those approaches.
Revised: Comparison of MAML, MANN, MAML-VL (ours) and MAML-VS (ours). The results are classiﬁcation accuracies averaged over 3 random seeds, ± one standard deviation. On Rainbow MNIST,

Original: As shown in Table 1, FTML-VS attains the best performance in 2 out of 4 settings in both datasets while achieving comparable results in the other cases.
Revised: MAML-VL (ours), that learns a learning rate per shot, does better in 0, 1, and 10-shot setting, which is the most effective in this relatively simple digit classiﬁcation task.

Original: In both experiments, training for variable shot with a single learning rate (MAML) causes the algorithm to sacriﬁce 0-shot and 1-shot performance in favor of better performance with more shots (10 and 20).
Revised: MAML-VS (ours) achieves comparable performance in all four settings in this experiment. On Contextual MiniImagenet, which is a challenging image classiﬁcation problem with large task space, MAML-VS (ours) achieves better performance in 20-shot classiﬁcation accuracies while maintaining competitive performances in 0, 1 and 10-shot settings. Overall, MAML-VS achieves the best or comparable performances in allsettings, which indicates that our theoretically-motivated inner learning rate scaling rule would excel in the online setting, which is empirically shown in Table [Table]

Revised: [Figure]

Revised: [Table]

Revised: is a challenging problem with large task space and attains comparable performance in all othersettings.

Revised: In both experiments, training for variable shot with a single learning rate (MAML) causes the algorithm to sacriﬁce 0-shot and 1-shot performance in favor of better performance with more shots (10 and 20).

Original: In the Contextual MiniImagenet setting especially, 1-shot performance is signiﬁcantly lower than even the 0-shot performance.
Revised: In the Contextual MiniImagenet setting especially, 1-shot performance is signiﬁcantly lower than even the 0-shot performance.

Original: For MANN, variable shot training instead biases the model toward good 0-shot and 1-shot performance, and sacriﬁces performance with more shots.
Revised: For MANN, variable shot training instead biases the model toward good 0-shot and 1-shot performance [Table] and sacriﬁces performance with more shots.

Original: We expect both issues to be problematic in the online setting: good few-shot performance is important in the later stages of training, where each task can be mastered with just a few samples, and good many-shot performance is critical for difﬁcult tasks or early on in training.
Revised: We expect both issues to be problematic in the online setting: good few-shot performance is important in the later stages of training, where each task can be mastered with just a few samples, and good many-shot performance is critical for difﬁcult tasks or early on in training.

Original: Our method, MAML-VS, performs well with all numbers of shots, and performance improves as more examples are observed.
Revised: Our method, MAML-VS and MAML-VL, performs well with all numbers of shots, and performance improves as more examples are observed.

Original: 7. O NLINE I NCREMENTAL M ETA -L EARNING E XPERIMENTS
Revised: 7. O NLINE I NCREMENTAL M ETA -L EARNING E XPERIMENTS

Original: To answer questions (3) and (4) , we evaluate FTML-VS in the online incremental setting.
Revised: To answer questions (3) and (4) , we evaluate FTML-VS in the online incremental setting.

Original: To evaluate the performance of our approach, we compare our FTML-VS to: (1) Train on everything (TOE)
Revised: To evaluate the performance of our approach, we compare our FTML-VS to: (1) Train on everything (TOE)

Original: (Finn et al., 2019) , which is the standard empirical risk minimization approach that trains a single predictive model on all the data available so far, i.e. { ˆ D i } ti =1 .
Revised: (Finn et al., 2019) , which is the standard empirical risk minimization approach that trains a single predictive model on all the data available so far, i.e. { ˆ D i } ti =1 .

Original: The trained model is directly tested on D test t without any adaptation; (2) FTML
Revised: The trained model is directly tested on D test t without any adaptation; (2) FTML

Original: (Finn et al., 2019) , which trains an online metalearner with ﬁxed inner learning rate and is evaluated on the adaptation performance on D test t ; (3)
Revised: (Finn et al., 2019) , which trains an online metalearner with ﬁxed inner learning rate and is evaluated on the adaptation performance on D test t ; (3)

Original: FTML-VL (Ours) , which trains an online meta-learner with a learned inner learning rate for each number of shots ranging from 0 to M .
Revised: FTML-VL (Ours) , which trains an online meta-learner with a learned inner learning rate for each number of shots ranging from 0 to M ; (4)FTML+Meta-SGD (Li et al., 2017a), which learns perparameter learning rates for FTML; (5) Incremental ProtoNet (Snell et al., 2017) , which adapts Snell et al.

Revised: (2017), one of the most widely used non-parametric meta-learning method to the online incremental meta-learning setting; (6) A-GEM (Chaudhry et al., 2018) , which is a popular continual learning algorithm that prevents catastropic forgetting.

Original: For Incremental Rainbow MNIST , we set the proﬁciency threshold C at 85% classiﬁcation accuracy computed on a minibatch of test data D test t ⊂ D t where D test t ∩ ˆ D t ( s ) = ∅ for each task t , and move on to the next task when this threshold is crossed or after 2000 steps.
Revised: For Incremental Rainbow MNIST , we set the proﬁciency threshold C at 85% classiﬁcation accuracy computed on a minibatch of test data D test t ⊂ D t where D test t ∩ ˆ D t ( s ) = ∅ for each task t , and move on to the next task when this threshold is crossed or after 2000 steps.

Original: For Incremental Contextual
Revised: For Incremental Contextual

Original: MiniImagenet , we set the proﬁciency threshold C at 75% classiﬁcation accuracy computed on D test t . 
Revised: MiniImagenet , we set the proﬁciency threshold C at 75% classiﬁcation accuracy computed on D test t . [Table]

Original:  For Incremental Pose Prediction , we consider mean-square error as the metric and do not advance the task automatically.
Revised: [Table] For Incremental Pose Prediction , we consider mean-square error as the metric and do not advance the task automatically.

Original: We discuss the details of the online procedure of each dataset in Appendix E.
Revised: We discuss the details of the online procedure of each dataset in Appendix E.

Original: We evaluate the performance based on the ﬁnal cumulative regret Regret T , which is computed by accumulating the test losses on D test t during evaluation time for each task t , as shown in Table 2 and include the regret curves over the incremental meta-learning process of all methods in all domains in
Revised: We evaluate the performance based on the ﬁnal cumulative regret Regret T , which is computed by accumulating the test losses on D test t during evaluation time for each task t , as shown in Table 2 and include the regret curves over the incremental meta-learning process of all methods in all domains in

Original: Appendix E. 
Revised: Appendix E. We also include additional ablation studies in Appendix G.

Original: According to Table 2, in all three domains, FTML-VS attains smallest regret after training all the tasks sequentially compared to other methods, suggesting the importance of using our learning rate rule in the incremental meta-learning setting with non-mutually exclusive datasets.
Revised: According to Table 2, in all three domains, FTML-VS attains smallest regret after training all the tasks sequentially compared to other methods, suggesting the importance of using our learning rate rule in the incremental meta-learning setting with non-mutually exclusive datasets.

Original: Moreover, FTML-VS outperforms TOE by a large margin, which indicates that our online meta-learning method is able to solve tasks more efﬁciently than empirical risk minimization by leveraging the shared task structure to quickly adapt to each new task. 
Revised: Moreover, FTML-VS outperforms TOE by a large margin, which indicates that our online meta-learning method is able to solve tasks more efﬁciently than empirical risk minimization by leveraging the shared task structure to quickly adapt to each new task. Finally, FTML-VS also achieves lower ﬁnal cumulative regret than FTML + Meta-SGD, suggesting that our theoretically sound rule of selecting learning rates for different number shots is more effective than simply learning learning rates for each parameter. It is also worth noting that learning per-parameter learning rates is complementary to our method and can be combined to further improve performances. We will leave this for future work.

Revised: Moreover, we conduct the comparison between FTML-VS and A-GEM on the Incremental Rainbow

Revised: MNIST dataset with the same online procedure described above.

Original: C ONCLUSION
Revised: As shown in Table 3, FTML-VS achieves much better cumulative regret compared to A-GEM. This result is unsurprising, as prior continual learning works focus primarily on minimizing negative backward transfer and compute considerations, as opposed to our goal of accelerating forward transfer through meta-learning. Finally, for the comparison to Incremental ProtoNet, we consider the Incremental Rainbow Mnist dataset. As described in Section 7.1, non-parametric meta-learning methods is applicable to settings where the number of shots is smaller than the number of classes.

Revised: Hence, in order to adapt Snell et al.

Revised: (2017), when we do not have the support datapoints for a particular class, i.e. a digit, of the current task, we compute the prototype of this class by sampling data of this class from previous tasks.

Revised: We compare FTML-VS to Incremental ProtoNet following such a data-generating protocol and the same online procedure as in the Incremental Rainbow MNIST experiment described in the paragraph above.

Revised: As shown in Table 4, FTML-VS outperforms Incremental ProtoNet by a signiﬁcant margin, suggesting that our theoretically motivated learning rate selection rule is pivotal in incremental meta-learning setting compared to non-parametric meta-learning methods, which require the minimum number of shots to be greater or equal to the number of classes.

Revised:  C ONCLUSION

Original: In this work, we studied meta-learning in the context of a sequence of tasks.
Revised: In this work, we studied meta-learning in the context of a sequence of tasks.

Original: While most meta-learning works study how to achieve few-shot adaptation, succeeding in sequential learning settings requires both meta-training and adaptation to be efﬁcient and performant.
Revised: While most meta-learning works study how to achieve few-shot adaptation, succeeding in sequential learning settings requires both meta-training and adaptation to be efﬁcient and performant.

Original: Unlike prior works, we focused on an online meta-learning setting where data for each new task arrives incrementally.
Revised: Unlike prior works, we focused on an online meta-learning setting where data for each new task arrives incrementally.

Original: Motivated by the challenges of this setting, we introduced a variable-shot meta-learning algorithm that optimizes for good performance after adapting with varying amounts of data.
Revised: Motivated by the challenges of this setting, we introduced a variable-shot meta-learning algorithm that optimizes for good performance after adapting with varying amounts of data.

Original: Our approach introduces a scaling rule for the learning rate that scales with the number of shots.
Revised: Our approach introduces a scaling rule for the learning rate that scales with the number of shots.

Original: This approach strongly outperformed a strictly more expressive approach of learning individual learning rates for each number of shots, validating the correctness of our derivation.
Revised: This approach strongly outperformed a strictly more expressive approach of learning individual learning rates for each number of shots, validating the correctness of our derivation.

Original: On both ofﬂine and online meta-learning settings, we observe signiﬁcant beneﬁts from our approach compared to prior methods.
Revised: On both ofﬂine and online meta-learning settings, we observe signiﬁcant beneﬁts from our approach compared to prior methods.

Original: A A LGORITHM O UTLINE
Revised: A A LGORITHM O UTLINE

Original: D tr j with K datapoints and D val j uniformly from ˆ D j 7: Computegradient g θt , g t using D tr j D val j , and Procedure8: Update parameters θ ← θ γ g θt // (or use Adam) 9:
Revised: Compute

Original: Updateβ ← α − γ g α K t ·∇β α K and η ← η − γ g α K t ·∇η α K 10: end for 11:
Revised: Update ·∇ ·∇ 10: end for 11:

Original: Return θ, α, η 12: end function 13: function VS-Update-Procedure ( θ , β , η , D ) 14: if |D| > M then 15:
Revised: Return θ α, η 12: end function 13: function VS-Update-Procedure ( θ , β , η , D ) 14: if D M then 15:

Original: Sample a minibatch ˆ D ∼ D with | ˆ D| = M and set D ←ˆ D 16: end if 17: for n g =1 , . . .
Revised: Sample a minibatch ˆ D ∼ D with | ˆ D| = M and set D ← D 16: end if 17: for n g =

Original: , N grad steps do 18: θ ←β · (cid:16)
Revised: N grad steps do 18: θ ←

Original: ) 19: end for 20:
Revised: 19: end for 20:

Original: Return θ 21: end function
Revised: Return θ 21: end function

Original: For each task t , we discuss the training procedure and evaluation protocol below.
Revised: For each task t , we discuss the training procedure and evaluation protocol below.

Original: At training time , we maintain a dataset ˆ D t ( s ) that receives one datapoint incrementally at each step s of task T t , i.e. | ˆ D t ( s ) |
Revised: At training time , we maintain a dataset ˆ D t ( s ) that receives one datapoint incrementally at each step s of task T t , i.e. | ˆ D t ( s ) |

Original: = s .
Revised: = s .

Original: After appending data to ˆ D t ( s ) , the model parameters θ , learned learning rate β , and the learned scaling factor η are updated as per Procedure 4 following the procedure discussed above and in VS-Meta-Update in Algorithm 2 in Appendix A.
Revised: After appending data to ˆ D t ( s ) , the model parameters θ , learned learning rate β , and the learned scaling factor η are updated as per Procedure 4 following the procedure discussed above and in VS-Meta-Update in Algorithm 2 in Appendix A.

Original: During evaluation , we sample a minibatch of test data D test t ⊂ D t where D test t ∩ ˆ D t ( s ) = ∅ .
Revised: During evaluation , we sample a minibatch of test data D test t ⊂ D t where D test t ∩ ˆ D t ( s ) = ∅ .

Original: We compute the adapted parameter θ (cid:48) t using VS-Meta-Update , using either all the data for task t so far, or a maximum of M datapoints if more than M datapoints have been collected.
Revised: We compute the adapted parameter θ (cid:48) t using VS-Meta-Update , using either all the data for task t so far, or a maximum of M datapoints if more than M datapoints have been collected.

Original: Then, we compute the loss on the test set D test t using θ (cid:48) .
Revised: Then, we compute the loss on the test set D test t using θ (cid:48) .

Original: Hence, the incremental meta-learner is evaluated based on the performance of adaptation to unseen data with an incremental number of shots ranging fromto M .
Revised: Hence, the incremental meta-learner is evaluated based on the performance of adaptation to unseen data with an incremental number of shots ranging fromto M .

Original: If the loss on the test set reaches the proﬁciency threshold C (e.g., 90% accuracy), we advance to the next task and record the data efﬁciency of T t .
Revised: If the loss on the test set reaches the proﬁciency threshold C (e.g., 90% accuracy), we advance to the next task and record the data efﬁciency of T t .

Original: The optimal scaled learning rate for each number of shots is computed as α K ( β, η ) =
Revised: The optimal scaled learning rate for each number of shots is computed as α K ( β, η ) =

Original: β (cid:16) 1 − 1 1+ ηK (cid:17) .
Revised: β (cid:16) 1 − 1 1+ ηK (cid:17) .

Original: We keep a buffer B , initialized as empty, to store the task data seen so far.
Revised: We keep a buffer B , initialized as empty, to store the task data seen so far.

Original: The cumulative regret
Revised: The cumulative regret

Original: Regret T is computed by accumulating the test losses during evaluation time for each task t .
Revised: Regret T is computed by accumulating the test losses during evaluation time for each task t .

Original: If the model learns to advance to the next task with improving data efﬁciency, the cumulative regret would tend to level off over number of tasks, which suggests that we achieve adaptation from many-shot to few-shot over the course of the incremental meta-learning.
Revised: If the model learns to advance to the next task with improving data efﬁciency, the cumulative regret would tend to level off over number of tasks, which suggests that we achieve adaptation from many-shot to few-shot over the course of the incremental meta-learning.

Original: B P ROOF OF T HEOREM
Revised: B P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: We apply variance and bias decomposition for the mean squared error to obtain:
Revised: We apply variance and bias decomposition for the mean squared error to obtain:

Original: [Equation]
Revised: [Equation]

Original: Minimizing this quadratic function with respect to α , we get α
Revised: Minimizing this quadratic function with respect to α , we get α

Original: [Equation]
Revised: [Equation]

Original: = (cid:32)
Revised: (cid:32)

Original: [Equation]
Revised: [Equation]

Original: β
Revised: β

Original: [Equation]
Revised: [Equation]

Original: with the C 1 and C 2 as deﬁned in Theorem 1.
Revised: with the C 1 and C 2 as deﬁned in Theorem 1.

Original: C I MPLEMENTATION D ETAILS FOR FTML-VS
Revised: C I MPLEMENTATION D ETAILS FOR FTML-VS

Original: In practice, we ﬁnd that using multiple inner gradient updates in U t ( θ, α K ( β, η ) , K ) improves the performance of the method.
Revised: In practice, we ﬁnd that using multiple inner gradient updates in U t ( θ, α K ( β, η ) , K ) improves the performance of the method.

Original: Moreover, it is also helpful to add meta-regularization (Yin et al., 2019) to improve the performance on few-shot adaptation in our variable-shot learning and incremental meta-learning settings.
Revised: Moreover, it is also helpful to add meta-regularization (Yin et al., 2019) to improve the performance on few-shot adaptation in our variable-shot learning and incremental meta-learning settings.

Original: Speciﬁcally, we model the parameters with stochasticity and at each round t , θ t ∼ q ( θ ; θ µ , θ σ ) is drawn from a distribution q with mean θ µ and variance θ σ .
Revised: Speciﬁcally, we model the parameters with stochasticity and at each round t , θ t ∼ q ( θ ; θ µ , θ σ ) is drawn from a distribution q with mean θ µ and variance θ σ .

Original: We typically use q ( θ ; θ µ , θ σ ) as a normal distribution.
Revised: We typically use q ( θ ; θ µ , θ σ ) as a normal distribution.

Original: Following Yin et al.
Revised: Following Yin et al.

Original: (2019), we regularize the meta-objective by adding a D KL ( q ( θ ; θ µ , θ σ ) || r ( θ )) term to Procedure 4, where r ( θ ) is a prior distribution on the model parameters (e.g., a standard normal distribution).
Revised: (2019), we regularize the meta-objective by adding a D KL ( q ( θ ; θ µ , θ σ ) || r ( θ )) term to Procedure 4, where r ( θ ) is a prior distribution on the model parameters (e.g., a standard normal distribution).

Original: D D ETAILED I NFORMATION FOR O FFLINE E XPERIMENT S ETUP 
Revised: D D ETAILED I NFORMATION FOR O FFLINE E XPERIMENT S ETUP D. V ISUALIZATION OF D ATASETS

Original: We present the complete results on Ofﬂine Rainbow MNIST and Ofﬂine Contextual MiniImageNet in Table 3.

Original: D. V ISUALIZATION OF D ATASETS

Original: We provide visualizations of images in Rainbow MNIST and Pose Prediction in Figure 2.
Revised: We provide visualizations of images in Rainbow MNIST and Pose Prediction in Figure 2.

Original: For visualizations of Contextual MiniImagenet, see Figure3.
Revised: For visualizations of Contextual MiniImagenet, see Figure

Original: [Equation] D ATASET
Revised: . D. R AINBOW MNIST D.2. D ATASET

Original: We use the same dataset as the Incremental Rainbow MNIST dataset in (Finn et al., 2019).
Revised: We use the same dataset as the Incremental Rainbow MNIST dataset in (Finn et al., 2019).

Original: Due to the relatively low zero-shot difﬁculty of the dataset, we only take the ﬁrst 20 tasks as the training set.
Revised: Due to the relatively low zero-shot difﬁculty of the dataset, we only take the ﬁrst 20 tasks as the training set.

Original: D.2. M ODEL A RCHITECTURES
Revised: D.2. M ODEL A RCHITECTURES

Original: We use the same neural network architecture for MAML and MAML-VS.
Revised: We use the same neural network architecture for MAML and MAML-VS.

Original: The network consists of3x3 kernel 32 channel convolution layers with leaky ReLU activation, and a ﬁnal fully connected layer to output logits for 10 classes.
Revised: The network consists of3x3 kernel 32 channel convolution layers with leaky ReLU activation, and a ﬁnal fully connected layer to output logits for 10 classes.

Original: For MANN, we ﬁrst have an embedding layer that converts the 10-class adaptation labels to 256 dimensional vectors.
Revised: For MANN, we ﬁrst have an embedding layer that converts the 10-class adaptation labels to 256 dimensional vectors.

Original: We pass the adaptation image through a feature network consists of 4 3x3 kernel 32 channel convolution layers, followed by a fully connected layer that output 256 dimensional image features.
Revised: We pass the adaptation image through a feature network consists of 4 3x3 kernel 32 channel convolution layers, followed by a fully connected layer that output 256 dimensional image features.

Original: We then concatenate these image features with the label embeddings, and feed it into a GRU layer.
Revised: We then concatenate these image features with the label embeddings, and feed it into a GRU layer.

Original: The GRU layer outputs a 256 dimensional summary for the adaptation data.
Revised: The GRU layer outputs a 256 dimensional summary for the adaptation data.

Original: We then pass the test images into the same image feature network to obtain a 256 dimensional test image features.
Revised: We then pass the test images into the same image feature network to obtain a 256 dimensional test image features.

Original: We concatentate the GRU layer’s output with the test image features, and feed them into a ﬁnal fully connected layer to obtain the 10 logits.
Revised: We concatentate the GRU layer’s output with the test image features, and feed them into a ﬁnal fully connected layer to obtain the 10 logits.

Original: D. C ONTEXTUAL M INI I MAGE N ET
Revised: D. C ONTEXTUAL M INI I MAGE N ET

Original: We present a visualization of the setup in Figure3.
Revised: We present a visualization of the setup in Figure

Original:  D.3. D ATASET
Revised: . D.3. D ATASET

Original: We use the same dataset as the MiniImageNet dataset in (Finn et al., 2017; Snell et al., 2017).
Revised: We use the same dataset as the MiniImageNet dataset in (Finn et al., 2017; Snell et al., 2017).

Original: In order to make the tasks non-mutually-exclusive, we cast it into a binary classiﬁcation problem where the model receives an ordered pair of two images and classiﬁes whether they belong to the same class.
Revised: In order to make the tasks non-mutually-exclusive, we cast it into a binary classiﬁcation problem where the model receives an ordered pair of two images and classiﬁes whether they belong to the same class.

Original: Different tasks in this dataset then correspond to the category of the ﬁrst image in the pair.
Revised: Different tasks in this dataset then correspond to the category of the ﬁrst image in the pair.

Original: Each time we sample a batch, we sample half batch of pairs with images from the same class and half batch of pairs with images from different tasks.
Revised: Each time we sample a batch, we sample half batch of pairs with images from the same class and half batch of pairs with images from different tasks.

Original: We use the ﬁrst 90 classes as our training set.
Revised: We use the ﬁrst 90 classes as our training set.

Original: D.3. M ODEL A RCHITECTURES
Revised: D.3. M ODEL A RCHITECTURES

Original: We use the same neural network architecture for MAML and MAML-VS.
Revised: We use the same neural network architecture for MAML and MAML-VS.

Original: The network consists of3x3 kernel 64 channel convolution layers with leaky ReLU activation, followed by a 2x2 max pooling layer, followed again by 2 3x3 kernel 64 channel convolution layers with leaky ReLU activation, followed by a 2x2 max pooling layer and a ﬁnal fully connected layer that ouputs 2 logits.
Revised: The network consists of3x3 kernel 64 channel convolution layers with leaky ReLU activation, followed by a 2x2 max pooling layer, followed again by 2 3x3 kernel 64 channel convolution layers with leaky ReLU activation, followed by a 2x2 max pooling layer and a ﬁnal fully connected layer that ouputs 2 logits.

Original: For MANN, the model is the same as we used in Rainbow MNIST experiments, except we use a different image feature network consists of 2 3x3 kernel 64 channel convolution layers with leaky ReLU activation, followed by a 2x2 max pooling layer, followed again by 2 3x3 kernel 64 channel convolution layers [Table] Comparison of MAML, MANN, MAML-VL (ours) and MAML-VS (ours). The results are classiﬁcation accuracies averaged over 3 random seeds, ± one standard deviation.
Revised: For MANN, the model is the same as we used in Rainbow MNIST experiments, except we use a different image feature network consists of 2 3x3 kernel 64 channel convolution layers with leaky ReLU activation, followed by a 2x2 max pooling layer, followed again by 2 3x3 kernel 64 channel convolution layers with leaky ReLU activation, followed by a 2x2 max pooling layer and a ﬁnal fully connected layer that ouputs 256 dimensional image embeddings.

Original: On Rainbow MNIST, MAML-VL (ours), that learns a learning rate per shot, does better in 0, 1, and 10-shot setting, which is the most effective in this relatively simple digit classiﬁcation task. MAML-VS (ours) achieves comparable performance in all four settings in this experiment. On Contextual MiniImagenet, which is a challenging image classiﬁcation problem with large task space, MAML-VS (ours) achieves better performance in 20-shot classiﬁcation accuracies while maintaining competitive performances in 0, 1 and 10-shot settings.
Revised: [Figure]

Original: Overall, MAML-VS achieves the best or comparable performances in allsettings, which indicates that our theoretically-motivated inner learning rate scaling rule would excel in the online setting, which is empirically shown in Table 2.

Original: [Figure]
Revised: [Figure]

Original: with leaky ReLU activation, followed by a 2x2 max pooling layer and a ﬁnal fully connected layer that ouputs 256 dimensional image embeddings.

Original: D. T RAINING FOR V ARIABLE N UMBER OF S HOTS
Revised: D. T RAINING FOR V ARIABLE N UMBER OF S HOTS

Original: During training time, we train the model with variable number of shots by averaging the loss for different shots.
Revised: During training time, we train the model with variable number of shots by averaging the loss for different shots.

Original: For zero-shot, we directly use the prior parameter without adaptation.
Revised: For zero-shot, we directly use the prior parameter without adaptation.

Original: E D ETAILED I NFORMATION FOR O NLINE I NCREMENTAL E XPERIMENT S ETUP E. I NCREMENTAL R AINBOW MNIST
Revised: E D ETAILED I NFORMATION FOR O NLINE I NCREMENTAL E XPERIMENT S ETUP E. I NCREMENTAL R AINBOW MNIST

Original: During training, we randomly sample a permutation of the 56 classes in Rainbow MNIST, where each class constitutes a task.
Revised: During training, we randomly sample a permutation of the 56 classes in Rainbow MNIST, where each class constitutes a task.

Original: For each task t = 1 , . . . , 56 , we add 4 datapoints to ˆ D t ( s ) every 10 steps.
Revised: For each task t = 1 , . . . , 56 , we add 4 datapoints to ˆ D t ( s ) every 10 steps.

Original: We set the maximum number of shots to be evaluated at M = 20 .
Revised: We set the maximum number of shots to be evaluated at M = 20 .

Original: When | ˆ D t ( s ) | ∈ { 0 , 4 , 8 , . .
Revised: When | ˆ D t ( s ) | ∈ { 0 , 4 , 8 , . .

Original: . , 40 } , we evaluate | ˆ D t ( s ) |-shot performance, using one half of the task dataset for adaptation (i.e., D train t ) and the other half for evaluation (i.e., D val t ).
Revised: . , 40 } , we evaluate | ˆ D t ( s ) |-shot performance, using one half of the task dataset for adaptation (i.e., D train t ) and the other half for evaluation (i.e., D val t ).

Original: Note that TOE does not perform few-shot adaptation, but does use the available data for the [Figure] [Figure]
Revised: Note that TOE does not perform few-shot adaptation, but does use the available data for the new task for standard supervised training.

Original: For meta-training, we sample 25 tasks from the buffer B at each step, and randomly sample the number of shots K ∼ Unif { 0 , 1 , 2 , . . .
Revised: For meta-training, we sample 25 tasks from the buffer B at each step, and randomly sample the number of shots K ∼ Unif { 0 , 1 , 2 , . . .

Original: We use the same architecture of the model as in the ofﬂine setting.
Revised: We use the same architecture of the model as in the ofﬂine setting.

Original: To train FTML, FTML-VL, FTML-VS, we take 5 inner gradient steps with inner gradient initialized at 0 .
Revised: To train FTML, FTML-VL, FTML-VS, we take 5 inner gradient steps with inner gradient initialized at 0 .

Original: Inner gradient magnitudes are clipped within 10 .
Revised: Inner gradient magnitudes are clipped within 10 .

Original: We train all methods  using Adam with learning rate 0 . 0001 with gradient clipping 10 .
Revised: We train all methods [Figure] using Adam with learning rate 0 . 0001 with gradient clipping 10 .

Original: As discussed in Section 5.2, we use meta-regularization (Yin et al., 2019) with coefﬁcient 0 .
Revised: As discussed in Section 5.2, we use meta-regularization (Yin et al., 2019) with coefﬁcient 0 .

Original: For FTML-VS, we initialize η = 1 . 0 .
Revised: For FTML-VS, we initialize η = 1 . 0 .

Original: We run all methods with 3 random seeds with and without the automatic advancement to the next task based on proﬁciency on the current task discussed in Section 5.2 and include the results in Figure 4.
Revised: We run all methods with 3 random seeds with and without the automatic advancement to the next task based on proﬁciency on the current task discussed in Section 5.2 and include the results in Figure 4.

Original: We add 2 datapoints to ˆ D t ( s ) every 5 and 2 steps for with and without automatic advancement experiments respectively.
Revised: We add 2 datapoints to ˆ D t ( s ) every 5 and 2 steps for with and without automatic advancement experiments respectively.

Original: We train maximum 2000 steps for each task in the setting where automatic advancement happens.
Revised: We train maximum 2000 steps for each task in the setting where automatic advancement happens.

Original: In this setting, we advance the next task once the model achieves over 85% success rate on the current task.
Revised: In this setting, we advance the next task once the model achieves over 85% success rate on the current task.

Original: In the case where we don’t advance the task based on the current task performance, we train each task for ﬁxed 800 steps.
Revised: In the case where we don’t advance the task based on the current task performance, we train each task for ﬁxed 800 steps.

Original: As shown in Figure 4, FTML-VS outperforms the baselines in both experimental settings, where the advantage of FTML-VS over other baselines is more apparent in the setting where automatic advancement doesn’t occur.
Revised: As shown in Figure 4, FTML-VS outperforms the baselines in both experimental settings, where the advantage of FTML-VS over other baselines is more apparent in the setting where automatic advancement doesn’t occur.

Original: E. 
Revised: E. I NCREMENTAL C ONTEXTUAL M INI I MAGENET

Original: I [Figure] C ONTEXTUAL M INI I MAGENET

Original: As before, we generate a randomly shufﬂed sequence of 100 classes from MiniImagenet.
Revised: As before, we generate a randomly shufﬂed sequence of 100 classes from MiniImagenet.

Original: We pretrain all methods on the ﬁrst 40 tasks and then feed the next 60 tasks incrementally.
Revised: We pretrain all methods on the ﬁrst 40 tasks and then feed the next 60 tasks incrementally.

Original: We add 8 datapoints to the dataset of the current task every 20 steps, for a maximum of 1400 steps per task.
Revised: We add 8 datapoints to the dataset of the current task every 20 steps, for a maximum of 1400 steps per task.

Original: We set the maximum number of shots to be M = 20 as before.
Revised: We set the maximum number of shots to be M = 20 as before.

Original: We set the proﬁciency threshold C at 75% classiﬁcation accuracy.
Revised: We set the proﬁciency threshold C at 75% classiﬁcation accuracy.

Original: When | ˆ D t ( s ) | ∈ { 0 , 8 , 16 , . .
Revised: When | ˆ D t ( s ) | ∈ { 0 , 8 , 16 , . .

Original: . , 80 } , we evaluate | ˆ D t ( s ) |-shot performance of all the methods except TOE, for the same reason as before.
Revised: . , 80 } , we evaluate | ˆ D t ( s ) |-shot performance of all the methods except TOE, for the same reason as before.

Original: During training, we sample 16 tasks from B and randomly sample K ∼ Unif { 0 , 2 , 4 , . . .
Revised: During training, we sample 16 tasks from B and randomly sample K ∼ Unif { 0 , 2 , 4 , . . .

Original: We also use the same architecture of the model as in the ofﬂine setting.
Revised: We also use the same architecture of the model as in the ofﬂine setting.

Original: Similar to Incremental Rainbow
Revised: Similar to Incremental Rainbow

Original: MNIST, we take 5 inner gradient steps with inner gradient initialized at 0 .
Revised: MNIST, we take 5 inner gradient steps with inner gradient initialized at 0 .

Original: We train all methods using Adam with learning rate 0 . 0001 with gradient clipping 10 .
Revised: We train all methods using Adam with learning rate 0 . 0001 with gradient clipping 10 .

Original: We also use meta-regularization (Yin et al., 2019) with coefﬁcient 0 .
Revised: We also use meta-regularization (Yin et al., 2019) with coefﬁcient 0 .

Original: For FTML-VS, we initialize η = 1 . 0 .
Revised: For FTML-VS, we initialize η = 1 . 0 .

Original: Similar to evalutions of Incremental Rainbow MNIST, We also include the results of 3 random seeds on settings with and without automatic advancement in Figure 5.
Revised: Similar to evalutions of Incremental Rainbow MNIST, We also include the results of 3 random seeds on settings with and without automatic advancement in Figure 5.

Original: We add 8 datapoints to ˆ D t ( s ) every 20 and 10 steps for with and without automatic advancement experiments respectively.
Revised: We add 8 datapoints to ˆ D t ( s ) every 20 and 10 steps for with and without automatic advancement experiments respectively.

Original: For automatic advancement, we train each method with maximum 1400 steps and pick the proﬁciency threshold at 75% .
Revised: For automatic advancement, we train each method with maximum 1400 steps and pick the proﬁciency threshold at 75% .

Original: In the setting without automatic advancement, we train each task for ﬁxed 700 steps.
Revised: In the setting without automatic advancement, we train each task for ﬁxed 700 steps.

Original: As shown in Figure 5e-5h, FTML-VS outperforms the FTML but gets slightly higher overall regret compared to FTML-VL where automatic advancement doesn’t happen whereas in Figure 5a-5d, FTML-VS achieves the minimal regret among all methods over the course of training.
Revised: As shown in Figure 5e-5h, FTML-VS outperforms the FTML but gets slightly higher overall regret compared to FTML-VL where automatic advancement doesn’t happen whereas in Figure 5a-5d, FTML-VS achieves the minimal regret among all methods over the course of training.

Original: E. I NCREMENTAL P OSE P REDICTION
Revised: E. I NCREMENTAL P OSE P REDICTION

Original: Similarly, we construct a random sequence of the 65 tasks in the 3D pose dataset.
Revised: Similarly, we construct a random sequence of the 65 tasks in the 3D pose dataset.

Original: We add 4 datapoints to the dataset of the current task every 10 steps, for a maximum of 240 steps per task.
Revised: We add 4 datapoints to the dataset of the current task every 10 steps, for a maximum of 240 steps per task.

Original: We set the maximum number of shots to be M = 15 .
Revised: We set the maximum number of shots to be M = 15 .

Original: When | ˆ D t ( s ) | ∈ { 0 , 4 , 8 , . .
Revised: When | ˆ D t ( s ) | ∈ { 0 , 4 , 8 , . .

Original: , 96 } , we evaluate | ˆ D t ( s ) |- shot performance as in Incremental Rainbow MNIST.
Revised: , 96 } , we evaluate | ˆ D t ( s ) |- shot performance as in Incremental Rainbow MNIST.

Original: In this setting, we consider mean-square error and does not advance the task automatically.
Revised: In this setting, we consider mean-square error and does not advance the task automatically.

Original: We set the proﬁciency threshold C at 85% classiﬁcation accuracy, and move on to the next task when this threshold is crossed or after 2000 steps.
Revised: We set the proﬁciency threshold C at 85% classiﬁcation accuracy, and move on to the next task when this threshold is crossed or after 2000 steps.

Original: As before, we present the cumulative regret curves of 3 random seeds on the Incremental Pose
Revised: As before, we present the cumulative regret curves of 3 random seeds on the Incremental Pose

Original: Prediction dataset.
Revised: Prediction dataset.

Original: As shown in Figure 6b -6d, FTML-VS attains the best regret among all approaches.
Revised: As shown in Figure 6b -6d, FTML-VS attains the best regret among all approaches.

Original: Note that TOE outperforms vanilla FTML in this dataset despite that FTML-VS achieves better cumulative regret over TOE, suggesting that empirical risk minimization works better than vanilla online meta-learning in this case while our method can further improve the performance of vanilla online meta-learning approaches and surpass the level of empirical risk minimization.
Revised: Note that TOE outperforms vanilla FTML in this dataset despite that FTML-VS achieves better cumulative regret over TOE, suggesting that empirical risk minimization works better than vanilla online meta-learning in this case while our method can further improve the performance of vanilla online meta-learning approaches and surpass the level of empirical risk minimization.

Original: F O FFLINE E XPERIMENTS ON M UTUALLY E XCLUSIVE T ASKS
Revised: F O FFLINE E XPERIMENTS ON M UTUALLY E XCLUSIVE T ASKS

Original: We compare out method against MAML in the variable-shot settings on the Omniglot dataset, which has mutually exclusive tasks.
Revised: We compare out method against MAML in the variable-shot settings on the Omniglot dataset, which has mutually exclusive tasks.

Original: We test 50-way classiﬁcation on 1, 2 and 5 shots in order to make the task more difﬁcult.
Revised: We test 50-way classiﬁcation on 1, 2 and 5 shots in order to make the task more difﬁcult.

Original: The results are shown in Table 4.
Revised: The results are shown in Table 5.

Original: We can see that our method performs about the same as our baseline method MAML.
Revised: We can see that our method performs about the same as our baseline method MAML.

Original: This is because in the mutually exclusive settings, the number of datapoints of K -shot classiﬁcation for computing inner gradient adaptation is K times the number of classes, compared to merely K in the mutually non-exclusive setting.
Revised: This is because in the mutually exclusive settings, the number of datapoints of K -shot classiﬁcation for computing inner gradient adaptation is K times the number of classes, compared to merely K in the mutually non-exclusive setting.

Original: For example, in this case, 5-shot corresponds to 250 images for computing inner gradient updates.
Revised: For example, in this case, 5-shot corresponds to 250 images for computing inner gradient updates.

Original: This is a fairly large number of images so that the gradient might have little variance, and therefore a large gradient step could be taken without overﬁtting.
Revised: This is a fairly large number of images so that the gradient might have little variance, and therefore a large gradient step could be taken without overﬁtting [Table]

Revised: G A BLATION STUDIES OF O NLINE I NCREMENTAL E XPERIMENTS

Original: [Table]
Revised: [Table]

Revised: We conduct an ablation study on the sensitivity of M , the maximum number of shots, on the incremental Rainbow MNIST dataset by comparing FTML and FTML-VS.

Revised: As shown in Table 6, as

Revised: M increases, the cumulative regret of both FTML and FTML-VS decreases since learning becomes easier with larger numbers of shots.

Revised: Meanwhile, FTML-VS attains better performance compared to

Revised: FTML with different values of M and the performance gap becomes larger as M decreases, suggesting that our theoretically motivated learning rate scaling rule based on the number of shots is important for different values of the maximum number of shots and is particularly effective when the maximum number of shots is small.