Original: T HE E FFECT OF D IVERSITY IN M ETA -L EARNING
Revised: T HE E FFECT OF D IVERSITY IN M ETA -L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Few-shot learning aims to learn representations that can tackle novel tasks given a small number of examples.
Revised: Few-shot learning aims to learn representations that can tackle novel tasks given a small number of examples.

Original: Recent studies show that task distribution plays a vital role in the performance of the model.
Revised: Recent studies show that task distribution plays a vital role in the performance of the model.

Original: Conventional wisdom is that task diversity should improve the performance of meta-learning.
Revised: Conventional wisdom is that task diversity should improve the performance of meta-learning.

Original: In this work, we ﬁnd evidence to the contrary; we study different task distributions on a myriad of models and datasets to evaluate the effect of task diversity on meta-learning algorithms.
Revised: In this work, we ﬁnd evidence to the contrary; we study different task distributions on a myriad of models and datasets to evaluate the effect of task diversity on meta-learning algorithms.

Original: For this experiment, we train on multiple datasets, and with three broad classes of meta-learning models - Metric-based (i.e., Protonet, Matching
Revised: For this experiment, we train on multiple datasets, and with three broad classes of meta-learning models - Metric-based (i.e., Protonet, Matching

Original: Networks), Optimization-based (i.e., MAML, Reptile, and MetaOptNet), and
Revised: Networks), Optimization-based (i.e., MAML, Reptile, and MetaOptNet), and

Original: Bayesian meta-learning models (i.e., CNAPs).
Revised: Bayesian meta-learning models (i.e., CNAPs).

Original: Our experiments demonstrate that the effect of task diversity on all these algorithms follows a similar trend, and task diversity does not seem to offer any beneﬁts to the learning of the model.
Revised: Our experiments demonstrate that the effect of task diversity on all these algorithms follows a similar trend, and task diversity does not seem to offer any beneﬁts to the learning of the model.

Original: Furthermore, we also demonstrate that even a handful of tasks, repeated over multiple batches, would be sufﬁcient to achieve a performance similar to uniform sampling and draws into question the need for additional tasks to create better models.
Revised: Furthermore, we also demonstrate that even a handful of tasks, repeated over multiple batches, would be sufﬁcient to achieve a performance similar to uniform sampling and draws into question the need for additional tasks to create better models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: It is widely recognized that humans can learn new concepts based on very little supervision, i.e., with few examples (or ”shots”), and generalize these concepts to unseen data as mentioned by Lake et al.
Revised: It is widely recognized that humans can learn new concepts based on very little supervision, i.e., with few examples (or ”shots”), and generalize these concepts to unseen data as mentioned by Lake et al.

Original: Recent advances in deep learning, on the other hand, have primarily relied on datasets with large amounts of labeled examples, primarily due to overﬁtting concerns in low data regimes.
Revised: Recent advances in deep learning, on the other hand, have primarily relied on datasets with large amounts of labeled examples, primarily due to overﬁtting concerns in low data regimes.

Original: Although the development of better data augmentation and regularization techniques can alleviate these concerns, many researchers now assume that future breakthroughs in low data regimes will emerge from meta-learning, or ”learning to learn.”
Revised: Although the development of better data augmentation and regularization techniques can alleviate these concerns, many researchers now assume that future breakthroughs in low data regimes will emerge from meta-learning, or ”learning to learn.”

Original: Here, we study the effect of task diversity in the low data regime and its effect on various models.
Revised: Here, we study the effect of task diversity in the low data regime and its effect on various models.

Original: In this meta-learning setting, a model is trained on a handful of labeled examples at a time under the assumption that it will learn how to correctly project examples of different classes and generalize this knowledge to unseen labels at test time.
Revised: In this meta-learning setting, a model is trained on a handful of labeled examples at a time under the assumption that it will learn how to correctly project examples of different classes and generalize this knowledge to unseen labels at test time.

Original: Although this setting is often used to illustrate the remaining gap between human capabilities and machine learning, we could argue that the domain of meta-learning is still nascent.
Revised: Although this setting is often used to illustrate the remaining gap between human capabilities and machine learning, we could argue that the domain of meta-learning is still nascent.

Original: The domain of task selection has remained virtually unexplored in this setting.
Revised: The domain of task selection has remained virtually unexplored in this setting.

Original: Conventional wisdom is that the performance of the model will improve as we train on more diverse tasks.
Revised: Conventional wisdom is that the performance of the model will improve as we train on more diverse tasks.

Original: To test this hypothesis to its limits, we deﬁne various task samplers which either limit task diversity by selecting a subset of overall tasks or improving task diversity using approaches such as
Revised: To test this hypothesis to its limits, we deﬁne various task samplers which either limit task diversity by selecting a subset of overall tasks or improving task diversity using approaches such as

Original: Determinantal Point Processes (DPPs) proposed by Macchi (1975).
Revised: Determinantal Point Processes (DPPs) proposed by Macchi (1975).

Original: Our contributions in this work are as follows:
Revised: Our contributions in this work are as follows:

Original: • We show that, against conventional wisdom, task diversity does not signiﬁcantly boost performance in meta-learning.
Revised: • We show that, against conventional wisdom, task diversity does not signiﬁcantly boost performance in meta-learning.

Original: Instead, limiting task diversity and repeating the same tasks over the training phase allows the model to obtain performances similar to models trained on Uniform Sampler without any adverse effects.
Revised: Instead, limiting task diversity and repeating the same tasks over the training phase allows the model to obtain performances similar to models trained on Uniform Sampler without any adverse effects.

Original: • We also show that increasing task diversity using sophisticated samplers such as DPP or Online Hard Task Mining (OHTM)
Revised: • We also show that increasing task diversity using sophisticated samplers such as DPP or Online Hard Task Mining (OHTM)

Original: Samplers do not signiﬁcantly boost performance.
Revised: Samplers do not signiﬁcantly boost performance.

Original: Instead, the dynamic-DPP Sampler harms the model due to the increased task diversity.
Revised: Instead, the dynamic-DPP Sampler harms the model due to the increased task diversity.

Original: • We empirically show that repeating tasks over the training phase can perform similarly to a model trained on the Uniform Sampler, achieving similar performance with only a
Revised: • We empirically show that repeating tasks over the training phase can perform similarly to a model trained on the Uniform Sampler, achieving similar performance with only a

Original: fragment of data
Revised: fragment of data

Original: This
Revised: This

Original: key
Revised: key

Original: ﬁnding questions the need to increase the support set pool to improve the model’s performance
Revised: ﬁnding questions the need to increase the support set pool to improve the model’s performance

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Meta-learning formulations typically rely on episodic training, wherein an algorithm adapts to a task, given its support set, to minimize the loss incurred on the query set.
Revised: Meta-learning formulations typically rely on episodic training, wherein an algorithm adapts to a task, given its support set, to minimize the loss incurred on the query set.

Original: Meta-learning methods differ in terms of the algorithms they learn, and can be broadly classiﬁed under four prominent classes: Metric-based , Model-based , Optimization-based and Bayesian-based approaches.
Revised: Meta-learning methods differ in terms of the algorithms they learn, and can be broadly classiﬁed under four prominent classes: Metric-based , Model-based , Optimization-based and Bayesian-based approaches.

Original: Metricbased methods such as Koch et al.
Revised: Metricbased methods such as Koch et al.

Original: ; Vinyals et al.
Revised: ; Vinyals et al.

Original: ; Snell et al.
Revised: ; Snell et al.

Original: ; Sung et al.
Revised: ; Sung et al.

Original: (2018) operate on the core idea similar to nearest neighbors algorithm and kernel density estimation.
Revised: (2018) operate on the core idea similar to nearest neighbors algorithm and kernel density estimation.

Original: These methods are also called non-parametric approaches.
Revised: These methods are also called non-parametric approaches.

Original: Model-based methods such as Santoro et al.
Revised: Model-based methods such as Santoro et al.

Original: (2016); Munkhdalai & Yu (2017) depend on a model designed speciﬁcally for fast learning, which updates its parameters rapidly with a few training steps, achieved by its internal architecture or controlled by another meta-learner model.
Revised: (2016); Munkhdalai & Yu (2017) depend on a model designed speciﬁcally for fast learning, which updates its parameters rapidly with a few training steps, achieved by its internal architecture or controlled by another meta-learner model.

Original: Generic deep learning models learn through backpropagation of gradients, which are neither designed to cope with a small number of training samples nor converge within a few optimization steps.
Revised: Generic deep learning models learn through backpropagation of gradients, which are neither designed to cope with a small number of training samples nor converge within a few optimization steps.

Original: To address this, Optimization-based methods such as Ravi &
Revised: To address this, Optimization-based methods such as Ravi &

Original: Larochelle (2016); Finn et al. (2017); Nichol et al.
Revised: Larochelle (2016); Finn et al. (2017); Nichol et al.

Original: (2018) were proposed, which were better suited to learn from a small number of samples.
Revised: (2018) were proposed, which were better suited to learn from a small number of samples.

Original: However, all the above approaches are deterministic and are not the most suited for few-shot problems that are generally ambiguous.
Revised: However, all the above approaches are deterministic and are not the most suited for few-shot problems that are generally ambiguous.

Original: Hence, Bayesian-based methods such as Yoon et al.
Revised: Hence, Bayesian-based methods such as Yoon et al.

Original: Requeima et al.
Revised: Requeima et al.

Original: (2019) were proposed which helped address the above issue.
Revised: (2019) were proposed which helped address the above issue.

Original: Although research in meta-learning models has attracted much attention recently, the effect of task diversity is virtually unexplored in the domain of meta-learning.
Revised: Although research in meta-learning models has attracted much attention recently, the effect of task diversity is virtually unexplored in the domain of meta-learning.

Original: However, task sampling and task diversity have been more extensively studied in other closely related problems such as active learning.
Revised: However, task sampling and task diversity have been more extensively studied in other closely related problems such as active learning.

Original: Active learning involves selecting unlabeled data items in order to improve an existing classiﬁer.
Revised: Active learning involves selecting unlabeled data items in order to improve an existing classiﬁer.

Original: Although most of the approaches in this domain are based on heuristics, there are few approaches to sample a batch of samples for active learning.
Revised: Although most of the approaches in this domain are based on heuristics, there are few approaches to sample a batch of samples for active learning.

Original: Ravi & Larochelle (2018) proposed an approach to sample a batch of samples using a protonet as the backbone architecture.
Revised: Ravi & Larochelle (2018) proposed an approach to sample a batch of samples using a protonet as the backbone architecture.

Original: The model tries to maximize the query set, given support set and unlabeled data.
Revised: The model tries to maximize the query set, given support set and unlabeled data.

Original: Other works such as Hsu et al.
Revised: Other works such as Hsu et al.

Original: (2018) proposed a framework named CACTUs, which samples tasks/examples using relatively simple task construction mechanisms such as clustering embeddings.
Revised: (2018) proposed a framework named CACTUs, which samples tasks/examples using relatively simple task construction mechanisms such as clustering embeddings.

Original: The unsupervised representations learned via these samples lead to a good performance on various downstream human-speciﬁed tasks.
Revised: The unsupervised representations learned via these samples lead to a good performance on various downstream human-speciﬁed tasks.

Original: Although nascent, a few recent works aim to improve meta-learning by explicitly looking at the task structure and relationships.
Revised: Although nascent, a few recent works aim to improve meta-learning by explicitly looking at the task structure and relationships.

Original: Among these, Yin et al.
Revised: Among these, Yin et al.

Original: (2019) proposed an approach to handle the lack of mutual exclusiveness among different tasks through an information-theoretic regularized objective.
Revised: (2019) proposed an approach to handle the lack of mutual exclusiveness among different tasks through an information-theoretic regularized objective.

Original: In addition, several popular meta-learning methods Lee et al.
Revised: In addition, several popular meta-learning methods Lee et al.

Original: ; Snell et al.
Revised: ; Snell et al.

Original: (2017) improve the meta-test performance by changing the number of ways or shots of the sampled metatraining tasks, thus increasing the complexity and diversity of the tasks.
Revised: (2017) improve the meta-test performance by changing the number of ways or shots of the sampled metatraining tasks, thus increasing the complexity and diversity of the tasks.

Original: Other works such as Liu et al.
Revised: Other works such as Liu et al.

Original: (2020a) proposed an approach to sample classes using class-pair-based sampling and classbased sampling.
Revised: (2020a) proposed an approach to sample classes using class-pair-based sampling and classbased sampling.

Original: The Class-pair based Sampler selects pairs of classes that confuse the model the most.
Revised: The Class-pair based Sampler selects pairs of classes that confuse the model the most.

Original: The class-based Sampler samples each class independently and does not consider the task’s difﬁculty as a whole.
Revised: The class-based Sampler samples each class independently and does not consider the task’s difﬁculty as a whole.

Original: Our OHTM sampler is similar to the Class-pair based Sampler.
Revised: Our OHTM sampler is similar to the Class-pair based Sampler.

Original: Other works such as Liu et al.
Revised: Other works such as Liu et al.

Original: (2020b) propose to augment the set of possible tasks by augmenting the predeﬁned set of classes that generate the tasks with varying degrees of rotated inputs as new classes.
Revised: (2020b) propose to augment the set of possible tasks by augmenting the predeﬁned set of classes that generate the tasks with varying degrees of rotated inputs as new classes.

Original: Other works such as Setlur et al.
Revised: Other works such as Setlur et al.

Original: (2020) look at the structure and diversity of tasks speciﬁcally through the lens of support set diversity, and show that, surprisingly, reducing diversity (by ﬁxing support set) not only maintains—but in many cases, signiﬁcantly improves—the performance of meta-learning.
Revised: (2020) look at the structure and diversity of tasks speciﬁcally through the lens of support set diversity, and show that, surprisingly, reducing diversity (by ﬁxing support set) not only maintains—but in many cases, signiﬁcantly improves—the performance of meta-learning.

Original: This experiment is very similar to our No Diversity Task Sampler if the size of the support set is equal to the number of classes per task.
Revised: This experiment is very similar to our No Diversity Task Sampler if the size of the support set is equal to the number of classes per task.

Original: However, in this work, we extend their work on MetaOptNet, Protonet to many other models and a myriad of samplers to better understand task diversity in meta-learning.
Revised: However, in this work, we extend their work on MetaOptNet, Protonet to many other models and a myriad of samplers to better understand task diversity in meta-learning.

Original: To the best of our knowledge, we are the ﬁrst to study the effect of task diversity in meta-learning to this extent.
Revised: To the best of our knowledge, we are the ﬁrst to study the effect of task diversity in meta-learning to this extent.

Original: [Figure]
Revised: [Figure]

Original: a
Revised: a

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: a
Revised: a

Original: [Figure]
Revised: [Figure]

Original: k
Revised: k

Original: [Figure]
Revised: [Figure]

Original: a
Revised: a

Original: [Figure]
Revised: [Figure]

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Here, we review some of the fundamental ideas required to understand our few-shot learning experiments better.
Revised: Here, we review some of the fundamental ideas required to understand our few-shot learning experiments better.

Original: 3. E PISODIC FEW - SHOT LEARNING
Revised: 3. E PISODIC FEW - SHOT LEARNING

Original: In episodic few-shot learning, an episode is represented as a K-way, N-shot classiﬁcation problem where N is the number of examples per class and K is the number of unique class labels.
Revised: In episodic few-shot learning, an episode is represented as a K-way, N-shot classiﬁcation problem where N is the number of examples per class and K is the number of unique class labels.

Original: During training, the data in each episode is provided as a support set S =
Revised: During training, the data in each episode is provided as a support set S =

Original: { ( x 1 , 1 , y 1 , 1 ) , ..., ( x N,K , y N,K ) } where x i,j ∈ R D is the i-th instance of the j-th class, and y j ∈ { 0 , 1 } K is its corresponding one-hot labeling vector.
Revised: { ( x 1 , 1 , y 1 , 1 ) , ..., ( x N,K , y N,K ) } where x i,j ∈ R D is the i-th instance of the j-th class, and y j ∈ { 0 , 1 } K is its corresponding one-hot labeling vector.

Original: Each episode aims to optimize a function f that classiﬁes new instances provided through a ”query” set Q , containing instances of the same class as S .
Revised: Each episode aims to optimize a function f that classiﬁes new instances provided through a ”query” set Q , containing instances of the same class as S .

Original: This task is difﬁcult because
Revised: This task is difﬁcult because

Original: N is typically very small (e,g, 1 to 10).
Revised: N is typically very small (e,g, 1 to 10).

Original: The classes change every episode.
Revised: The classes change every episode.

Original: The actual test set used to evaluate a model does not contain classes seen in support sets during training.
Revised: The actual test set used to evaluate a model does not contain classes seen in support sets during training.

Original: In the task-distribution view, meta-learning is a general-purpose learning algorithm that can generalize across tasks and ideally enable each new task to be learned better than the last.
Revised: In the task-distribution view, meta-learning is a general-purpose learning algorithm that can generalize across tasks and ideally enable each new task to be learned better than the last.

Original: We can evaluate the performance of ω over a distribution of tasks p ( T ) .
Revised: We can evaluate the performance of ω over a distribution of tasks p ( T ) .

Original: Here we loosely deﬁne a task to be a dataset and loss function
Revised: Here we loosely deﬁne a task to be a dataset and loss function

Original: T =
Revised: T =

Original: {D , L} .
Revised: {D , L} .

Original: Learning how to learn thus becomes:
Revised: Learning how to learn thus becomes:

Original: [Equation]
Revised: [Equation]

Original: where L ( D ; ω ) measures the performance of a model trained using ω on dataset D and p ( τ ) indicates the task distribution.
Revised: where L ( D ; ω ) measures the performance of a model trained using ω on dataset D and p ( τ ) indicates the task distribution.

Original: In this experiment, we extend this setting such that we vary the task diversity in the train split to study the effects on test split, which remains to use uniform or random sampling for tasks.
Revised: In this experiment, we extend this setting such that we vary the task diversity in the train split to study the effects on test split, which remains to use uniform or random sampling for tasks.

Original: 3. D ETERMINANTAL P OINT P ROCESSES (DPP S )
Revised: 3. D ETERMINANTAL P OINT P ROCESSES (DPP S )

Original: A DPP is a probability distribution over subsets of a ground set Y , where we assume Y = { 1 , 2 , ..., N } and N = |Y| .
Revised: A DPP is a probability distribution over subsets of a ground set Y , where we assume Y = { 1 , 2 , ..., N } and N = |Y| .

Original: An L-ensemble deﬁnes a DPP using a real, symmetric, and positive- deﬁnite matrix L indexed by the elements of
Revised: An L-ensemble deﬁnes a DPP using a real, symmetric, and positive- deﬁnite matrix L indexed by the elements of

Original: Y .
Revised: Y .

Original: The probability of sampling a subset Y =
Revised: The probability of sampling a subset Y =

Original: A ⊆ Y can be written as:
Revised: A ⊆ Y can be written as:

Original: [Equation]
Revised: [Equation]

Original: where L A :=
Revised: where L A :=

Original: [ L i,j ] i,j ∈ A is the restriction of L to the entries indexed by the elements of A.
Revised: [ L i,j ] i,j ∈ A is the restriction of L to the entries indexed by the elements of A.

Original: As L is a positive semi-deﬁnite, there exists a d × N matrix Ψ such that L = Ψ T Ψ where d ≤ N .
Revised: As L is a positive semi-deﬁnite, there exists a d × N matrix Ψ such that L = Ψ T Ψ where d ≤ N .

Original: Using this principle, we deﬁne the probability of sampling as:
Revised: Using this principle, we deﬁne the probability of sampling as:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: where the RHS is the squared volume of the parallelepiped spanned by { Ψ i } i ∈ A .
Revised: where the RHS is the squared volume of the parallelepiped spanned by { Ψ i } i ∈ A .

Original: In Eq.
Revised: In Eq.

Original: 3, Ψ i is deﬁned as the feature vector of element i , and each element L i,j in L is the similarity measured by dot products between elements i and j .
Revised: 3, Ψ i is deﬁned as the feature vector of element i , and each element L i,j in L is the similarity measured by dot products between elements i and j .

Original: Hence, we can verify that a DPP places higher probabilities on diverse sets because the more orthogonal the feature vectors are, the larger the volume parallelepiped spanned by the feature vector is.
Revised: Hence, we can verify that a DPP places higher probabilities on diverse sets because the more orthogonal the feature vectors are, the larger the volume parallelepiped spanned by the feature vector is.

Original: In this work, these feature embeddings represent class embeddings, which are derived using either a pre-trained protonet model or the model being trained as discussed in Sec. 3.3.
Revised: In this work, these feature embeddings represent class embeddings, which are derived using either a pre-trained protonet model or the model being trained as discussed in Sec. 3.3.

Original: In a DPP, the cardinality of a sampled subset, | A | , is random in general.
Revised: In a DPP, the cardinality of a sampled subset, | A | , is random in general.

Original: A k -DPP is an extension of the DPP proposed in the work of Kuhn et al.
Revised: A k -DPP is an extension of the DPP proposed in the work of Kuhn et al.

Original: (2003), where the cardinality of subsets are ﬁxed as k (i.e., | A | = k ).
Revised: (2003), where the cardinality of subsets are ﬁxed as k (i.e., | A | = k ).

Original: In this work, we use k -DPPs as an off-the-shelf implementation to retrieve classes that represent a task used in the meta-learning step.
Revised: In this work, we use k -DPPs as an off-the-shelf implementation to retrieve classes that represent a task used in the meta-learning step.

Original: 3. T ASK S AMPLING
Revised: 3. T ASK S AMPLING

Original: In this work, we experiment with eight distinct task samplers, each offering a different level of task diversity.
Revised: In this work, we experiment with eight distinct task samplers, each offering a different level of task diversity.

Original: To demonstrate the task samplers, we use a 2-way classiﬁcation problem with a meta-batch size of 2 and denote each class with a unique alphabet from the Omniglot dataset.
Revised: To demonstrate the task samplers, we use a 2-way classiﬁcation problem with a meta-batch size of 2 and denote each class with a unique alphabet from the Omniglot dataset.

Original: Uniform Sampler This is the most widely used Sampler used in the setting of meta-learning.
Revised: Uniform Sampler This is the most widely used Sampler used in the setting of meta-learning.

Original: The
Revised: The

Original: Sampler gives equal probability to every task and is intuitively a random sampler.
Revised: Sampler gives equal probability to every task and is intuitively a random sampler.

Original: An illustration of this Sampler is shown in Figure 1.
Revised: An illustration of this Sampler is shown in Figure 1.

Original: No Diversity Task Sampler
Revised: No Diversity Task Sampler

Original: In this setting, we uniformly sample one set of the task at the beginning and propagate the same task across all batches and meta-batches.
Revised: In this setting, we uniformly sample one set of the task at the beginning and propagate the same task across all batches and meta-batches.

Original: Note that repeating the same class over and over again does not simply repeat the same images/inputs as we episodically retrieve different images for each class.
Revised: Note that repeating the same class over and over again does not simply repeat the same images/inputs as we episodically retrieve different images for each class.

Original: An illustration of this Sampler is shown in Figure 1.
Revised: An illustration of this Sampler is shown in Figure 1.

Original: No Diversity Batch Sampler
Revised: No Diversity Batch Sampler

Original: In this setting, we uniformly sample one set of tasks for batch one and propagate the same tasks across all other batches.
Revised: In this setting, we uniformly sample one set of tasks for batch one and propagate the same tasks across all other batches.

Original: Furthermore, we shufﬂe these tasks to enforce that the model does not overﬁt.
Revised: Furthermore, we shufﬂe these tasks to enforce that the model does not overﬁt.

Original: An illustration of this Sampler is shown in Figure 1.
Revised: An illustration of this Sampler is shown in Figure 1.

Original: No Diversity Tasks per Batch Sampler
Revised: No Diversity Tasks per Batch Sampler

Original: In this setting, we uniformly sample one set of tasks for a given batch and propagate the same tasks for all meta-batches.
Revised: In this setting, we uniformly sample one set of tasks for a given batch and propagate the same tasks for all meta-batches.

Original: We then repeat this same principle for sampling the next batch.
Revised: We then repeat this same principle for sampling the next batch.

Original: Furthermore, we shufﬂe these tasks to enforce that the model does not overﬁt.
Revised: Furthermore, we shufﬂe these tasks to enforce that the model does not overﬁt.

Original: An illustration of this Sampler is shown in Figure 2.
Revised: An illustration of this Sampler is shown in Figure 2.

Original: Single Batch Uniform Sampler
Revised: Single Batch Uniform Sampler

Original: In this setting, we set the meta-batch size to one.
Revised: In this setting, we set the meta-batch size to one.

Original: This Sampler is intuitively the same as no diversity task per batch sampler, without the repetition of tasks.
Revised: This Sampler is intuitively the same as no diversity task per batch sampler, without the repetition of tasks.

Original: This
Revised: This

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Sampler would be an ideal ablation study for the repetition of tasks in the meta-learning setting.
Revised: Sampler would be an ideal ablation study for the repetition of tasks in the meta-learning setting.

Original: An illustration of this Sampler is shown in Figure 2.
Revised: An illustration of this Sampler is shown in Figure 2.

Original: Online Hard Task Mining Sampler
Revised: Online Hard Task Mining Sampler

Original: This setting is inspired by the works of Shrivastava et al.
Revised: This setting is inspired by the works of Shrivastava et al.

Original: (2016) where they proposed OHEM, which yielded signiﬁcant boosts in detection performance on benchmarks like PASCAL VOC 2007 and 2012.
Revised: (2016) where they proposed OHEM, which yielded signiﬁcant boosts in detection performance on benchmarks like PASCAL VOC 2007 and 2012.

Original: However, to reproduce OHEM for meta-learning, we only apply the OHEM sampler for half the meta-batch size and uniform sampler for the remaining half.
Revised: However, to reproduce OHEM for meta-learning, we only apply the OHEM sampler for half the meta-batch size and uniform sampler for the remaining half.

Original: This approach would allow us to involve many tasks and not restrict us to only known tasks.
Revised: This approach would allow us to involve many tasks and not restrict us to only known tasks.

Original: Furthermore, to avoid OHEM in the initial stages, we sample tasks with a uniform sampler until the buffer of tasks seen by the model becomes sufﬁciently big, say 50 in our case.
Revised: Furthermore, to avoid OHEM in the initial stages, we sample tasks with a uniform sampler until the buffer of tasks seen by the model becomes sufﬁciently big, say 50 in our case.

Original: An illustration of this
Revised: An illustration of this

Original: Sampler is shown in Figure 3.
Revised: Sampler is shown in Figure 3.

Original: Static DPP Sampler Determinantal Point Processes (DPP) have been used for several machine learning problems such as the works of Kulesza & Taskar (2012).
Revised: Static DPP Sampler Determinantal Point Processes (DPP) have been used for several machine learning problems such as the works of Kulesza & Taskar (2012).

Original: They have also been used in other problems such as the active learning settings in the works of Bıyık et al.
Revised: They have also been used in other problems such as the active learning settings in the works of Bıyık et al.

Original: and mini-batch sampling problems in the works of Zhang et al.
Revised: and mini-batch sampling problems in the works of Zhang et al.

Original: These algorithms have also inspired other works in active learning in the batch mode setting, such as Ravi & Larochelle (2018).
Revised: These algorithms have also inspired other works in active learning in the batch mode setting, such as Ravi & Larochelle (2018).

Original: In this setting, we use DPP as an off-the-shelf implementation to sample tasks based on task embeddings.
Revised: In this setting, we use DPP as an off-the-shelf implementation to sample tasks based on task embeddings.

Original: These task embeddings are generated using our pre-trained protonet model.
Revised: These task embeddings are generated using our pre-trained protonet model.

Original: The DPP instance is used to sample the most diverse tasks based on these embeddings and used for meta-learning.
Revised: The DPP instance is used to sample the most diverse tasks based on these embeddings and used for meta-learning.

Original: An illustration of this Sampler is shown in Figure 3.
Revised: An illustration of this Sampler is shown in Figure 3.

Original: Dynamic DPP Sampler
Revised: Dynamic DPP Sampler

Original: In this setting, we extend the previous sDPP setting such that the model in training generates the task embeddings.
Revised: In this setting, we extend the previous sDPP setting such that the model in training generates the task embeddings.

Original: The Sampler is motivated by the intuition that selecting the most diverse tasks for a given model will help learn better.
Revised: The Sampler is motivated by the intuition that selecting the most diverse tasks for a given model will help learn better.

Original: Furthermore, to avoid DPP in the initial stages, we sample tasks with a uniform sampler until the model becomes sufﬁciently trained, say 500 batches in our case.
Revised: Furthermore, to avoid DPP in the initial stages, we sample tasks with a uniform sampler until the model becomes sufﬁciently trained, say 500 batches in our case.

Original: An illustration of this Sampler is shown in Figure 3.
Revised: An illustration of this Sampler is shown in Figure 3.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The experiment aims to answer the following questions: (a) How does task diversity affect metalearning?
Revised: The experiment aims to answer the following questions: (a) How does task diversity affect metalearning?

Original: (b) Do sophisticated samplers such as OHEM or DPP offer any signiﬁcant boost in performance?
Revised: (b) Do sophisticated samplers such as OHEM or DPP offer any signiﬁcant boost in performance?

Original: (c) Are there any rule of thumb or general good practices when it comes to sampling tasks?
Revised: (c) Are there any rule of thumb or general good practices when it comes to sampling tasks?

Original: To make an exhaustive study on the effect of task diversity in meta-learning, we train on four datasets: Omniglot Lake et al.
Revised: To make an exhaustive study on the effect of task diversity in meta-learning, we train on four datasets: Omniglot Lake et al.

Original: (2011), mini Imagenet Ravi & Larochelle (2016), tiered ImageNet
Revised: (2011), mini Imagenet Ravi & Larochelle (2016), tiered ImageNet

Original: Ren et al.
Revised: Ren et al.

Original: (2018), and Meta-Dataset Triantaﬁllou et al.
Revised: (2018), and Meta-Dataset Triantaﬁllou et al.

Original: With this selection of datasets, we cover both simple datasets, such as Omniglot and mini ImageNet, as well as the most difﬁcult ones, such as tiered ImageNet and Meta-Dataset.
Revised: With this selection of datasets, we cover both simple datasets, such as Omniglot and mini ImageNet, as well as the most difﬁcult ones, such as tiered ImageNet and Meta-Dataset.

Original: We train three broad classes of meta-learning models on
Revised: We train three broad classes of meta-learning models on

Original: [Figure]

Original: [Figure]

Original: N

Original: [Figure]
Revised: [Figure]

Original: N

Original: [Figure]

Original: N

Original: [Figure]
Revised: [Figure]

Original: N

Original: [Figure] r a c y [Figure] these datasets - Metric-based (i.e., Protonet, Matching Networks), Optimization-based (i.e., MAML,
Revised: these datasets - Metric-based (i.e., Protonet, Matching Networks), Optimization-based (i.e., MAML,

Original: Reptile, and MetaOptNet), and Bayesian meta-learning models (i.e., CNAPs).
Revised: Reptile, and MetaOptNet), and Bayesian meta-learning models (i.e., CNAPs).

Original: More details about the datasets which were used in our experiments are discussed in App. A.1.
Revised: More details about the datasets which were used in our experiments are discussed in App. A.1.

Original: More details about the models and their hyperparameters are discussed in App.
Revised: More details about the models and their hyperparameters are discussed in App.

Original: A.2.
Revised: A.2.

Original: We created a common pool ofrandomly sampled held-out tasks to test every algorithm in our experiments to make an accurate comparison.
Revised: We created a common pool ofrandomly sampled held-out tasks to test every algorithm in our experiments to make an accurate comparison.

Original: For all experiments, we assessed the statistical signiﬁcance of our results based on a paired-difference t-test, with a p-value p = 0 . 05 .
Revised: For all experiments, we assessed the statistical signiﬁcance of our results based on a paired-difference t-test, with a p-value p = 0 . 05 .

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: In this section, we present the results of our experiments.
Revised: In this section, we present the results of our experiments.

Original: Figure 4 presents the performance of the six models on the Omniglot and mini ImageNet under different task samplers in the 5-way 1-shot setting.
Revised: Figure 4 presents the performance of the six models on the Omniglot and mini ImageNet under different task samplers in the 5-way 1-shot setting.

Original: Table 1 in the Appendix presents the same results with higher precision.
Revised: Table 2 in the Appendix presents the same results with higher precision.

Original: We also reproduce our experiments on the 20-way 1-shot setting on the Omniglot dataset to establish that these trends are shared across different settings.
Revised: We also reproduce our experiments on the 20-way 1-shot setting on the Omniglot dataset to establish that these trends are shared across different settings.

Original: Figure 5 presents our performance of the models under this setting.
Revised: Figure 5 presents our performance of the models under this setting.

Original: Furthermore, the results on the 20-way 1-shot experiments are presented in
Revised: Furthermore, the results on the 20-way 1-shot experiments are presented in

Original: Table 2 in the Appendix with higher precision.
Revised: Table 3 in the Appendix with higher precision.

Original: To further establish our ﬁndings, we also present our result on notoriously harder datasets such as tiered ImageNet and Meta-Dataset.
Revised: To further establish our ﬁndings, we also present our result on notoriously harder datasets such as tiered ImageNet and Meta-Dataset.

Original: Figure 6 presents the performance of the models on the tiered ImageNet.
Revised: Figure 6 presents the performance of the models on the tiered ImageNet.

Original: Again, Table 1 in the Appendix presents the same results with higher precision.
Revised: Again, Table 2 in the Appendix presents the same results with higher precision.

Original: MAML
Revised: MAML

Original: Reptile
Revised: Reptile

Original: Protonet
Revised: Protonet

Original: Matching Net.
Revised: Matching Net.

Original: MetaOptNet
Revised: MetaOptNet

Original: CNAPs
Revised: CNAPs

Original: H H
Revised: H H

Original: H H
Revised: H H

Original: H
Revised: H

Original: H H H
Revised: H H H

Original: H H
Revised: H H

Original: H
Revised: H

Original: H H
Revised: H H

Original: N
Revised: N

Original: H H
Revised: H H

Original: H H
Revised: H H

Original: H H
Revised: H H

Original: H
Revised: H

Original: H H H
Revised: H H H

Original: H H
Revised: H H

Original: H H
Revised: H H

Original: N
Revised: N

Original: H H
Revised: H H

Original: H H
Revised: H H

Original: H H
Revised: H H

Original: H
Revised: H

Original: A cc u r a c y tiered ImageNet 5-way 1-shot
Revised: A cc u r a c y tieredImageNet 5-way 1-shot

Original: Uniform
Revised: Uniform

Revised: [Figure]

Original: No
Revised: No

Original: [Figure]
Revised: [Figure]

Original: Task No Diversity Batch No Diversity Tasks/Batch
Revised: /Batch

Original: Single
Revised: Single

Original: OHTM s-DPP d-DPP
Revised: OHTM s-DPP d-DPP

Original: MAML
Revised: MAML

Original: Reptile
Revised: Reptile

Original: Protonet
Revised: Protonet

Original: Matching Net.
Revised: Matching Net.

Original: MetaOptNet
Revised: MetaOptNet

Original: CNAPs
Revised: CNAPs

Original: [Figure]
Revised: [Figure]

Original: N
Revised: N

Original: H H
Revised: H H

Original: H
Revised: H

Original: H H
Revised: H H

Original: N
Revised: N

Original: H H
Revised: H H

Original: H
Revised: H

Original: H H
Revised: H H

Original: N
Revised: N

Original: H H
Revised: H H

Original: H
Revised: H

Original: N
Revised: N

Original: H H
Revised: H H

Original: H
Revised: H

Original: N
Revised: N

Original: H H
Revised: H H

Original: H
Revised: H

Original: N
Revised: N

Original: H
Revised: H

Original: A cc u r a c y
Revised: A cc u r a c y

Original: Meta-Dataset (Traﬃc Signs) 5-way 1-shot
Revised: Meta-Dataset (Traﬃc Signs) 5-way 1-shot

Original: MAML
Revised: MAML

Original: Reptile
Revised: Reptile

Original: Protonet
Revised: Protonet

Original: Matching Net.
Revised: Matching Net.

Original: MetaOptNet
Revised: MetaOptNet

Original: CNAPs
Revised: CNAPs

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: H H
Revised: H H

Original: [Figure]
Revised: [Figure]

Original: H
Revised: H

Original: H H
Revised: H H

Original: H H
Revised: H H

Original: N N
Revised: N N

Original: N
Revised: N

Original: H H
Revised: H H

Original: N
Revised: N

Original: H
Revised: H

Original: N N
Revised: N N

Original: N
Revised: N

Original: H N
Revised: H N

Original: N
Revised: N

Original: H
Revised: H

Original: [Figure]
Revised: [Figure]

Original: Figure 6 presents the performance of the models on the Meta-Dataset Trafﬁc Sign and Meta-Dataset
Revised: Figure 6 presents the performance of the models on the Meta-Dataset Trafﬁc Sign and Meta-Dataset

Original: MSCOCO datasets.
Revised: MSCOCO datasets.

Original: We only present the results on Trafﬁc Sign and MSCOCO of the Meta-Dataset, as these two sub-datasets are exclusively used for testing and are an accurate representation of the generalization power of the models when trained with different levels of task diversity.
Revised: We only present the results on Trafﬁc Sign and MSCOCO of the Meta-Dataset, as these two sub-datasets are exclusively used for testing and are an accurate representation of the generalization power of the models when trained with different levels of task diversity.

Original: Other results on the Meta-Dataset are presented in Sec.
Revised: Other results on the Meta-Dataset are presented in Table 4.

Original: We empirically show that task diversity does not lead to any signiﬁcant boost in the performance of the models.
Revised: We empirically show that task diversity does not lead to any signiﬁcant boost in the performance of the models.

Original: In the subsequent section, we discuss some of the other key ﬁndings from our work.
Revised: In the subsequent section, we discuss some of the other key ﬁndings from our work.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this section, we discuss few empirical results from our experiments and shed light on some of the key ﬁndings from our research.
Revised: In this section, we discuss few empirical results from our experiments and shed light on some of the key ﬁndings from our research.

Original: Poor performance by NDT Sampler The lowest performance is consistently obtained by the
Revised: Poor performance by NDT Sampler The lowest performance is consistently obtained by the

Original: No Diversity Task Sampler, which is reasonable since the model only sees one task throughout its training.
Revised: No Diversity Task Sampler, which is reasonable since the model only sees one task throughout its training.

Original: What is fascinating is that just one task is sufﬁcient for the model to reach a reasonably decent performance in most cases.
Revised: What is fascinating is that just one task is sufﬁcient for the model to reach a reasonably decent performance in most cases.

Original: We do notice instances where NDT Sampler performs very well on a few sub-datasets of the Meta-Dataset.
Revised: We do notice instances where NDT Sampler performs very well on a few sub-datasets of the Meta-Dataset.

Original: This can be explained by the fact that the model has only be trained for a single sub-dataset and has relatively less noise when compared to training on multiple sub-datasets.
Revised: This can be explained by the fact that the model has only be trained for a single sub-dataset and has relatively less class variability (noise) when compared to training on multiple sub-datasets.

Original: Poor performance by Single Batch Uniform Sampler Consequently, the Single Batch Uniform Sampler does perform poorly on most datasets, including Omniglot, mini ImageNet, and tiered ImageNet.
Revised: Poor performance by Single Batch Uniform Sampler Consequently, the Single Batch Uniform Sampler does perform poorly on most datasets, including Omniglot, mini ImageNet, and tiered ImageNet.

Original: This is reasonable since the model is trained on a tiny pool of the dataset.
Revised: This is reasonable since the model is trained on a tiny pool of the dataset.

Original: However, we notice instances where the Sampler performs very well on a few sub-datasets of the Meta-Dataset.
Revised: However, we notice instances where the Sampler performs very well on a few sub-datasets of the MetaDataset.

Original: We hypothesize that training on fewer samples keeps the model unaware of the inherent noise generated by training on diverse datasets and aids better performance in the case of Meta-Dataset.
Revised: We hypothesize that training on fewer samples keeps the model unaware of the inherent class variability generated by training on diverse datasets and aids better performance Dataset

Original: Disparity between Single Batch Uniform and NDTB Sampler Another exciting result is the
Revised: Disparity between Single Batch Uniform and NDTB Sampler Another exciting result is the

Original: Disparity between Single Batch Uniform Sampler and No Diversity Tasks per Batch Sampler.
Revised: Disparity between Single Batch Uniform Sampler and No Diversity Tasks per Batch Sampler.

Original: As mentioned earlier, the only difference between the two samplers is that tasks are repeated in the latter.
Revised: As mentioned earlier, the only difference between the two samplers is that tasks are repeated in the latter.

Original: However, this repetition seems to offer a great deal of information to the model and allows the model to perform on par with the Uniform Sampler.
Revised: However, this repetition seems to offer a great deal of information to the model and allows the model to perform on par with the Uniform Sampler.

Original: It might be possible that the Single Batch
Revised: It might be possible that the Single Batch

Original: Uniform Sampler obtains the performance observed by the No Diversity Tasks per Batch Sampler if trained for enough epochs.
Revised: Uniform Sampler obtains the performance observed by the No Diversity Tasks per Batch Sampler if trained for enough epochs.

Original: However, it would be safe to comment that the convergence of the model is signiﬁcantly faster in the latter.
Revised: However, it would be safe to comment that the convergence of the model is signiﬁcantly faster in the latter.

Original: Thus, repeating tasks might help speed up the convergence of the model when we have a ﬁxed and handful amount of data.
Revised: Thus, repeating tasks might help speed up the convergence of the model when we have a ﬁxed and handful amount of data.

Original: However, the same is not valid for models trained on Meta-Dataset.
Revised: However, the same is not valid for models trained on Meta-Dataset.

Original: Although both samplers are trained over a similar pool of datasets, the NDTB sampler sees more data and might lead to more inherent noise generated by training on diverse datasets.
Revised: Although both samplers are trained over a similar pool of datasets, the NDTB sampler sees more data and might lead to more inherent class variability generated by training on diverse datasets.

Original: This might explain why repeating tasks leads to lower performance in this case.
Revised: This might explain why repeating tasks leads to lower performance in this case.

Original: Disparity between s-DPP and d-DPP Sampler We also note that s-DPP and d-DPP samplers do not offer any boost in performance when compared to the regular Uniform Sampler.
Revised: Disparity between s-DPP and d-DPP Sampler We also note that s-DPP and d-DPP samplers do not offer any boost in performance when compared to the regular Uniform Sampler.

Original: Furthermore, there seems to be a signiﬁcant disparity between these two samplers.
Revised: Furthermore, there seems to be a signiﬁcant disparity between these two samplers.

Original: We believe that d-DPP, which computes the most diverse tasks at regular intervals, harms the model with the diverse tasks since we observe that the model’s performance degrades over epochs.
Revised: We believe that d-DPP, which computes the most diverse tasks at regular intervals, harms the model with the diverse tasks since we observe that the model’s performance degrades over epochs.

Original: For example, consider the scenario where the model is trained on tasks involving dogs and tractors.
Revised: For example, consider the scenario where the model is trained on tasks involving dogs and tractors.

Original: This task is relatively easy to learn and would not require the model to ﬁne-tune a great deal.
Revised: This task is relatively easy to learn and would not require the model to ﬁne-tune a great deal.

Original: However, during test time, suppose our task involves classifying cats and dogs; this would be a problem since the model has not learned the intricacies of the two classes.
Revised: However, during test time, suppose our task involves classifying cats and dogs; this would be a problem since the model has not learned the intricacies of the two classes.

Original: Thus, diversity seems to do more harm than good in this case.
Revised: Thus, diversity seems to do more harm than good in this case.

Original: The best example of this is observed by Matching Networks in Omniglot 5-way 1-shot setting as shown in Figure-7, where each instance of diverse sampling harms the model signiﬁcantly.
Revised: The best example of this is observed by Matching Networks in Omniglot 5-way 1-shot setting as shown in Figure 10, where each instance of diverse sampling harms the model signiﬁcantly.

Original: Limitation of samplers with DPP backbone Samplers such as s-DPP and d-DPP, which use
Revised: Limitation of samplers with DPP backbone Samplers such as s-DPP and d-DPP, which use

Original: DPP to sample diverse tasks, require task embeddings of every class in the dataset.
Revised: DPP to sample diverse tasks, require task embeddings of every class in the dataset.

Original: Computing these task embeddings, although intensive, might be sustainable for small datasets such as Omniglot, mini ImageNet, and tiered ImageNet.
Revised: Computing these task embeddings, although intensive, might be sustainable for small datasets such as Omniglot, mini ImageNet, and tiered ImageNet.

Original: However, computing the task embedding for every class of a dataset as large as the Meta-Dataset is nearly impossible due to the time and memory constraints.
Revised: However, computing the task embedding for every class of a dataset as large as the Meta-Dataset is nearly impossible due to the time and memory constraints.

Original: Hence, in our experiments on the Meta-Dataset, we do not run the model using the s-DPP and d-DPP
Revised: Hence, in our experiments on the Meta-Dataset, we do not run the model using the s-DPP and d-DPP

Original: Sampler and only report the ﬁndings from the remaining samplers.
Revised: Sampler and only report the ﬁndings from the remaining samplers.

Original: OHTM Sampler offers no signiﬁcant performance boost The OHTM Sampler is quite sophisticated since it regularly samples diverse tasks, as well as selects the most challenging tasks to improve the model.
Revised: The OHTM Sampler is quite sophisticated since it regularly samples diverse tasks, as well as selects the most challenging tasks to improve the model.

Original: It is needless to say; the model requires more computational power and time than the
Revised: It is needless to say; the model requires more computational power and time than the

Original: Uniform Sampler.
Revised: Uniform Sampler.

Original: However, the OHTM Sampler offers no signiﬁcant boost in performance when compared to the Uniform Sampler in the case of Omniglot, mini ImageNet, and tiered ImageNet.
Revised: However, the OHTM Sampler offers no signiﬁcant boost in performance when compared to the Uniform Sampler in the case of Omniglot, mini ImageNet, and tiered ImageNet.

Original: However, for Meta-Dataset, we notice that the OHTM Sampler sometimes leads to improved performance.
Revised: However, for Meta-Dataset, we notice that the OHTM Sampler sometimes leads to improved performance.

Original: This ﬁnding is quite puzzling since the Sampler works similarly across all datasets and does not address the inherent noise generated from training on diverse datasets such as the MetaDataset.
Revised: This ﬁnding is quite puzzling since the Sampler works similarly across all datasets and does not address the inherent class variability generated from training on diverse datasets such as the Meta-Dataset.

Original: The behavior of the OHTM Sampler on Meta-Datasets warrants further research.
Revised: The behavior of the OHTM Sampler on Meta-Datasets warrants further research.

Original: Comparison between NDTB, NDB, and Uniform Sampler From our experiments, we also notice that the No Diversity Tasks per Batch Sampler and No Diversity Batch Sampler are pretty similar to the Uniform Sampler in terms of performance.
Revised: Comparison between NDTB, NDB, and Uniform Sampler From our experiments, we also notice that the No Diversity Tasks per Batch Sampler and No Diversity Batch Sampler are pretty similar to the Uniform Sampler in terms of performance.

Original: This would suggest that the model trained on only a data fragment can perform similarly to that trained on the Uniform Sampler.
Revised: This would suggest that the model trained on only a data fragment can perform similarly to that trained on the Uniform Sampler.

Original: Abnormal run of matching networks d-DPP (20-way 1-shot)
Revised: Abnormal run of matching networks d-DPP (20-way 1-shot)

Original: In our run on the matching networks with the d-DPP Sampler under the 20-way 1-shot setting, we ran across a peculiar error.
Revised: In our run on the matching networks with the d-DPP Sampler under the 20-way 1-shot setting, we ran across a peculiar error.

Original: The prototypes generated by the matching networks were sometimes not ﬁt to be used by the dDPP Sampler to sample 20 unique classes.
Revised: The prototypes generated by the matching networks were sometimes not ﬁt to be used by the dDPP Sampler to sample 20 unique classes.

Original: The reason is that the rank of the matrix generated using the embeddings was lower than the required number of classes per task (i.e., 20).
Revised: The reason is that the rank of the matrix generated using the embeddings was lower than the required number of classes per task (i.e., 20).

Original: To create a workaround for this sole experiment, we chose to sample 5 diverse classes at a time and append them to create the task.
Revised: To create a workaround for this sole experiment, we chose to sample 5 diverse classes at a time and append them to create the task.

Original: We hypothesize that the prototypes created by matching networks are unsuitable for downstream tasks and warrant further research regarding this behavior.
Revised: We hypothesize that the prototypes created by matching networks are unsuitable for downstream tasks and warrant further research regarding this behavior.

Original: Poor performance of MAML on Meta-Dataset In our experiments on Meta-Dataset, we notice that MAML performs signiﬁcantly worse than other models.
Revised: Poor performance of MAML on Meta-Dataset In our experiments on Meta-Dataset, we notice that MAML performs signiﬁcantly worse than other models.

Original: Some of the reasons for this disparity of performance when compared to Triantaﬁllou et al.
Revised: Some of the reasons for this disparity of performance when compared to Triantaﬁllou et al.

Original: (2019) have been discussed in detail in Appendix
Revised: (2019) have been discussed in detail in Appendix

Original: A.1.
Revised: A.1.

Original: Furthermore, we hypothesize that the poor performance of MAML can be attributed to the way its adaptation process works: MAML learns within episode weights of the model before adapting to a set of new tasks via meta-update or outer loop update.
Revised: Furthermore, we hypothesize that the poor performance of MAML can be attributed to the way its adaptation process works: MAML learns within episode weights of the model before adapting to a set of new tasks via meta-update or outer loop update.

Original: This new task is again sampled from the pool of tasks and is a different set of data altogether.
Revised: This new task is again sampled from the pool of tasks and is a different set of data altogether.

Original: In most cases, where the model is trained on only one dataset, this intuition would make sense and lead to high-performing models.
Revised: In most cases, where the model is trained on only one dataset, this intuition would make sense and lead to high-performing models.

Original: However, in the case of Meta-Dataset, where the new set of tasks might be of entirely different datasets or domains, this approach tends to do more harm to the model rather than aid.
Revised: However, in the case of Meta-Dataset, where the new set of tasks might be of entirely different datasets or domains, this approach tends to do more harm to the model rather than aid.

Original: In the work of Triantaﬁllou et al.
Revised: In the work of Triantaﬁllou et al.

Original: (2019), the authors adapted MAML such that it focuses on learning the within-episode initialization θ of the embedding network so that it can be rapidly adapted for a new task.
Revised: (2019), the authors adapted MAML such that it focuses on learning the within-episode initialization θ of the embedding network so that it can be rapidly adapted for a new task.

Original: This allows the model to learn from a variable number of ways and shots per episode.
Revised: This allows the model to learn from a variable number of ways and shots per episode.

Original: Peculiar behavior with MetaOptNet model Compared to all other models, MetaOptNet seems to be immune to the effects of task diversity to a great extent.
Revised: Peculiar behavior with MetaOptNet model Compared to all other models, MetaOptNet seems to be immune to the effects of task diversity to a great extent.

Original: The convergence of the model seems to follow a general pattern and achieve similar performance across task distributions except for the
Revised: The convergence of the model seems to follow a general pattern and achieve similar performance across task distributions except for the

Original: Single Batch Uniform Sampler and No Diversity Task sampler.
Revised: Single Batch Uniform Sampler and No Diversity Task sampler.

Original: Furthermore, we do not observe the expected pattern of d-DPP Sampler, where the performance drops upon mining diverse tasks.
Revised: Furthermore, we do not observe the expected pattern of d-DPP Sampler, where the performance drops upon mining diverse tasks.

Original: We present the convergence graph of the MetaOptNet model on Omniglot 5-way 1-shot run in Figurein the Appendix with an added smoothing factor of 1.
Revised: We present the convergence graph of the MetaOptNet model on Omniglot 5-way 1-shot run in Figurein the Appendix with an added smoothing factor of 1.

Original: General Trend From our experiments, we notice that there are generally two classes of samplers:
Revised: General Trend From our experiments, we notice that there are generally two classes of samplers:

Original: High Performing Samplers and Low Performing Samplers.
Revised: High Performing Samplers and Low Performing Samplers.

Original: The High Performing Samplers include
Revised: The High Performing Samplers include

Original: No Diversity Batch, No Diversity Tasks per Batch, Uniform, OHTM, and s-DPP Sampler.
Revised: No Diversity Batch, No Diversity Tasks per Batch, Uniform, OHTM, and s-DPP Sampler.

Original: The Low
Revised: The Low

Original: Performing Samplers include No Diversity Task, Single Batch Uniform, and d-DPP Sampler.
Revised: Performing Samplers include No Diversity Task, Single Batch Uniform, and d-DPP Sampler.

Original: This trend is shared across all datasets and models.
Revised: This trend is shared across all datasets and models.

Original: There are some perturbations in ranking within the two classes, but the High Performing Samplers tend to perform better than the Low Performing
Revised: There are some perturbations in ranking within the two classes, but the High Performing Samplers tend to perform better than the Low Performing

Original: Samplers.
Revised: Samplers.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have studied the effect of task diversity in meta-learning.
Revised: In this paper, we have studied the effect of task diversity in meta-learning.

Original: We have empirically shown that task diversity does not lead to any signiﬁcant boost in performance in meta-learning.
Revised: We have empirically shown that task diversity does not lead to any signiﬁcant boost in performance in meta-learning.

Original: Instead, limiting task diversity and repeating the same tasks over the training phase allows us to obtain similar performances to the Uniform Sampler without any signiﬁcant adverse effects.
Revised: Instead, limiting task diversity and repeating the same tasks over the training phase allows us to obtain similar performances to the Uniform Sampler without any signiﬁcant adverse effects.

Original: Furthermore,
Revised: Furthermore,

Original: We also show that sophisticated samplers such as OHEM or DPP samplers do not offer any signiﬁcant boost in performance.
Revised: We also show that sophisticated samplers such as OHEM or DPP samplers do not offer any signiﬁcant boost in performance.

Original: In contradiction, we notice that increasing task diversity using the d-DPP
Revised: In contradiction, we notice that increasing task diversity using the d-DPP

Original: Sampler hampers the performance of the meta-learning model.
Revised: Sampler hampers the performance of the meta-learning model.

Original: Our experiments using the NDTB and NDB empirically show that a model trained on even a tiny data fragment can perform similarly to a model trained using Uniform Sampler.
Revised: Our experiments using the NDTB and NDB empirically show that a model trained on even a tiny data fragment can perform similarly to a model trained using Uniform Sampler.

Original: This is a crucial ﬁnding since this questions the need to increase the support set pool to improve the models’ performance.
Revised: This is a crucial ﬁnding since this questions the need to increase the support set pool to improve the models’ performance.

Original: We believe that the experiments we performed lay the roadwork to further research for the effect of task diversity domain in meta-learning and lay some groundwork and rules of thumb for task sampling for meta-learning.
Revised: We believe that the experiments we performed lay the roadwork to further research for the effect of task diversity domain in meta-learning and lay some groundwork and rules of thumb for task sampling for meta-learning.

Original: R EPRODUCABILITY S TATEMENT
Revised: R EPRODUCABILITY S TATEMENT

Original: In this paper, we work with four different datasets - Omniglot, mini ImageNet, tiered ImageNet and
Revised: In this paper, we work with four different datasets - Omniglot, mini ImageNet, tiered ImageNet and

Original: Meta-Dataset.
Revised: Meta-Dataset.

Original: Additional details about setting up these datasets is available in Appendix A.1.
Revised: Additional details about setting up these datasets is available in Appendix A.1.

Original: Furthermore, we experiment with six different models - MAML, Reptile, Protonet, Matching Networks,
Revised: Furthermore, we experiment with six different models - MAML, Reptile, Protonet, Matching Networks,

Original: MetaOptNet, and CNAPs.
Revised: MetaOptNet, and CNAPs.

Original: All these models were run after reproducing from their open-source codes.
Revised: All these models were run after reproducing from their open-source codes.

Original: Additional details about setting up these models are available in Appendix A.2.
Revised: Additional details about setting up these models are available in Appendix A.2.

Original: Our source code is made available under the supplementary materials for additional reference.

Original: A A PPENDIX A. D ATASETS
Revised: A A PPENDIX A. D ATASETS

Original: Omniglot, mini ImageNet and tiered ImageNet were extracted using pytorch-meta.
Revised: Omniglot, mini ImageNet and tiered ImageNet were extracted using pytorch-meta.

Original: The Meta-Dataset was downloaded using the setup information from the ofﬁcial repository: https://github. com/google-research/meta-dataset .
Revised: The Meta-Dataset was downloaded using the setup information from the ofﬁcial repository: https://github. com/google-research/meta-dataset .

Original: Omniglot Omniglot is a benchmark dataset proposed by Lake et al.
Revised: Omniglot Omniglot is a benchmark dataset proposed by Lake et al.

Original: (2011) for few-shot image classiﬁcation tasks.
Revised: (2011) for few-shot image classiﬁcation tasks.

Original: Omniglot dataset consists of 20 instances and 1623 characters from 50 different alphabets.
Revised: Omniglot dataset consists of 20 instances and 1623 characters from 50 different alphabets.

Original: We experiment with both 5-way 1-shot and 2-way 1-shot in this work.
Revised: We experiment with both 5-way 1-shot and 2-way 1-shot in this work.

Original: mini ImageNet mini ImageNet is another benchmark dataset proposed by Ravi & Larochelle (2016) for few-shot image classiﬁcation tasks.
Revised: mini ImageNet mini ImageNet is another benchmark dataset proposed by Ravi & Larochelle (2016) for few-shot image classiﬁcation tasks.

Original: The mini ImageNet dataset involves 64 training classes, 12 validation classes, and 24 test classes.
Revised: The mini ImageNet dataset involves 64 training classes, 12 validation classes, and 24 test classes.

Original: We run under the setting 5-way 1-shot for this experiment.
Revised: We run under the setting 5-way 1-shot for this experiment.

Original: tiered ImageNet tiered ImageNet is notorious as a difﬁcult benchmark dataset proposed by Ren et al.
Revised: tiered ImageNet tiered ImageNet is notorious as a difﬁcult benchmark dataset proposed by Ren et al.

Original: The tiered ImageNet dataset is a larger subset of ILSVRC-12 with 608 classes (779, images) grouped into 34 higher-level nodes in the ImageNet human-curated hierarchy.
Revised: The tiered ImageNet dataset is a larger subset of ILSVRC-12 with 608 classes (779, images) grouped into 34 higher-level nodes in the ImageNet human-curated hierarchy.

Original: This set of nodes is partitioned into 20, 6, and 8 disjoint sets of training, validation, and testing nodes, and the corresponding classes form the respective meta-sets.
Revised: This set of nodes is partitioned into 20, 6, and 8 disjoint sets of training, validation, and testing nodes, and the corresponding classes form the respective meta-sets.

Original: As argued in Ren et al.
Revised: As argued in Ren et al.

Original: (2018), this split near the root of the ImageNet hierarchy results in a more challenging yet realistic regime with test classes that are less similar to training classes.
Revised: (2018), this split near the root of the ImageNet hierarchy results in a more challenging yet realistic regime with test classes that are less similar to training classes.

Original: We run under the setting 5-way 1-shot for this experiment.
Revised: We run under the setting 5-way 1-shot for this experiment.

Original: Meta-Dataset Meta-Dataset is notorious as a difﬁcult benchmark dataset proposed by Triantaﬁllou et al.
Revised: Meta-Dataset Meta-Dataset is notorious as a difﬁcult benchmark dataset proposed by Triantaﬁllou et al.

Original: The Meta-Dataset is much larger than any previous benchmark and is comprised of multiple existing datasets.
Revised: The Meta-Dataset is much larger than any previous benchmark and is comprised of multiple existing datasets.

Original: This invites research into how diverse data sources can be exploited by a meta-learner and allows us to evaluate a more challenging generalization problem to new datasets altogether.
Revised: This invites research into how diverse data sources can be exploited by a meta-learner and allows us to evaluate a more challenging generalization problem to new datasets altogether.

Original: Speciﬁcally, Meta-Dataset leverages data from the following 10 datasets: ILSVRC-
Revised: Speciﬁcally, Meta-Dataset leverages data from the following 10 datasets: ILSVRC-

Original: (Russakovsky et al.
Revised: (Russakovsky et al.

Original: , Omniglot (Lake et al.
Revised: , Omniglot (Lake et al.

Original: , Aircraft (Maji et al.
Revised: , Aircraft (Maji et al.

Original: , CUB-2002011 (Wah et al.
Revised: , CUB-2002011 (Wah et al.

Original: Describable Textures (Cimpoi et al.
Revised: Describable Textures (Cimpoi et al.

Original: , Quick Draw (Jongejan et al.
Revised: , Quick Draw (Jongejan et al.

Original: (2016)), Fungi (Schroeder & Cui (2018)), VGG Flower (Nilsback & Zisserman (2008)), Trafﬁc
Revised: (2016)), Fungi (Schroeder & Cui (2018)), VGG Flower (Nilsback & Zisserman (2008)), Trafﬁc

Original: Signs (Houben et al.
Revised: Signs (Houben et al.

Original: and MSCOCO (Lin et al.
Revised: and MSCOCO (Lin et al.

Original: There exist few classes with fewer image samples than 16.
Revised: There exist few classes with fewer image samples than 16.

Original: This becomes an issue since we need 1 for training and 15 for testing.
Revised: This becomes an issue since we need 1 for training and 15 for testing.

Original: We repeat some of the support images for these classes to make up for the lack of examples.
Revised: We repeat some of the support images for these classes to make up for the lack of examples.

Original: Since the number of such classes is minimal, we justify this use as this solution cannot signiﬁcantly increase the model’s performance.
Revised: Since the number of such classes is minimal, we justify this use as this solution cannot signiﬁcantly increase the model’s performance.

Original: Furthermore, unlike previous experiments on this dataset which use a variable number of ways and shots during training, we train with a ﬁxed number of ways and shots.
Revised: Furthermore, unlike previous experiments on this dataset which use a variable number of ways and shots during training, we train with a ﬁxed number of ways and shots.

Original: Furthermore, unlike the works in Triantaﬁllou et al.
Revised: Furthermore, unlike the works in Triantaﬁllou et al.

Original: (2019), we do not perform anytype of pre-training to help aid the model.
Revised: (2019), we do not perform any pre-training to help aid the model.

Original: We believe these are the main attributions for the disparity of our performance.
Revised: We believe these are the main attributions for the disparity of our performance.

Original: However, since we are focused on the relative performance of the samplers for a given model, this discrepancy would not affect our study of task diversity in any manner.
Revised: However, since we are focused on the relative performance of the samplers for a given model, this discrepancy would not affect our study of task diversity in any manner.

Original: We again run under the setting 5-way 1-shot for this experiment.
Revised: We again run under the setting 5-way 1-shot for this experiment.

Original: A. M ODELS
Revised: A. M ODELS

Original: This section describes some of the models we used for our experiments and the hyperparameters used for their training.
Revised: This section describes some of the models we used for our experiments and the hyperparameters used for their training.

Original: A.2. P ROTOTYPICAL N ETWORKS
Revised: A.2. P ROTOTYPICAL N ETWORKS

Original: Prototypical Networks proposed by Snell et al.
Revised: Prototypical Networks proposed by Snell et al.

Original: (2017) constructs a prototype for each class and then classiﬁes each query example as the class whose prototype is ’nearest’ to it under Euclidean distance.
Revised: (2017) constructs a prototype for each class and then classiﬁes each query example as the class whose prototype is ’nearest’ to it under Euclidean distance.

Original: More concretely, the probability that a query example x ∗ belongs to class k is deﬁned as:
Revised: More concretely, the probability that a query example x ∗ belongs to class k is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Where c k is the ’prototype’ for class k : the average embeddings of class k ’s support examples.
Revised: Where c k is the ’prototype’ for class k : the average embeddings of class k ’s support examples.

Original: H YPERPARAMETERS
Revised: H YPERPARAMETERS

Original: In our experiments on Omniglot and mini ImageNet, and tiered ImageNet under a 5-way, 1-shot setting, we run the model for 100 epochs with a batch size of 32 and a metalearning rate of 0.001.
Revised: In our experiments on Omniglot and mini ImageNet, and tiered ImageNet under a 5-way, 1-shot setting, we run the model for 100 epochs with a batch size of 32 and a metalearning rate of 0.001.

Original: We use an Adam optimizer to make gradient steps and a StepLR scheduler with step size 0.4 and gamma 0.5.
Revised: We use an Adam optimizer to make gradient steps and a StepLR scheduler with step size 0.4 and gamma 0.5.

Original: The same hyperparameters are used for training our model on
Revised: The same hyperparameters are used for training our model on

Original: Omniglot under a 20-way 1-shot setting.
Revised: Omniglot under a 20-way 1-shot setting.

Original: We use the same parameters as the mini ImageNet to train our model on the Meta-Dataset under a 5-way 1-shot setting.
Revised: We use the same parameters as the mini ImageNet to train our model on the Meta-Dataset under a 5-way 1-shot setting.

Original: However, we run with a batch size of 16 rather than 32 to accommodate the extensive training period and memory constraints.
Revised: However, we run with a batch size of 16 rather than 32 to accommodate the extensive training period and memory constraints.

Original: A.2. M ATCHING N ETWORKS
Revised: A.2. M ATCHING N ETWORKS

Original: Matching Networks proposed by Vinyals et al.
Revised: Matching Networks proposed by Vinyals et al.

Original: (2016) labels each query example as a cosine distance-weighted linear combination of the support labels:
Revised: (2016) labels each query example as a cosine distance-weighted linear combination of the support labels:

Original: [Equation]
Revised: [Equation]

Original: where Φ A is the indicator function and α ( x ∗ , x i ) is the cosine similarity between g ( x ∗ ) and g ( x i ) , softmax normalized over all support examples x i , where 1 ≤ i ≤ |S| .
Revised: where Φ A is the indicator function and α ( x ∗ , x i ) is the cosine similarity between g ( x ∗ ) and g ( x i ) , softmax normalized over all support examples x i , where 1 ≤ i ≤ |S| .

Original: We had trouble reproducing the results from matching networks using cosine distance since the convergence seemed to be slow and the ﬁnal performance dependent on the random initialization.
Revised: We had trouble reproducing the results from matching networks using cosine distance since the convergence seemed to be slow and the ﬁnal performance dependent on the random initialization.

Original: This is similar to what is observed by other repositories such as https://github.com/ oscarknagg/few-shot .
Revised: This is similar to what is observed by other repositories such as https://github.com/ oscarknagg/few-shot .

Original: Since we are focused on the relative performance of the samplers for a given model, this discrepancy would not affect our study of task diversity in any manner.
Revised: Since we are focused on the relative performance of the samplers for a given model, this discrepancy would not affect our study of task diversity in any manner.

Original: H YPERPARAMETERS
Revised: H YPERPARAMETERS

Original: In our experiments on Omniglot, mini ImageNet, and tiered ImageNet under a 5-way, 1-shot setting, we run the model for 100 epochs with a batch size of 32 and an Adam optimizer with a meta-learning rate of 0.001 and a weight decay of 0.0001.
Revised: In our experiments on Omniglot, mini ImageNet, and tiered ImageNet under a 5-way, 1-shot setting, we run the model for 100 epochs with a batch size of 32 and an Adam optimizer with a meta-learning rate of 0.001 and a weight decay of 0.0001.

Original: The same hyperparameters are used for training our model on Omniglot under a 20-way 1-shot setting.
Revised: The same hyperparameters are used for training our model on Omniglot under a 20-way 1-shot setting.

Original: We use the same parameters as the mini ImageNet to train our model on the Meta-Dataset under a 5-way 1-shot setting.
Revised: We use the same parameters as the mini ImageNet to train our model on the Meta-Dataset under a 5-way 1-shot setting.

Original: However, we run with a batch size of 16 rather than 32 to accommodate the extensive training period and memory constraints.
Revised: However, we run with a batch size of 16 rather than 32 to accommodate the extensive training period and memory constraints.

Original: A.2. MAML
Revised: A.2. MAML

Original: MAML proposed by Finn et al.
Revised: MAML proposed by Finn et al.

Original: (2017) uses a linear layer parametrized by W and b on top of the embedding function g ( . ; θ ) and classiﬁes a query example as:
Revised: (2017) uses a linear layer parametrized by W and b on top of the embedding function g ( . ; θ ) and classiﬁes a query example as:

Original: [Equation]
Revised: [Equation]

Original: where the output layer parameters W’ and b’ and the embedding function parameters θ (cid:48) are obtained by performing a small number of within-episode training steps on the support set S , starting from initial parameter values ( b , W , θ ) .
Revised: where the output layer parameters W’ and b’ and the embedding function parameters θ (cid:48) are obtained by performing a small number of within-episode training steps on the support set S , starting from initial parameter values ( b , W , θ ) .

Original: H YPERPARAMETERS
Revised: H YPERPARAMETERS

Original: In our experiments on Omniglot, mini ImageNet, and tiered ImageNet under a 5-way, 1-shot setting, we run the model for 150 epochs with a batch size of 32, with the Adam optimizer with a meta-learning rate of 0.001, number of inner adaptations as 1, and step size 0.4.
Revised: In our experiments on Omniglot, mini ImageNet, and tiered ImageNet under a 5-way, 1-shot setting, we run the model for 150 epochs with a batch size of 32, with the Adam optimizer with a meta-learning rate of 0.001, number of inner adaptations as 1, and step size 0.4.

Original: For our experiments on Omniglot under the 20-way 1-shot setting, we set the step size of 0.1 and the number of inner adaptations to 5, batch size of 16, and kept all other hyperparameters constant.
Revised: For our experiments on Omniglot under the 20-way 1-shot setting, we set the step size of 0.1 and the number of inner adaptations to 5, batch size of 16, and kept all other hyperparameters constant.

Original: We use the same parameters as the mini ImageNet to train our model on the Meta-Dataset under a 5-way 1-shot setting.
Revised: We use the same parameters as the mini ImageNet to train our model on the Meta-Dataset under a 5-way 1-shot setting.

Original: However, we ran with a batch size of 16 rather than 32 and only trained for 100 epochs to accommodate the extensive training period and memory constraints.
Revised: However, we ran with a batch size of 16 rather than 32 and only trained for 100 epochs to accommodate the extensive training period and memory constraints.

Original: A.2. R EPTILE
Revised: A.2. R EPTILE

Original: Like MAML, Reptile proposed by Nichol et al.
Revised: Like MAML, Reptile proposed by Nichol et al.

Original: (2018) learns an initialization for the parameters of a neural network model, such that when we optimize these parameters at test time, learning is fast - i.e., the model generalizes from a small number of test tasks.
Revised: (2018) learns an initialization for the parameters of a neural network model, such that when we optimize these parameters at test time, learning is fast - i.e., the model generalizes from a small number of test tasks.

Original: Reptile converges towards a solution φ that is close (in Euclidean distance) to each task τ ’s manifold of optimal solutions.
Revised: Reptile converges towards a solution φ that is close (in Euclidean distance) to each task τ ’s manifold of optimal solutions.

Original: Let φ denote the network initialization, and W = φ + ∆ φ denote the network weights after performing some sort of update.
Revised: Let φ denote the network initialization, and W = φ + ∆ φ denote the network weights after performing some sort of update.

Original: Let W ∗ τ denote the set of optimal network weights for task τ .
Revised: Let W ∗ τ denote the set of optimal network weights for task τ .

Original: We want to ﬁnd φ such that the distance D ( φ, W ∗ τ ) is small for all tasks:
Revised: We want to ﬁnd φ such that the distance D ( φ, W ∗ τ ) is small for all tasks:

Original: [Equation]
Revised: [Equation]

Original: The ofﬁcial repository seems to train the model with a 5-way 15-shot and test the model on a 5-way 1-shot.
Revised: The ofﬁcial repository seems to train the model with a 5-way 15-shot and test the model on a 5-way 1-shot.

Original: However, we do not consider this to be an accurate study for the effect of task diversity.
Revised: However, we do not consider this to be an accurate study for the effect of task diversity.

Original: In our work, we train and test the model in a 5-way 1-shot setting to ensure fair and accurate comparison with other models.
Revised: In our work, we train and test the model in a 5-way 1-shot setting to ensure fair and accurate comparison with other models.

Original: We believe this to be the source of discrepancy in our performance scores.
Revised: We believe this to be the source of discrepancy in our performance scores.

Original: Since we are focused on the relative performance of the samplers for a given model, this discrepancy would not affect our study of task diversity in any manner.
Revised: Since we are focused on the relative performance of the samplers for a given model, this discrepancy would not affect our study of task diversity in any manner.

Original: H YPERPARAMETERS
Revised: H YPERPARAMETERS

Original: In our experiments on Omniglot and mini ImageNet under a 5-way, 1-shot setting, we run the model for 150 epochs with a batch size of 32, a learning rate of 0.01, a metalearning rate of 0.001, and a number of inner adaptations as 5.
Revised: In our experiments on Omniglot and mini ImageNet under a 5-way, 1-shot setting, we run the model for 150 epochs with a batch size of 32, a learning rate of 0.01, a metalearning rate of 0.001, and a number of inner adaptations as 5.

Original: We use the SGD optimizer of inner steps and Adam optimizer for the outer steps.
Revised: We use the SGD optimizer of inner steps and Adam optimizer for the outer steps.

Original: For our experiments on tiered ImageNet, we change the number of inner adaptations to 10, keeping all other hyperparameters constant.
Revised: For our experiments on tiered ImageNet, we change the number of inner adaptations to 10, keeping all other hyperparameters constant.

Original: For our experiments on Omniglot under the 20-way 1-shot setting, we set the meta-learning rate to 0.
Revised: For our experiments on Omniglot under the 20-way 1-shot setting, we set the meta-learning rate to 0.

Original: and the number of inner adaptations to 10 and kept all other hyperparameters constant.
Revised: and the number of inner adaptations to 10 and kept all other hyperparameters constant.

Original: Furthermore, we only run the model for 50 epochs due to the very high training time.
Revised: Furthermore, we only run the model for 50 epochs due to the very high training time.

Original: We use the same parameters as the tiered ImageNet to train our model on the Meta-Dataset under a 5-way 1-shot setting.
Revised: We use the same parameters as the tiered ImageNet to train our model on the Meta-Dataset under a 5-way 1-shot setting.

Original: However, we run with a batch size of 16 rather than 32 to accommodate the extensive training period and memory constraints.
Revised: However, we run with a batch size of 16 rather than 32 to accommodate the extensive training period and memory constraints.

Original: A.2. CNAP S
Revised: A.2. CNAP S

Original: Conditional Neural Adaptive Processes proposed by Requeima et al.
Revised: Conditional Neural Adaptive Processes proposed by Requeima et al.

Original: (2019) is able to efﬁciently solve new multi-class classiﬁcation problems after an initial training phase.
Revised: (2019) is able to efﬁciently solve new multi-class classiﬁcation problems after an initial training phase.

Original: The proposed approach, based on Conditional Neural Processes (CNPs) mentioned in Garnelo et al.
Revised: The proposed approach, based on Conditional Neural Processes (CNPs) mentioned in Garnelo et al.

Original: (2018), adapts a small number of task-speciﬁc parameters for each new task encountered at test time.
Revised: (2018), adapts a small number of task-speciﬁc parameters for each new task encountered at test time.

Original: These parameters are conditioned on a set of training examples for the new task.
Revised: These parameters are conditioned on a set of training examples for the new task.

Original: They do not require any additional tuning to adapt both the ﬁnal classiﬁcation layer and feature extraction process.
Revised: They do not require any additional tuning to adapt both the ﬁnal classiﬁcation layer and feature extraction process.

Original: This allows the model to handle various input distributions.
Revised: This allows the model to handle various input distributions.

Original: The CNPs construct predictive distributions given x ∗ as:
Revised: The CNPs construct predictive distributions given x ∗ as:

Original: [Equation]
Revised: [Equation]

Original: where θ are global classifer parameters shared across tasks, Ψ τ are local task-speciﬁc parameters, produced by a function Ψ φ ( . ) that acts of D τ .
Revised: where θ are global classifer parameters shared across tasks, Ψ τ are local task-speciﬁc parameters, produced by a function Ψ φ ( . ) that acts of D τ .

Original: Ψ φ ( . ) has another set of global parameters φ called adaptation network parameters .
Revised: Ψ φ ( . ) has another set of global parameters φ called adaptation network parameters .

Original: θ and φ are the learnable parameters in the model.
Revised: θ and φ are the learnable parameters in the model.

Original: H YPERPARAMETERS
Revised: H YPERPARAMETERS

Original: In all our experiments with CNAPs, we run the model for ten epochs with a batch size of 16 and a meta-learning rate of 0.01.
Revised: In all our experiments with CNAPs, we run the model for ten epochs with a batch size of 16 and a meta-learning rate of 0.01.

Original: A.2. M ETA O PT N ET
Revised: A.2. M ETA O PT N ET

Original: MetaOptNet proposed by Lee et al.
Revised: MetaOptNet proposed by Lee et al.

Original: (2019) proposes a linear classiﬁer as the base learner for a metalearning based approach for few-shot learning.
Revised: (2019) proposes a linear classiﬁer as the base learner for a metalearning based approach for few-shot learning.

Original: The approach uses a linear support vector machine (SVM) to learn a classiﬁer given a set of labeled training examples.
Revised: The approach uses a linear support vector machine (SVM) to learn a classiﬁer given a set of labeled training examples.

Original: The generalization error is computed on a novel set of examples from the same task.
Revised: The generalization error is computed on a novel set of examples from the same task.

Original: The objective is to learn an embedding model φ that minimizes generalization (or test) error across tasks given a base learner A .
Revised: The objective is to learn an embedding model φ that minimizes generalization (or test) error across tasks given a base learner A .

Original: Formally, the learning objective is:
Revised: Formally, the learning objective is:

Original: [Equation]
Revised: [Equation]

Original: The choice of base learner A has a signiﬁcant impact on the above equation.
Revised: The choice of base learner A has a signiﬁcant impact on the above equation.

Original: The base learner that computes θ =
Revised: The base learner that computes θ =

Original: A ( D test ; φ ) has to be efﬁcient since the expectation has to be computed over a distribution of tasks.
Revised: A ( D test ; φ ) has to be efﬁcient since the expectation has to be computed over a distribution of tasks.

Original: This work considers base learners based on multi-class linear classiﬁers such as SVM, where the base learner’s object is convex.
Revised: This work considers base learners based on multi-class linear classiﬁers such as SVM, where the base learner’s object is convex.

Original: Thus, the base learner can be simpliﬁed as:
Revised: Thus, the base learner can be simpliﬁed as:

Original: [Equation]
Revised: [Equation]

Original: where D train = { ( x n , y n ) } , C is the regularization parameter and δ .,.
Revised: where D train = { ( x n , y n ) } , C is the regularization parameter and δ .,.

Original: is the Kronecker delta func- tion.
Revised: is the Kronecker delta func- tion.

Original: Furthermore, the ofﬁcial repository seems to train the model with a 5-way 15-shot and test the model on a 5-way 1-shot.
Revised: Furthermore, the ofﬁcial repository seems to train the model with a 5-way 15-shot and test the model on a 5-way 1-shot.

Original: However, we do not consider this to be an accurate study for the effect of task diversity.
Revised: However, we do not consider this to be an accurate study for the effect of task diversity.

Original: In our work, we train and test the model in a 5-way 1-shot setting to ensure fair and accurate comparison with other models.
Revised: In our work, we train and test the model in a 5-way 1-shot setting to ensure fair and accurate comparison with other models.

Original: We believe this to be the source of discrepancy in our performance scores.
Revised: We believe this to be the source of discrepancy in our performance scores.

Original: Since we are focused on the relative performance of the samplers for a given model, this discrepancy would not affect our study of task diversity in any manner.
Revised: Since we are focused on the relative performance of the samplers for a given model, this discrepancy would not affect our study of task diversity in any manner.

Original: H YPERPARAMETERS
Revised: H YPERPARAMETERS

Original: In our experiments on Omniglot, mini ImageNet and tiered ImageNet under a 5-way, 1-shot setting, we run the model for 60 epochs with a batch size of 32 and a meta-learning rate of 0.01.
Revised: In our experiments on Omniglot, mini ImageNet and tiered ImageNet under a 5-way, 1-shot setting, we run the model for 60 epochs with a batch size of 32 and a meta-learning rate of 0.01.

Original: We use an SGD optimizer with a momentum of 0.9 and a weight decay of 0.0001 to make gradient steps.
Revised: We use an SGD optimizer with a momentum of 0.9 and a weight decay of 0.0001 to make gradient steps.

Original: We also use a LambdaLR scheduler to train our model.
Revised: We also use a LambdaLR scheduler to train our model.

Original: The same hyperparameters are used for training our model on Omniglot under a 20-way 1-shot setting.
Revised: The same hyperparameters are used for training our model on Omniglot under a 20-way 1-shot setting.

Original: We use the same parameters as the mini ImageNet to train our model on the Meta-Dataset under a 5-way 1-shot setting.
Revised: We use the same parameters as the mini ImageNet to train our model on the Meta-Dataset under a 5-way 1-shot setting.

Original: However, we run with a batch size of 16 rather than 32 to accommodate the extensive training period and memory constraints
Revised: However, we run with a batch size of 16 rather than 32 to accommodate the extensive training period and memory constraints.

Original: [Table] A. A DDITIONAL R ESULTS 
Revised: A. A DDITIONAL R ESULTS A.3. R ESULTS ON R EGRESSION

Original: In this section
Revised: We further experiment on the few-shot regression regime to maintain our ﬁndings on classiﬁcation. For this problem, we trained on three few-shot regression datasets: (i) Sinusoid (Finn et al. (2017)), (ii)Sinusoid & Line

Original: [Table]
Revised: (Finn et al. (2018)), and (iii) Harmonic (Lacoste et al. Table 1 presents our results using the mean square error metric on the 5-shot and 10-shot setting for all three datasets.

Original:  we present the results with higher precision from our earlier experiments in a Tableand Table-2.
Revised: Figure 7 and Figure 8 presents our results on the Sinusoid dataset speciﬁcally for easier comparison. For both MAML and Reptile, we use the same network, with two hidden layers, each of size 40. H YPERPARAMETERS For both MAML and Reptile, we train the model for 150 epochs, and use the same hyperparameters as Omniglot.

Original: Subsequently
Revised: Only in the case of MAML, we use a step size of 0.001 for Sinusoid and Sinusoid & Line dataset, instead of the traditional 0.4. Results We again observe that samplers that limit diversity, such as NDB, NDT/B samplers achieve performances similar to the uniform sampler with no adverse effects. Astonishingly, we also notice that the NDT sampler seems to outperform all other samplers in the regression domain.

Original: [Table]
Revised: Furthermore, similar to our previous ﬁndings, samplers that increase diversity, such as the OHTM sampler, do not perform very well in this domain either, as shown in Figure 7 and Figure 8. We cannot train using DPP-oriented samplers for regression since the data is continuous, and computing features or representations for the entire range would be theoretically impossible. A.3. R ESULTS ON C LASSIFICATION In this section, we present the results with higher precision from our earlier experiments in a Tableand Table 3.

Original:  we also plot convergence curves to aid better visualizations of ﬁndings mentioned earlier in Figure-7 and Figure-8.
Revised: Subsequently, we also plot convergence curves to aid better visualizations of ﬁndings mentioned earlier in Figure 10 and Figure 11.

Original: We also present the results of our models on theMetaDataset in Table-
Revised: We also present the results of our models on the

Original: [Table] [Table] [Table]
Revised: Meta-Dataset in Table 4.

Original: Statistical Results We compare the performance of different models to the Uniform Sampler.
Revised: Statistical Results We compare the performance of different models to the Uniform Sampler.

Original: All samplers are poorer than the Uniform Sampler and are statistically signiﬁcant with a conﬁdence interval of 95%.
Revised: All samplers are poorer than the Uniform Sampler and are statistically signiﬁcant with a conﬁdence interval of 95%.

Original: We use the symbol † to represent the instances where the results are not statistically  signiﬁcant and similar to the performance achieved by uniform Sampler.
Revised: We use the symbol † to represent the instances where the results are not statistically [Table] [Figure] signiﬁcant and similar to the performance achieved by uniform Sampler.

Original: We only observe  few instances where a sampler performs signiﬁcantly better than the Uniform Sampler, which we represent using the symbol ‡. To assess statistical signiﬁcance, we used a paired-difference t-test, with a p-value p = 0 . 
Revised: We only observe a few cases where a sampler performs signiﬁcantly better than the Uniform Sampler, which we represent using the symbol ‡. To assess statistical signiﬁcance, we used a paired-difference t-test, with a pvalue p = 0 . Comparisons with SBU Sampler

Revised: Previously, we compared the performance of NDTB and SBU samplers.

Revised: However, due to the possibility of unfair comparisons, limited by the number of iterations of the SBU sampler, we extend the SBU Sampler to propose two new samplers: (1) SBUunbounded sampler - This sampler is similar to the SBU sampler but is run for more iterations.

Revised: However, this sampler allows the model to be trained on more classes than our traditional SBU sampler.

Revised: Since this remains an unfair comparison, we use this as a higher bound/ideal performance and propose a more appropriate sampler.

Original:  [Figure]
Revised: (2) SBU-bounded sampler - This sampler is again similar to the SBU sampler but ran for more iterations with a bounded pool of tasks, similar to the NDTB [Figure] ) Uniform    4 1. [Figure]

Revised: Diversity Task     1.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: sampler or the SBU sampler.

Revised: Our results from these experiments are presented in Figure 9.

Revised: Tableshows the same results with higher precision.

Revised: We notice that training for more iterations using samplers such as SBU-unbounded and SBUbounded leads to a boost in performance.

Revised: However, we also observe that the NDTB sampler remains to perform better in most cases.

Revised: With this, we maintain our previous ﬁndings.

Revised: Furthermore, we also notice a peculiar behavior where the SBU-bounded sampler performs better than the SBUunbounded sampler in most cases.

Revised: This ﬁnding is interesting because the SBU-unbounded sampler has access to more data, and the model being trained should lead to better representations and performance.

Revised: However, we notice that repeating the tasks and ﬁxing the number of classes achieves the same performance with little to no adverse effects

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: MAML

Revised: Reptile

Revised: Protonet

Revised: Matching Net.

Revised: MetaOptNet

Revised: CNAPs

Original: [Figure] 
Revised: [Figure] H H H

Revised: H

Revised: H H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: [Figure]

Revised: cc u r a c y

Revised: Omniglot 5-way 1-shot

Revised: No Diversity Tasks/Batch

Revised: [Figure]

Revised: Single-unbounded

Revised: Single-bounded

Revised: MAML

Revised: Reptile

Revised: Protonet

Revised: Matching Net.

Revised: MetaOptNet

Revised: CNAPs

Revised: [Figure]

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H

Revised: [Figure]

Revised: mini ImageNet 5-way 1-shot

Revised: [Figure]

Revised: Reptile

Revised: Protonet

Revised: Matching Net.

Revised: MetaOptNet

Revised: CNAPs

Revised: [Figure]

Revised: H

Revised: H

Revised: N

Revised: H

Revised: [Figure]

Revised: H

Revised: H

Revised: H

Revised: H

Revised: H H

Revised: H

Revised: N

Revised: N

Revised: N

Revised: H

Revised: H

Revised: H

Revised: N

Revised: N

Revised: N

Revised: H

Revised: [Figure]

Revised: tiered ImageNet 5-way 1-

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]