Original: O N H ARD E PISODES IN META -L EARNING
Revised: O N H ARD E PISODES IN META -L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Existing meta-learners primarily focus on improving the average task accuracy across multiple episodes.
Revised: Existing meta-learners primarily focus on improving the average task accuracy across multiple episodes.

Original: Different episodes, however, may vary in hardness and quality leading to a wide gap in the meta-learner’s performance across episodes.
Revised: Different episodes, however, may vary in hardness and quality leading to a wide gap in the meta-learner’s performance across episodes.

Original: Understanding this issue is particularly critical in industrial few-shot settings, where there is limited control over test episodes as they are typically uploaded by end-users.
Revised: Understanding this issue is particularly critical in industrial few-shot settings, where there is limited control over test episodes as they are typically uploaded by end-users.

Original: In this paper, we empirically analyse the behaviour of meta-learners on episodes of varying hardness across three standard benchmark datasets: CIFARFS, mini-ImageNet, and tiered-ImageNet.
Revised: In this paper, we empirically analyse the behaviour of meta-learners on episodes of varying hardness across three standard benchmark datasets: CIFARFS, mini-ImageNet, and tiered-ImageNet.

Original: Surprisingly, we observe a wide gap in accuracy of around 50% between the hardest and easiest episodes across all the standard benchmarks and meta-learners.
Revised: Surprisingly, we observe a wide gap in accuracy of around 50% between the hardest and easiest episodes across all the standard benchmarks and meta-learners.

Original: We additionally investigate various properties of hard episodes and highlight their connection to catastrophic forgetting during meta-training.
Revised: We additionally investigate various properties of hard episodes and highlight their connection to catastrophic forgetting during meta-training.

Original: To address the issue of sub-par performance on hard episodes, we investigate and benchmark different meta-training strategies based on adversarial training and curriculum learning.
Revised: To address the issue of sub-par performance on hard episodes, we investigate and benchmark different meta-training strategies based on adversarial training and curriculum learning.

Original: We ﬁnd that adversarial training strategies are much more powerful than curriculum learning in improving the prediction performance on hard episodes.
Revised: We ﬁnd that adversarial training strategies are much more powerful than curriculum learning in improving the prediction performance on hard episodes.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Humans have a remarkable ability to learn new concepts from very few examples and generalize effectively to unseen tasks.
Revised: Humans have a remarkable ability to learn new concepts from very few examples and generalize effectively to unseen tasks.

Original: However, standard deep learning approaches still lag behind human capabilities in learning from few examples.
Revised: However, standard deep learning approaches still lag behind human capabilities in learning from few examples.

Original: For large over-parameterized deep models, learning with general supervision from only a few examples leads to over-ﬁtting and thus poor generalization.
Revised: For large over-parameterized deep models, learning with general supervision from only a few examples leads to over-ﬁtting and thus poor generalization.

Original: To circumvent this, the paradigm of few-shot learning (Wang et al., 2020; Fei-fei et al., 2006;
Revised: To circumvent this, the paradigm of few-shot learning (Wang et al., 2020; Fei-fei et al., 2006;

Original: Vinyals et al., 2017) aims to effectively learn new concepts from very few labeled examples.
Revised: Vinyals et al., 2017) aims to effectively learn new concepts from very few labeled examples.

Original: These learned concepts can generalize well to future unseen learning tasks.
Revised: These learned concepts can generalize well to future unseen learning tasks.

Original: Several frameworks have been proposed for tackling the few-shot learning scenario: transfer-learning (Dhillon et al., 2019), selftraining (Phoo & Hariharan, 2020) and meta-learning (Hospedales et al., 2020; Finn et al., 2017;
Revised: Several frameworks have been proposed for tackling the few-shot learning scenario: transfer-learning (Dhillon et al., 2019), selftraining (Phoo & Hariharan, 2020) and meta-learning (Hospedales et al., 2020; Finn et al., 2017;

Original: Snell et al., 2017).
Revised: Snell et al., 2017).

Original: Meta-learning in particular aims to learn the process of learning from few examples and has shown remarkable performance across various few-shot benchmarks (Hospedales et al., 2020).
Revised: Meta-learning in particular aims to learn the process of learning from few examples and has shown remarkable performance across various few-shot benchmarks (Hospedales et al., 2020).

Original: In meta-learning, several few-shot tasks (episodes) are sampled from a set of base classes and the underlying model is trained to perform well on these tasks leading to improved generalization in learning from only few examples belonging to novel and unseen classes.
Revised: In meta-learning, several few-shot tasks (episodes) are sampled from a set of base classes and the underlying model is trained to perform well on these tasks leading to improved generalization in learning from only few examples belonging to novel and unseen classes.

Original: Existing meta-learners such as prototypical networks (Snell et al., 2017), MAML (Finn et al., 2017),
Revised: Existing meta-learners such as prototypical networks (Snell et al., 2017), MAML (Finn et al., 2017),

Original: MetaOptNet (Lee et al., 2019), and R2D2 (Bertinetto et al., 2018) primarily focus on improving prediction performance on average across multiple episodes.
Revised: MetaOptNet (Lee et al., 2019), and R2D2 (Bertinetto et al., 2018) primarily focus on improving prediction performance on average across multiple episodes.

Original: However, different episodes have distinct characteristics and hardness which might lead to a wide variance in prediction accuracy across episodes.
Revised: However, different episodes have distinct characteristics and hardness which might lead to a wide variance in prediction accuracy across episodes.

Original: This problem is much more prevalent in few-shot models deployed in the industry.
Revised: This problem is much more prevalent in few-shot models deployed in the industry.

Original: For example, meta-trained models are often deployed in the cloud for the end-users to use for various tasks such as object recognition, detection, semantic segmentation in computer vision and natural language understanding in NLP.
Revised: For example, meta-trained models are often deployed in the cloud for the end-users to use for various tasks such as object recognition, detection, semantic segmentation in computer vision and natural language understanding in NLP.

Original: In such settings, the end-users upload their own few-shot dataset to perform predictions on new and unseen examples belonging to novel classes.
Revised: In such settings, the end-users upload their own few-shot dataset to perform predictions on new and unseen examples belonging to novel classes.

Original: In practice, different users may upload few-shot datasets of varying quality and hardness, leading to a wide disparity in performance across different users.
Revised: In practice, different users may upload few-shot datasets of varying quality and hardness, leading to a wide disparity in performance across different users.

Original: To draw a parallel to the widely accepted experimental protocols in meta-learning, each of the uploaded few-shot dataset and the corresponding unseen examples is equivalent to a test episode.
Revised: To draw a parallel to the widely accepted experimental protocols in meta-learning, each of the uploaded few-shot dataset and the corresponding unseen examples is equivalent to a test episode.

Original: In this paper, we study this issue and investigate how existing state-of-the-art meta-learners (Snell et al., 2017; Bertinetto et al., 2018; Lee et al., 2019) perform on episodes of varying hardness.
Revised: In this paper, we study this issue and investigate how existing state-of-the-art meta-learners (Snell et al., 2017; Bertinetto et al., 2018; Lee et al., 2019) perform on episodes of varying hardness.

Original: Across three benchmark datasets: CIFAR-FS, mini-ImageNet, and tieredImageNet, we observe that there is a gap of ≈ 50% in prediction accuracy between the easiest and hardest episodes.
Revised: Across three benchmark datasets: CIFAR-FS, mini-ImageNet, and tieredImageNet, we observe that there is a gap of ≈ 50% in prediction accuracy between the easiest and hardest episodes.

Original: To this end, we identify several intriguing properties of hard episodes in meta-learning.
Revised: To this end, we identify several intriguing properties of hard episodes in meta-learning.

Original: For instance, we ﬁnd that hard episodes are forgotten more easily than easy episodes during meta-training.
Revised: For instance, we ﬁnd that hard episodes are forgotten more easily than easy episodes during meta-training.

Original: Episode forgetting occurs when the underlying meta-learner forgets acquired knowledge by the end of the meta-training.
Revised: Episode forgetting occurs when the underlying meta-learner forgets acquired knowledge by the end of the meta-training.

Original: To improve prediction performance on hard episodes, we investigate and benchmark various adversarial training and curriculum learning strategies that can be used jointly with any existing meta-learner.
Revised: To improve prediction performance on hard episodes, we investigate and benchmark various adversarial training and curriculum learning strategies that can be used jointly with any existing meta-learner.

Original: Empirically, we ﬁnd that adversarial training strategies are much more powerful than curriculum learning in improving the prediction performance on hard episodes.
Revised: Empirically, we ﬁnd that adversarial training strategies are much more powerful than curriculum learning in improving the prediction performance on hard episodes.

Original: The aim of our paper is not to chase another state-of-the-art in meta-learning, but to perform a ﬁne-grained inspection of hard episodes across various meta-learning methods.
Revised: The aim of our paper is not to chase another state-of-the-art in meta-learning, but to perform a ﬁne-grained inspection of hard episodes across various meta-learning methods.

Original: In summary, we make the following contributions:
Revised: In summary, we make the following contributions:

Original: • We present a detailed analysis of episode hardness in meta-learning across few-shot benchmarks and state-of-the-art meta-learners.
Revised: • We present a detailed analysis of episode hardness in meta-learning across few-shot benchmarks and state-of-the-art meta-learners.

Original: In particular, we study various properties (e
Revised: In particular, we study various properties (e

Original: g., semantic characteristics, forgetting) of episode hardness across different meta-learners and architectures.
Revised: g., semantic characteristics, forgetting) of episode hardness across different meta-learners and architectures.

Original: • We ﬁnd strong connections between episode hardness and catastrophic forgetting in metalearning.
Revised: • We ﬁnd strong connections between episode hardness and catastrophic forgetting in metalearning.

Original: While catastrophic forgetting can occur when meta-training with multiple datasets in sequence (Yap et al., 2020), we observe that forgetting events can occur even when the tasks during meta-training are drawn from a single dataset.
Revised: While catastrophic forgetting can occur when meta-training with multiple datasets in sequence (Yap et al., 2020), we observe that forgetting events can occur even when the tasks during meta-training are drawn from a single dataset.

Original: In particular, we ﬁnd that hard episodes are easy to forget, while easy episodes are difﬁcult to forget.
Revised: In particular, we ﬁnd that hard episodes are easy to forget, while easy episodes are difﬁcult to forget.

Original: • Based on our analysis, we investigate and benchmark different adversarial training and curriculum training strategies to augment general purpose meta-training for improving predic-
Revised: • Based on our analysis, we investigate and benchmark different adversarial training and curriculum training strategies to augment general purpose meta-training for improving predic-

Original: tion performance on hard episodes.
Revised: tion performance on hard episodes.

Original: Empirically, we ﬁnd that although there is no one-sizeﬁts-all solution, adversarial meta-training strategies are more powerful when compared to curriculum learning strategies.
Revised: Empirically, we ﬁnd that although there is no one-sizeﬁts-all solution, adversarial meta-training strategies are more powerful when compared to curriculum learning strategies.

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original: Meta-learning aims to learn an underlying model that can generalize and adapt well to examples from unseen classes by the process of learning to learn.
Revised: Meta-learning aims to learn an underlying model that can generalize and adapt well to examples from unseen classes by the process of learning to learn.

Original: This is primarily achieved by mimicking the evaluation and adaptation procedure during meta-training.
Revised: This is primarily achieved by mimicking the evaluation and adaptation procedure during meta-training.

Original: In general, there are three types of metalearners: (a) Memory-based methods (Ravi & Larochelle, 2017; Munkhdalai et al., 2018;
Revised: In general, there are three types of metalearners: (a) Memory-based methods (Ravi & Larochelle, 2017; Munkhdalai et al., 2018;

Original: Santoro et al., 2016) adapt to novel classes with a memory attached to the meta-learner; (b) Metric-learning based methods (Snell et al., 2017; Sung et al., 2017) aim to learn transferable deep representations which can adapt to unseen classes without any additional ﬁne-tuning; (c) Optimization based methods (Finn et al., 2017; Lee et al., 2019; Bertinetto et al., 2018) learn a good pre-training initialization for effective transfer to unseen tasks with only a few optimization steps.
Revised: Santoro et al., 2016) adapt to novel classes with a memory attached to the meta-learner; (b) Metric-learning based methods (Snell et al., 2017; Sung et al., 2017) aim to learn transferable deep representations which can adapt to unseen classes without any additional ﬁne-tuning; (c) Optimization based methods (Finn et al., 2017; Lee et al., 2019; Bertinetto et al., 2018) learn a good pre-training initialization for effective transfer to unseen tasks with only a few optimization steps.

Original: Although the primary focus of our work is meta-learning, we note that other few-shot learning paradigms such as transfer learning (Chen et al., 2021; Sun et al., 2019; Dhillon et al., 2020) have also shown competitive performance with meta-learning.
Revised: Although the primary focus of our work is meta-learning, we note that other few-shot learning paradigms such as transfer learning (Chen et al., 2021; Sun et al., 2019; Dhillon et al., 2020) have also shown competitive performance with meta-learning.

Original: While there has been a signiﬁcant progress in improving the state-of-the-art in meta-learning, very few work investigates the effectiveness of existing meta-learning approaches on episodes of varying hardness.
Revised: While there has been a signiﬁcant progress in improving the state-of-the-art in meta-learning, very few work investigates the effectiveness of existing meta-learning approaches on episodes of varying hardness.

Original: A recent and concurrent work by Arnold et al.
Revised: A recent and concurrent work by Arnold et al.

Original: (2021) discusses episode difﬁculty and the impact of random episodic sampling during meta-training.
Revised: (2021) discusses episode difﬁculty and the impact of random episodic sampling during meta-training.

Original: Based on their analysis, Arnold et al.
Revised: Based on their analysis, Arnold et al.

Original: (2021) propose a re-weighted optimization framework for meta-training based on importance sampling.
Revised: (2021) propose a re-weighted optimization framework for meta-training based on importance sampling.

Original: Although our paper and Arnold et al.
Revised: Although our paper and Arnold et al.

Original: (2021) tackle similar problems of episodic hardness, there are several points which distinguishes our work:
Revised: (2021) tackle similar problems of episodic hardness, there are several points which distinguishes our work:

Original: • We provide a much more ﬁne-grained analysis of episode hardness than Arnold et al.
Revised: • We provide a much more ﬁne-grained analysis of episode hardness than Arnold et al.

Original: Arnold et al.
Revised: Arnold et al.

Original: (2021) primarily discuss the transferability of episodes across different meta-learners, while we ﬁnd and investigate a strong connection between episode hardness and catastrophic forgetting.
Revised: (2021) primarily discuss the transferability of episodes across different meta-learners, while we ﬁnd and investigate a strong connection between episode hardness and catastrophic forgetting.

Original: • Arnold et al.
Revised: • Arnold et al.

Original: (2021) propose a loss re-weighting framework for improving the average accuracy across episodes.
Revised: (2021) propose a loss re-weighting framework for improving the average accuracy across episodes.

Original: In contrary, we investigate the effectiveness of adversarial training (Gong et al., 2020) and
Revised: In contrary, we investigate the effectiveness of adversarial training (Gong et al., 2020) and

Original: general curriculum learning techniques in improving the average as well as worst-case prediction performance in meta-learning.
Revised: general curriculum learning techniques in improving the average as well as worst-case prediction performance in meta-learning.

Original: Adversarial meta-learning techniques have previously been used in conjunction with dataaugmentation (Ni et al., 2021) to select the augmentation type resulting in the worst-case loss among different augmentation techniques.
Revised: Adversarial meta-learning techniques have previously been used in conjunction with dataaugmentation (Ni et al., 2021) to select the augmentation type resulting in the worst-case loss among different augmentation techniques.

Original: In this paper, we focus on how such strategies can be useful in improving the prediction performance of the hard episodes in addition to the average accuracy.
Revised: In this paper, we focus on how such strategies can be useful in improving the prediction performance of the hard episodes in addition to the average accuracy.

Original:  R ETHINKING E PISODIC A CCURACY
Revised:  R ETHINKING E PISODIC A CCURACY

Original: Existing state-of-the-art meta-learners (Finn et al., 2017; Lee et al., 2019;
Revised: Existing state-of-the-art meta-learners (Finn et al., 2017; Lee et al., 2019;

Original: Snell et al., 2017;
Revised: Snell et al., 2017;

Original: Bertinetto et al., 2018) primarily focus on optimizing for the average loss across multiple training episodes or tasks.
Revised: Bertinetto et al., 2018) primarily focus on optimizing for the average loss across multiple training episodes or tasks.

Original: However, solely the average performance in isolation does not give enough insights into how meta-learners perform on episodes of varying quality and hardness.
Revised: However, solely the average performance in isolation does not give enough insights into how meta-learners perform on episodes of varying quality and hardness.

Original: Such insights can be particularly crucial to investigate and debug meta-learning models deployed in the wild, where the model can encounter diverse test episodes.
Revised: Such insights can be particularly crucial to investigate and debug meta-learning models deployed in the wild, where the model can encounter diverse test episodes.

Original: In this section, we go beyond the average accuracy across different test episodes and evaluate meta-learners on episodes of varying hardness.
Revised: In this section, we go beyond the average accuracy across different test episodes and evaluate meta-learners on episodes of varying hardness.

Original: First, we discuss how to quantify the hardness of an episode and then discuss the performance of meta-learners on hard episodes.
Revised: First, we discuss how to quantify the hardness of an episode and then discuss the performance of meta-learners on hard episodes.

Original: 3. W HAT IS A GOOD MEASURE OF EPISODE HARDNESS ?
Revised: 3. W HAT IS A GOOD MEASURE OF EPISODE HARDNESS ?

Original: Episodic sampling (i.e. sampling various few-shot tasks from a base dataset) in meta-learning takes place in two steps: (i)
Revised: Episodic sampling (i.e. sampling various few-shot tasks from a base dataset) in meta-learning takes place in two steps: (i)

Original: First the episode classes are sampled from the class distribution of the base classes : c ∼ p ( C base ) ; (ii) Next, an episode τ is sampled from the data distribution conditioned on the set of sampled classes c : τ ∼ p ( D| c ) , where D is the base dataset.
Revised: First the episode classes are sampled from the class distribution of the base classes : c ∼ p ( C base ) ; (ii) Next, an episode τ is sampled from the data distribution conditioned on the set of sampled classes c : τ ∼ p ( D| c ) , where D is the base dataset.

Original: An episode τ consists of a set of support examples τ s and query examples τ q .
Revised: An episode τ consists of a set of support examples τ s and query examples τ q .

Original: In few-shot learning, a n -way, k -shot episode is sampled which results in sampling n classes and k support examples per class.
Revised: In few-shot learning, a n -way, k -shot episode is sampled which results in sampling n classes and k support examples per class.

Original: Based on this, the meta-learning optimization objective can be generalized as the following:
Revised: Based on this, the meta-learning optimization objective can be generalized as the following:

Original: [Equation]
Revised: [Equation]

Original: where F is the base architecture with θ as the model parameters and θ (cid:48)
Revised: where F is the base architecture with θ as the model parameters and θ (cid:48)

Original: =
Revised: =

Original: A ( θ, τ s ) is the ﬁnetuning step with the support examples.
Revised: A ( θ, τ s ) is the ﬁnetuning step with the support examples.

Original: Different meta-learners have different types of ﬁne-tuning procedures and we direct the readers to (Finn et al., 2017; Snell et al., 2017; Bertinetto et al., 2018) for more information on the characteristics of A .
Revised: Different meta-learners have different types of ﬁne-tuning procedures and we direct the readers to (Finn et al., 2017; Snell et al., 2017; Bertinetto et al., 2018) for more information on the characteristics of A .

Original: Based on this deﬁnition, we deﬁne the hardness of an episode H ( τ ) in terms of the loss incurred on the query examples in an episode:
Revised: Based on this deﬁnition, we deﬁne the hardness of an episode H ( τ ) in terms of the loss incurred on the query examples in an episode:

Original: [Equation]
Revised: [Equation]

Original: We choose query loss as a metric for hardness because of its inherent simplicity in computation as well as interpretation.
Revised: We choose query loss as a metric for hardness because of its inherent simplicity in computation as well as interpretation.

Original: In addition, we ﬁnd a strong negative correlation between the episodic loss and the accuracy ( ≈ − 0 .
Revised: In addition, we ﬁnd a strong negative correlation between the episodic loss and the accuracy ( ≈ − 0 .

Original: 92 for mini-ImageNet and ≈ − 0 .
Revised: 92 for mini-ImageNet and ≈ − 0 .

Original: 89 for tieredImageNet with prototypical networks).
Revised: 89 for tieredImageNet with prototypical networks).

Original: This is true for other meta-learners such as R2D2 too (See Appendix A for more details).
Revised: This is true for other meta-learners such as R2D2 too (See Appendix A for more details).

Original: Alternatively, hardness of an episode can also be deﬁned as the average log-odds of the query example (Dhillon et al., 2020).
Revised: Alternatively, hardness of an episode can also be deﬁned as the average log-odds of the query example (Dhillon et al., 2020).

Original: 3. P ERFORMANCE OF META - LEARNERS ON HARD EPISODES
Revised: 3. P ERFORMANCE OF META - LEARNERS ON HARD EPISODES

Original: To understand the effectiveness of meta-learners on episodes of varying hardness, we ﬁrst order the test episodes in decreasing order of their hardness.
Revised: To understand the effectiveness of meta-learners on episodes of varying hardness, we ﬁrst order the test episodes in decreasing order of their hardness.

Original: Then, we evaluate different meta-learners on the easiest and the hardest test episode.
Revised: Then, we evaluate different meta-learners on the easiest and the hardest test episode.

Original: Across all the few-shot benchmark datasets such as miniImageNet, CIFAR-FS and tieredImageNet, we ﬁnd in Fig.
Revised: Across all the few-shot benchmark datasets such as miniImageNet, CIFAR-FS and tieredImageNet, we ﬁnd in Fig.

Original: (1) that there is a gap in accuracy of ≈ 50% between the episodes with the highest and the lowest loss.
Revised: (1) that there is a gap in accuracy of ≈ 50% between the episodes with the highest and the lowest loss.

Original: Furthermore, we ﬁnd that the meta-learner and architecture which performs well on average, does not necessarily perform well on the hardest episode.
Revised: Furthermore, we ﬁnd that the meta-learner and architecture which performs well on average, does not necessarily perform well on the hardest episode.

Original: For example, in the case of mini-ImageNet, prototypical networks with a stronger architecture such as ResNet performs better than the 4-layered convolutional architecture on an average, but not on the hard episodes.
Revised: For example, in the case of mini-ImageNet, prototypical networks with a stronger architecture such as ResNet performs better than the 4-layered convolutional architecture on an average, but not on the hard episodes.

Original: Moreover in Fig.
Revised: Moreover in Fig.

Original: (1), we notice that for tieredImageNet, prototypical networks with ResNet performs the best on easy episodes, while R2D2 with ResNet performs slightly better on the hard episodes.
Revised: (1), we notice that for tieredImageNet, prototypical networks with ResNet performs the best on easy episodes, while R2D2 with ResNet performs slightly better on the hard episodes.

Original: The wide gap in accuracy between hard and easy episodes can be magniﬁed for meta-learners deployed in the wild, where the model might encounter episodes which are signiﬁcantly hard and diverse in nature.
Revised: The wide gap in accuracy between hard and easy episodes can be magniﬁed for meta-learners deployed in the wild, where the model might encounter episodes which are signiﬁcantly hard and diverse in nature.

Original: Going forward, we believe that metalearning studies should not only report average episodic accuracy, but also the prediction accuracy on easy and hard episodes, to present a complete picture of the effectiveness of the meta-learner.
Revised: Going forward, we believe that metalearning studies should not only report average episodic accuracy, but also the prediction accuracy on easy and hard episodes, to present a complete picture of the effectiveness of the meta-learner.

Original:  V ISUAL S EMANTICS OF H ARD E PISODES
Revised:  V ISUAL S EMANTICS OF H ARD E PISODES

Original: Based on the observed disparity in performance between hard and easy episodes, a natural question arises: what causes certain episodes to incur high loss?
Revised: Based on the observed disparity in performance between hard and easy episodes, a natural question arises: what causes certain episodes to incur high loss?

Original: While quantitatively, the hardness of an episode can be deﬁned in terms of the loss incurred on the query examples, it does not provide salient insights into the qualitative nature of hard episodes.
Revised: While quantitatively, the hardness of an episode can be deﬁned in terms of the loss incurred on the query examples, it does not provide salient insights into the qualitative nature of hard episodes.

Original: In general, we ﬁnd that episodes incur high loss when there is a mismatch in the semantic characteristics between the support and query examples.
Revised: In general, we ﬁnd that episodes incur high loss when there is a mismatch in the semantic characteristics between the support and query examples.

Original: For example, when a majority of the support examples have objects of only one category in the frame and the query examples have multiple objects of different categories surrounding the primary object of interest, the underlying meta-learner often leads to a wrong prediction.
Revised: For example, when a majority of the support examples have objects of only one category in the frame and the query examples have multiple objects of different categories surrounding the primary object of interest, the underlying meta-learner often leads to a wrong prediction.

Original: Furthermore, when the shape of the objects in the query examples is slightly different from the objects in the support examples, the prediction is often erroneous.
Revised: Furthermore, when the shape of the objects in the query examples is slightly different from the objects in the support examples, the prediction is often erroneous.

Original: In Fig.
Revised: In Fig.

Original: (2)-(a)
Revised: (2)-(a)

Original: , we notice that the query examples that have different objects (e.g., humans) along with the primary object (i.e., guitar) are often misclassiﬁed.
Revised: , we notice that the query examples that have different objects (e.g., humans) along with the primary object (i.e., guitar) are often misclassiﬁed.

Original: In Fig.
Revised: In Fig.

Original: (2)-(b), where most of the query examples are classiﬁed correctly, we ﬁnd that the misclassiﬁed examples are of two types: (i) query images in which the primary object is occluded with a secondary object; (ii) the shape of the object in the query example is different from the object shapes in the support examples.
Revised: (2)-(b), where most of the query examples are classiﬁed correctly, we ﬁnd that the misclassiﬁed examples are of two types: (i) query images in which the primary object is occluded with a secondary object; (ii) the shape of the object in the query example is different from the object shapes in the support examples.

Original: We provide additional examples of hard episodes in Appendix E.
Revised: We provide additional examples of hard episodes in Appendix E.

Original:  H ARD E PISODES S UFFER FROM F ORGETTING
Revised:  H ARD E PISODES S UFFER FROM F ORGETTING

Original: In supervised learning, catastrophic forgetting is prevalent when tasks from different distributions are learned sequentially (Kirkpatrick et al., 2016).
Revised: In supervised learning, catastrophic forgetting is prevalent when tasks from different distributions are learned sequentially (Kirkpatrick et al., 2016).

Original: In such cases, old tasks are forgotten as the model encounters and learns from new tasks.
Revised: In such cases, old tasks are forgotten as the model encounters and learns from new tasks.

Original: Toneva et al.
Revised: Toneva et al.

Original: (2019) has shown that certain examples can be forgotten with high frequency during the course of supervised training even when the samples are drawn from a single dataset.
Revised: (2019) has shown that certain examples can be forgotten with high frequency during the course of supervised training even when the samples are drawn from a single dataset.

Original: In meta-learning, Yap et al.
Revised: In meta-learning, Yap et al.

Original: (2020) has shown that in meta-training with tasks from multiple task distributions sequentially, tasks from the old distribution in the sequence can be forgotten as the meta-learner encounters new tasks.
Revised: (2020) has shown that in meta-training with tasks from multiple task distributions sequentially, tasks from the old distribution in the sequence can be forgotten as the meta-learner encounters new tasks.

Original: However, we observe that even in the case of meta-training with tasks drawn from a single task distribution, certain types of tasks (episodes) can be forgotten during the course of training.
Revised: However, we observe that even in the case of meta-training with tasks drawn from a single task distribution, certain types of tasks (episodes) can be forgotten during the course of training.

Original: In particular, we analyze the connection between the hardness of episodes and catastrophic forgetting in meta-learning.
Revised: In particular, we analyze the connection between the hardness of episodes and catastrophic forgetting in meta-learning.

Original: We track the behaviour of easy and hard training episodes during meta-training and in summary ﬁnd that:
Revised: We track the behaviour of easy and hard training episodes during meta-training and in summary ﬁnd that:

Original: (i) For hard episodes, we notice that the ﬁnal accuracy at the end of the training drops signiﬁcantly ( ≈ 15% in some cases) from the maximum accuracy obtained during the course of meta-training.
Revised: (i) For hard episodes, we notice that the ﬁnal accuracy at the end of the training drops signiﬁcantly ( ≈ 15% in some cases) from the maximum accuracy obtained during the course of meta-training.

Original: (ii) Hard episodes have more number of forgetting events in comparison to the easy episodes during the course of meta-training.
Revised: (ii) Hard episodes have more number of forgetting events in comparison to the easy episodes during the course of meta-training.

Original: This behaviour is more pronounced in the later stages of meta-training where the accuracies of the easy episodes have already stabilized
Revised: This behaviour is more pronounced in the later stages of meta-training where the accuracies of the easy episodes have already stabilized

Original: 5. D EFINING F ORGETTING E VENTS IN META LEARNING
Revised: 5. D EFINING F ORGETTING E VENTS IN META LEARNING

Original: During the course of meta-training, the set of sampled training tasks are different in each epoch.
Revised: During the course of meta-training, the set of sampled training tasks are different in each epoch.

Original: In order to track forgetting events during meta-training, we ﬁrst randomly select a set of k episodes ( E = { τ } ki =1 ) and track their accuracy, throughout the course of meta-training.
Revised: In order to track forgetting events during meta-training, we ﬁrst randomly select a set of k episodes ( E = { τ } ki =1 ) and track their accuracy, throughout the course of meta-training.

Original: In our experiments, we set k = 160 .
Revised: In our experiments, we set k = 160 .

Original: We primarily deﬁne two types of forgetting events: (i) Local forgetting event; (ii)
Revised: We primarily deﬁne two types of forgetting events: (i) Local forgetting event; (ii)

Original: Global forgetting event.
Revised: Global forgetting event.

Original: Global forgetting events .
Revised: Global forgetting events .

Original: For a given episode, a global forgetting event is encountered if the accuracy of the episode at the end of meta-training is less than the maximum accuracy reached during the course of training by a particular threshold.
Revised: For a given episode, a global forgetting event is encountered if the accuracy of the episode at the end of meta-training is less than the maximum accuracy reached during the course of training by a particular threshold.

Original: Formally, given an episode τ with the maximum accuracy acc max ( τ ) = max j acc j ( τ ) , a global forgetting event occurs if acc max ( τ ) ≥ acc end ( τ ) + α , where α is a threshold and acc end ( τ ) is the accuracy at the end of meta-training.
Revised: Formally, given an episode τ with the maximum accuracy acc max ( τ ) = max j acc j ( τ ) , a global forgetting event occurs if acc max ( τ ) ≥ acc end ( τ ) + α , where α is a threshold and acc end ( τ ) is the accuracy at the end of meta-training.

Original: Note that for each episode, a global forgetting event can occur only once.
Revised: Note that for each episode, a global forgetting event can occur only once.

Original: Local forgetting events .
Revised: Local forgetting events .

Original: For an episode τ in the j th epoch of meta-training, a local forgetting event is encountered if the accuracy of the episode at the j th epoch ( acc j ( τ ) ) is less than the accuracy at the ( j − 1)
Revised: For an episode τ in the j th epoch of meta-training, a local forgetting event is encountered if the accuracy of the episode at the j th epoch ( acc j ( τ ) ) is less than the accuracy at the ( j − 1)

Original: th epoch ( acc j − 1 ( τ ) ) by a particular threshold, denoted by α .
Revised: th epoch ( acc j − 1 ( τ ) ) by a particular threshold, denoted by α .

Original: Formally, a local forgotten event is encountered if acc j ( τ ) + α ≤ acc j − 1 ( τ ) .
Revised: Formally, a local forgotten event is encountered if acc j ( τ ) + α ≤ acc j − 1 ( τ ) .

Original: Empirically, we study local forgetting events for 0 .
Revised: Empirically, we study local forgetting events for 0 .

Original: 03 ≤ α ≤ 0 . 15 .
Revised: 03 ≤ α ≤ 0 . 15 .

Original: 5. F ORGETTING E VENTS AND H ARD E PISODES
Revised: 5. F ORGETTING E VENTS AND H ARD E PISODES

Original: Global forgetting events.
Revised: Global forgetting events.

Original: In Fig.
Revised: In Fig.

Original: (3), we track the accuracy of the hardest and easiest training episode from each of the few-shot datasets during the entire course of meta-training across different epochs.
Revised: (3), we track the accuracy of the hardest and easiest training episode from each of the few-shot datasets during the entire course of meta-training across different epochs.

Original: Visually, we observe that for the hard episode, the accuracy decreases after a certain point during the course of meta-training.
Revised: Visually, we observe that for the hard episode, the accuracy decreases after a certain point during the course of meta-training.

Original: However for the easy episode, this is not the case and the accuracy increases till the end of meta-training.
Revised: However for the easy episode, this is not the case and the accuracy increases till the end of meta-training.

Original: To draw more insights, we compute the global forgetting behaviour of different episodes.
Revised: To draw more insights, we compute the global forgetting behaviour of different episodes.

Original: We ﬁrst choose the 15 hardest and easiest episodes respectively from CIFAR-FS, mini-ImageNet, tieredImageNet and compute their ﬁnal episodic accuracy and the maximum episodic accuracy reached during meta-training.
Revised: We ﬁrst choose the 15 hardest and easiest episodes respectively from CIFAR-FS, mini-ImageNet, tieredImageNet and compute their ﬁnal episodic accuracy and the maximum episodic accuracy reached during meta-training.

Original: In Fig.
Revised: In Fig.

Original: (4), we ﬁnd that for the hard episodes, the gap between the ﬁnal accuracy and the maximum accuracy reached during meta-training is signiﬁcantly larger than the easy episodes.
Revised: (4), we ﬁnd that for the hard episodes, the gap between the ﬁnal accuracy and the maximum accuracy reached during meta-training is signiﬁcantly larger than the easy episodes.

Original: The gap in particular is large for mini-ImageNet and tieredImageNet, while for CIFAR-FS the gap is relatively narrow.
Revised: The gap in particular is large for mini-ImageNet and tieredImageNet, while for CIFAR-FS the gap is relatively narrow.

Original: For example, in case of the mini-ImageNet dataset, the gap can be ≈ 15% , whereas for the tieredImageNet dataset, this gap can be ≈ 10% .
Revised: For example, in case of the mini-ImageNet dataset, the gap can be ≈ 15% , whereas for the tieredImageNet dataset, this gap can be ≈ 10% .

Original: For CIFAR-FS, this gap is ≈ 6% .
Revised: For CIFAR-FS, this gap is ≈ 6% .

Original: This characteristic shows that hard episodes are globally forgotten in comparison to the easier episodes during the entire course of meta-training.
Revised: This characteristic shows that hard episodes are globally forgotten in comparison to the easier episodes during the entire course of meta-training.

Original: We provide further results on global forgetting events in Appendix C.
Revised: We provide further results on global forgetting events in Appendix C.

Original: Local forgetting events.
Revised: Local forgetting events.

Original: In order to compute the frequency of local forgetting events for easy and hard episodes across the three few-shot datasets, we ﬁrst choose 15 easy and hard episodes from each dataset.
Revised: In order to compute the frequency of local forgetting events for easy and hard episodes across the three few-shot datasets, we ﬁrst choose 15 easy and hard episodes from each dataset.

Original: Across this set of easy and hard episodes, we then compute the number of local forgetting events across various thresholds.
Revised: Across this set of easy and hard episodes, we then compute the number of local forgetting events across various thresholds.

Original: In general, across the entire duration of meta-training, we ﬁnd that the hard episodes have more local forgetting events than the easy episodes.
Revised: In general, across the entire duration of meta-training, we ﬁnd that the hard episodes have more local forgetting events than the easy episodes.

Original: For instance, in Fig.
Revised: For instance, in Fig.

Original: (5)-(a), we observe that there is a substantial gap in the number of forgetting events encountered for easy and hard episodes during the entire course of meta-training.
Revised: (5)-(a), we observe that there is a substantial gap in the number of forgetting events encountered for easy and hard episodes during the entire course of meta-training.

Original: Furthermore, to gain more insights about this gap in the number of encountered forgetting events, we understand how this gap behaves in the ﬁrst 20 epochs of meta-training (Fig.
Revised: Furthermore, to gain more insights about this gap in the number of encountered forgetting events, we understand how this gap behaves in the ﬁrst 20 epochs of meta-training (Fig.

Original: (5)-(b))
Revised: (5)-(b))

Original: and the last 20 epochs of meta-training (Fig.
Revised: and the last 20 epochs of meta-training (Fig.

Original: (5)-(c)).
Revised: (5)-(c)).

Original: In particular, we ﬁnd that the gap is narrow during the initial stages of meta-training, whereas the gap widens substantially during the later stages.
Revised: In particular, we ﬁnd that the gap is narrow during the initial stages of meta-training, whereas the gap widens substantially during the later stages.

Original: To summarize, we ﬁnd that forgetting occurs in meta-learning even when the tasks are drawn from a single task distribution.
Revised: To summarize, we ﬁnd that forgetting occurs in meta-learning even when the tasks are drawn from a single task distribution.

Original: Furthermore, we ﬁnd a strong connection between episode hardness and forgetting events, where we show that hard episodes are more easily forgotten than easy episodes, both in the local and global contexts.
Revised: Furthermore, we ﬁnd a strong connection between episode hardness and forgetting events, where we show that hard episodes are more easily forgotten than easy episodes, both in the local and global contexts.

Original: In the next section, we investigate two meta-training strategies to improve few-shot performance on hard episodes.
Revised: In the next section, we investigate two meta-training strategies to improve few-shot performance on hard episodes.

Original:  I MPROVING P ERFORMANCE ON H ARD E PISODES
Revised:  I MPROVING P ERFORMANCE ON H ARD E PISODES

Original: In this section, we investigate and benchmark two different meta-training strategies based on adversarial training and curriculum learning in order to improve prediction performance on hard episodes.
Revised: In this section, we investigate and benchmark two different meta-training strategies based on adversarial training and curriculum learning in order to improve prediction performance on hard episodes.

Original: Recent work (Gong et al., 2020; Ni et al., 2021) uses adversarial training to select episode speciﬁc augmentations from a wide pool of diverse data augmentation methods resulting in the highest loss.
Revised: Recent work (Gong et al., 2020; Ni et al., 2021) uses adversarial training to select episode speciﬁc augmentations from a wide pool of diverse data augmentation methods resulting in the highest loss.

Original: This loss is then optimized with respect to the model parameters during training.
Revised: This loss is then optimized with respect to the model parameters during training.

Original: Such data augmentation selection strategies have been shown to mitigate over-ﬁtting and improve generalization.
Revised: Such data augmentation selection strategies have been shown to mitigate over-ﬁtting and improve generalization.

Original: In our work, we study how such training strategies can be used to select hard episodes and improve prediction performance across these episodes.
Revised: In our work, we study how such training strategies can be used to select hard episodes and improve prediction performance across these episodes.

Original: In particular, we investigate two variants of adversarial training as proposed ﬁrst in Gong et al.
Revised: In particular, we investigate two variants of adversarial training as proposed ﬁrst in Gong et al.

Original: (2020): (i) General adversarial training (AT); (ii) Adversarial curriculum training (ACT).
Revised: (2020): (i) General adversarial training (AT); (ii) Adversarial curriculum training (ACT).

Original: 6. G ENERAL ADVERSARIAL TRAINING
Revised: 6. G ENERAL ADVERSARIAL TRAINING

Original: We adopt the adversarial training procedure from (Gong et al., 2020) to ﬁrst select episodes with a high loss and optimize the meta-learner only with respect to the loss incurred by such hard episodes.
Revised: We adopt the adversarial training procedure from (Gong et al., 2020) to ﬁrst select episodes with a high loss and optimize the meta-learner only with respect to the loss incurred by such hard episodes.

Original: Speciﬁcally, this involves solving a saddle-point optimization problem where the underlying loss is minimized with respect to the parameters of the model and maximized with respect to the input.
Revised: Speciﬁcally, this involves solving a saddle-point optimization problem where the underlying loss is minimized with respect to the parameters of the model and maximized with respect to the input.

Original: In particular, during each update of the meta-learner, we ﬁrst draw a batch of episodes each containing support and query examples.
Revised: In particular, during each update of the meta-learner, we ﬁrst draw a batch of episodes each containing support and query examples.

Original: Then for each element (episode) of the batch, we additionally sample new episodes with support and query examples belonging to the same classes as the original element of the batch.
Revised: Then for each element (episode) of the batch, we additionally sample new episodes with support and query examples belonging to the same classes as the original element of the batch.

Original: Then the episode with the highest loss is selected amongst the set consisting of the original episode of the batch and the additional sampled episodes.
Revised: Then the episode with the highest loss is selected amongst the set consisting of the original episode of the batch and the additional sampled episodes.

Original: The model parameters are then updated with respect to the gradient of the loss incurred by the selected hard episode.
Revised: The model parameters are then updated with respect to the gradient of the loss incurred by the selected hard episode.

Original: Formally, we solve the following min-max optimization:
Revised: Formally, we solve the following min-max optimization:

Original: [Equation]
Revised: [Equation]

Original: where θ (cid:48)
Revised: where θ (cid:48)

Original: =
Revised: =

Original: A ( θ, t s ) is the ﬁne-tuning step with the support examples from the selected episodes with the highest loss, g ( τ ) is an operator which samples additional episodes to select from for each task τ in the batch, F is the base learner and (cid:96) is the loss function.
Revised: A ( θ, t s ) is the ﬁne-tuning step with the support examples from the selected episodes with the highest loss, g ( τ ) is an operator which samples additional episodes to select from for each task τ in the batch, F is the base learner and (cid:96) is the loss function.

Original: In our experiments, we let g ( τ ) select four additional episodes per sampled episode τ .
Revised: In our experiments, we let g ( τ ) select four additional episodes per sampled episode τ .

Original: We provide a more detailed description of the hyper-parameters in Appendix B.1.
Revised: We provide a more detailed description of the hyper-parameters in Appendix B.1.

Original: 6. A DVERSARIAL C URRICULUM T RAINING
Revised: 6. A DVERSARIAL C URRICULUM T RAINING

Original: Curriculum learning (Hacohen & Weinshall, 2019; Bengio et al., 2009) aims to mimic the learning process of humans and animals.
Revised: Curriculum learning (Hacohen & Weinshall, 2019; Bengio et al., 2009) aims to mimic the learning process of humans and animals.

Original: In particular, humans learn new tasks in a well deﬁned order; i.e., it ﬁrst learns easy tasks and then gradually moves towards learning more complex tasks.
Revised: In particular, humans learn new tasks in a well deﬁned order; i.e., it ﬁrst learns easy tasks and then gradually moves towards learning more complex tasks.

Original: Inspired by this, we modify Eq.
Revised: Inspired by this, we modify Eq.

Original: (3) and investigate a curriculum meta-training strategy.
Revised: (3) and investigate a curriculum meta-training strategy.

Original: Speciﬁcally, during the initial phase of meta-training, we select only a set of easy episodes to learn from, while in the later stages of training, harder episodes are selected.
Revised: Speciﬁcally, during the initial phase of meta-training, we select only a set of easy episodes to learn from, while in the later stages of training, harder episodes are selected.

Original: Formally, given the underlying model is meta-trained for | e | epochs, for the ﬁrst | e | / 2 epochs, the following loss is optimized:
Revised: Formally, given the underlying model is meta-trained for | e | epochs, for the ﬁrst | e | / 2 epochs, the following loss is optimized:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: +ACT
Revised: +ACT

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: MetaOptNet +
Revised: MetaOptNet +

Original: [Table]
Revised: [Table]

Original: - lum training (ACT) across different meta-learners.
Revised: - lum training (ACT) across different meta-learners.

Original: and for the last | e | / 2 epochs during meta-training, the general min-max adversarial loss described in Eq.
Revised: and for the last | e | / 2 epochs during meta-training, the general min-max adversarial loss described in Eq.

Original: (3) is used.
Revised: (3) is used.

Original:  E XPERIMENTS 7. E XPERIMENTAL S ETUP
Revised:  E XPERIMENTS 7. E XPERIMENTAL S ETUP

Original: Datasets .
Revised: Datasets .

Original: We use three standard few-shot classiﬁcation datasets for our experiments : (i) CIFARFS (Bertinetto et al., 2018); (ii) mini-ImageNet (Vinyals et al., 2016) and (iii) tieredImageNet (Ren et al., 2018).
Revised: We use three standard few-shot classiﬁcation datasets for our experiments : (i) CIFARFS (Bertinetto et al., 2018); (ii) mini-ImageNet (Vinyals et al., 2016) and (iii) tieredImageNet (Ren et al., 2018).

Original: CIFAR-FS is sampled from CIFAR-100, whereas mini-ImageNet and tieredImageNet are subsets of ImageNet (Deng et al., 2009).
Revised: CIFAR-FS is sampled from CIFAR-100, whereas mini-ImageNet and tieredImageNet are subsets of ImageNet (Deng et al., 2009).

Original: We use the class splits from (Lee et al., 2019) for all the three datasets.
Revised: We use the class splits from (Lee et al., 2019) for all the three datasets.

Original: Note that tieredImageNet is a more challenging dataset than mini-ImageNet as the splits are constructed from near the root of the ImageNet hierarchy.
Revised: Note that tieredImageNet is a more challenging dataset than mini-ImageNet as the splits are constructed from near the root of the ImageNet hierarchy.

Original: We provide more details on the datasets in Appendix B.2.
Revised: We provide more details on the datasets in Appendix B.2.

Original: Architectures .
Revised: Architectures .

Original: We primarily use two standard architectures for our experiments: (i) 4-layer convolutional network as introduced in Vinyals et al.
Revised: We primarily use two standard architectures for our experiments: (i) 4-layer convolutional network as introduced in Vinyals et al.

Original: (2017); (ii) ResNet-

Revised: (2017); (ii) ResNet-

Original: (He et al., 2015) which is used by Oreshkin et al.
Revised: (He et al., 2015) which is used by Oreshkin et al.

Original: (2018) in the few-shot setting.
Revised: (2018) in the few-shot setting.

Original: Both of these architectures use batch-normalization after every convolutional layer and use ReLU as the activation function.
Revised: Both of these architectures use batch-normalization after every convolutional layer and use ReLU as the activation function.

Original: Similar architectures for few-shot learning have been previously used in Lee et al.
Revised: Similar architectures for few-shot learning have been previously used in Lee et al.

Original: ; Arnold et al.
Revised: ; Arnold et al.

Original: Meta-learners .
Revised: Meta-learners .

Original: We use prototypical networks (Snell et al., 2017) from the metric learning family of few-shot algorithms.
Revised: We use prototypical networks (Snell et al., 2017) from the metric learning family of few-shot algorithms.

Original: In addition, we use MetaOptNet (Lee et al., 2019) and R2D2 (Bertinetto et al., 2018) as representative algorithms from the optimization based meta-learning methods.
Revised: In addition, we use MetaOptNet (Lee et al., 2019) and R2D2 (Bertinetto et al., 2018) as representative algorithms from the optimization based meta-learning methods.

Original: Model Selection .
Revised: Model Selection .

Original: We use the validation set for each dataset to select the best model.
Revised: We use the validation set for each dataset to select the best model.

Original: Primarily, we run the validation procedure every 1k iterations on 2k episodes from the validation set to select the best model.
Revised: Primarily, we run the validation procedure every 1k iterations on 2k episodes from the validation set to select the best model.

Original: Finally, we evaluate on 1k test episodes from the test set for each dataset.
Revised: Finally, we evaluate on 1k test episodes from the test set for each dataset.

Original: 7. D ISCUSSION 7.2. A DVERSARIAL TRAINING IMPROVES PERFORMANCE ON HARD EPISODES
Revised: 7. D ISCUSSION 7.2. A DVERSARIAL TRAINING IMPROVES PERFORMANCE ON HARD EPISODES

Original: Across different meta-learners (ProtoNets, R2D2, MetaOptNet) and few-shot datasets (CIFAR-FS, mini-ImageNet, tieredImageNet), we ﬁnd that the Adversarial Training (AT) strategy works well in general to improve both the average episodic performance as well as the episodic performance on hard episodes.
Revised: Across different meta-learners (ProtoNets, R2D2, MetaOptNet) and few-shot datasets (CIFAR-FS, mini-ImageNet, tieredImageNet), we ﬁnd that the Adversarial Training (AT) strategy works well in general to improve both the average episodic performance as well as the episodic performance on hard episodes.

Original: The comprehensive results for the average performance of meta-learners is presented in Table (1), while the results on hard episodes are presented in Table (2).
Revised: The comprehensive results for the average performance of meta-learners is presented in Table (1), while the results on hard episodes are presented in Table (2).

Original: In particular, we ﬁnd
Revised: In particular, we ﬁnd

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: +ACT
Revised: +ACT

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: +ACT
Revised: +ACT

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: MetaOptNet +AT +
Revised: MetaOptNet +AT +

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: across different meta-learners on hard episodes.
Revised: across different meta-learners on hard episodes.

Original: We report the mean accuracy over 30 hardest episodes for each meta-learner.
Revised: We report the mean accuracy over 30 hardest episodes for each meta-learner.

Original: that adversarial meta-training strategies never hurt the average episodic performance and improves over the baseline in a majority of our experimental settings.
Revised: that adversarial meta-training strategies never hurt the average episodic performance and improves over the baseline in a majority of our experimental settings.

Original: However, we ﬁnd that the Adversarial
Revised: However, we ﬁnd that the Adversarial

Original: Training strategy (AT) leads to a large gain over the baseline meta-training strategy for hard episodes.
Revised: Training strategy (AT) leads to a large gain over the baseline meta-training strategy for hard episodes.

Original: Speciﬁcally, we ﬁnd that the improvements are more signiﬁcant for the 1-shot case when compared to the 5-shot case.
Revised: Speciﬁcally, we ﬁnd that the improvements are more signiﬁcant for the 1-shot case when compared to the 5-shot case.

Original: For example, in the 1-shot case, we observe a 5% gain for CIFAR-FS with R2-D and 3% gain for mini-ImageNet with prototypical networks.
Revised: For example, in the 1-shot case, we observe a 5% gain for CIFAR-FS with R2-D and 3% gain for mini-ImageNet with prototypical networks.

Original: For tieredImageNet, we observe ≈ 2% improvement on episodic performance for hard episodes with prototypical networks.
Revised: For tieredImageNet, we observe ≈ 2% improvement on episodic performance for hard episodes with prototypical networks.

Original: 7.2. A DVERSARIAL TRAINING IS BETTER THAN CURRICULUM TRAINING
Revised: 7.2. A DVERSARIAL TRAINING IS BETTER THAN CURRICULUM TRAINING

Original: Although curriculum training leads to better generalization in supervised learning (Hacohen & Weinshall, 2019; Bengio et al., 2009), we ﬁnd that in meta-learning, the Adversarial Curriculum strategy (ACT) generally performs worse than both the baseline and general Adversarial Training (AT) in a majority of our experimental settings.
Revised: Although curriculum training leads to better generalization in supervised learning (Hacohen & Weinshall, 2019; Bengio et al., 2009), we ﬁnd that in meta-learning, the Adversarial Curriculum strategy (ACT) generally performs worse than both the baseline and general Adversarial Training (AT) in a majority of our experimental settings.

Original: Our observation on curriculum training for meta-learning is consistent with the recent work of (Arnold et al., 2021) where they show that curriculum metatraining strategies underperform signiﬁcantly when compared to the baseline meta-training.
Revised: Our observation on curriculum training for meta-learning is consistent with the recent work of (Arnold et al., 2021) where they show that curriculum metatraining strategies underperform signiﬁcantly when compared to the baseline meta-training.

Original: We note that although the curriculum formulation in (Arnold et al., 2021) is different than ours, both methods present easy episodes to the meta-learner ﬁrst followed by hard episodes.
Revised: We note that although the curriculum formulation in (Arnold et al., 2021) is different than ours, both methods present easy episodes to the meta-learner ﬁrst followed by hard episodes.

Original: While we present a negative result on curriculum meta-training, we believe that this observation can be used as a guide to develop more advanced and improved curriculum meta-training strategies in the future.
Revised: While we present a negative result on curriculum meta-training, we believe that this observation can be used as a guide to develop more advanced and improved curriculum meta-training strategies in the future.

Original: In summary, we ﬁnd that although there is no one-size-ﬁts-all solution to improve performance on hard episodes, adversarial meta-training strategies perform better than the baseline and curriculum learning.
Revised: In summary, we ﬁnd that although there is no one-size-ﬁts-all solution to improve performance on hard episodes, adversarial meta-training strategies perform better than the baseline and curriculum learning.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we investigated how different meta-learners perform on episodes of varying hardness.
Revised: In this paper, we investigated how different meta-learners perform on episodes of varying hardness.

Original: We found that there exists a wide gap in the performance of the meta-learners between the easiest and the hardest episode across different few-shot datasets.
Revised: We found that there exists a wide gap in the performance of the meta-learners between the easiest and the hardest episode across different few-shot datasets.

Original: Furthermore, we investigated various facets of the hard episodes and uncovered two major properties: (i) Hard episodes usually have multiple diverse objects in the query examples, whereas the support set primarily consists of objects of a single category, (ii) Hard episodes are forgotten by the underlying meta-learner at a higher frequency than the easier episodes.
Revised: Furthermore, we investigated various facets of the hard episodes and uncovered two major properties: (i) Hard episodes usually have multiple diverse objects in the query examples, whereas the support set primarily consists of objects of a single category, (ii) Hard episodes are forgotten by the underlying meta-learner at a higher frequency than the easier episodes.

Original: To improve prediction performance over hard episodes, we investigated and benchmarked different training strategies such as adversarial training and curriculum learning during meta-training.
Revised: To improve prediction performance over hard episodes, we investigated and benchmarked different training strategies such as adversarial training and curriculum learning during meta-training.

Original: We found that adversarial training strategies are beneﬁcial towards meta-training and improve the average episodic performance as well as performance over hard episodes.
Revised: We found that adversarial training strategies are beneﬁcial towards meta-training and improve the average episodic performance as well as performance over hard episodes.

Original: Based on our analysis in this paper, designing more robust meta-learning algorithms that can generalize to hard episodes is an important direction of future work.
Revised: Based on our analysis in this paper, designing more robust meta-learning algorithms that can generalize to hard episodes is an important direction of future work.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: Small experimental details are crucial to reproduce results in deep learning.
Revised: Small experimental details are crucial to reproduce results in deep learning.

Original: In order to foster the reproducibility of our paper, we provide all the small details of our experiments in the Appendix (Section : Hyperparameters).
Revised: In order to foster the reproducibility of our paper, we provide all the small details of our experiments in the Appendix (Section : Hyperparameters).

Original: We would like to point out that we use the general hyper-parameters in our experimental setup, as used in (Lee et al., 2019) to ensure reproducibility of the baselines which is particularly important in few-shot learning research.
Revised: We would like to point out that we use the general hyper-parameters in our experimental setup, as used in (Lee et al., 2019) to ensure reproducibility of the baselines which is particularly important in few-shot learning research.

Original: Additionally, considering a majority of our paper is on analysis of meta-learners and generating plots, we provide all the necessary details (e.g, number of episodes chosen for generating the plots) available in each corresponding section (See the subsections of Section 5).
Revised: Additionally, considering a majority of our paper is on analysis of meta-learners and generating plots, we provide all the necessary details (e.g, number of episodes chosen for generating the plots) available in each corresponding section (See the subsections of Section 5).

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: In recent times, there has been a signiﬁcant increase in the usage of deep learning models in the industry.
Revised: In recent times, there has been a signiﬁcant increase in the usage of deep learning models in the industry.

Original: Domains of application include NLP, computer vision and healthcare to name a few.
Revised: Domains of application include NLP, computer vision and healthcare to name a few.

Original: Training task and domain speciﬁc deep models require large amounts of labeled training data which is not available for a variety of use-cases.
Revised: Training task and domain speciﬁc deep models require large amounts of labeled training data which is not available for a variety of use-cases.

Original: The paradigm of few-shot learning (or meta-learning) aims to learn generalizable models from only a few training examples.
Revised: The paradigm of few-shot learning (or meta-learning) aims to learn generalizable models from only a few training examples.

Original: Our paper is primarily geared towards understanding the science of such few-shot learning methods.
Revised: Our paper is primarily geared towards understanding the science of such few-shot learning methods.

Original: We believe our work can accelerate research on few-shot learning with a speciﬁc focus on improving learning from hard episodes.
Revised: We believe our work can accelerate research on few-shot learning with a speciﬁc focus on improving learning from hard episodes.

Original: One possible indirect risk is the deployment of our model without following the best practices for collecting fair and unbiased datasets for training.
Revised: One possible indirect risk is the deployment of our model without following the best practices for collecting fair and unbiased datasets for training.

Original: Our hope is that the positive impacts outweigh the negatives, speciﬁcally because standard best-use practices should mitigate most of the risks.
Revised: Our hope is that the positive impacts outweigh the negatives, speciﬁcally because standard best-use practices should mitigate most of the risks.

Original: A L OSS AS A MEASURE OF HARDNESS
Revised: A L OSS AS A MEASURE OF HARDNESS

Original: B I MPLEMENTATION D ETAILS B. H YPERPARAMETERS
Revised: B I MPLEMENTATION D ETAILS B. H YPERPARAMETERS

Original: Similar to (Lee et al., 2019), for the optimizer we use SGD with Nesterov momentum and weight decay.
Revised: Similar to (Lee et al., 2019), for the optimizer we use SGD with Nesterov momentum and weight decay.

Original: We set the momentum to be 0.9 and weight decay to be 0.0005.
Revised: We set the momentum to be 0.9 and weight decay to be 0.0005.

Original: During meta-training, the underlying meta-learner is trained for 60 epochs with each epoch consisting of 1000 episodes.
Revised: During meta-training, the underlying meta-learner is trained for 60 epochs with each epoch consisting of 1000 episodes.

Original: We use 8 episodes per batch during the meta-learner update.
Revised: We use 8 episodes per batch during the meta-learner update.

Original: The initial learning rate is kept at 0.
Revised: The initial learning rate is kept at 0.

Original: which is then decayed using a learning rate scheduler at epoch 20, 40 and 50 as shown in (Lee et al., 2019).
Revised: which is then decayed using a learning rate scheduler at epoch 20, 40 and 50 as shown in (Lee et al., 2019).

Original: We use a 5-way classiﬁcation for each of our experimental settings.
Revised: We use a 5-way classiﬁcation for each of our experimental settings.

Original: During each iteration of meta-training, we use 6 query examples, while during meta-testing we use 15 query examples.
Revised: During each iteration of meta-training, we use 6 query examples, while during meta-testing we use 15 query examples.

Original: Following the practice of (Gidaris & Komodakis, 2018; Qiao et al., 2017; Lee et al., 2019), we use horizontal ﬂip, random crop, and color jitter as data augmentation techniques during meta-training.
Revised: Following the practice of (Gidaris & Komodakis, 2018; Qiao et al., 2017; Lee et al., 2019), we use horizontal ﬂip, random crop, and color jitter as data augmentation techniques during meta-training.

Original: Across each of the experimental setup, the best model for meta-testing is chosen based on the best validation set performance.
Revised: Across each of the experimental setup, the best model for meta-testing is chosen based on the best validation set performance.

Original: In general, few-shot learning can be performed in two scenarios: (i)
Revised: In general, few-shot learning can be performed in two scenarios: (i)

Original: Inductive setting where each test example is evaluated independently; (ii) Transductive setting where the few-shot learner has access to all the test examples.
Revised: Inductive setting where each test example is evaluated independently; (ii) Transductive setting where the few-shot learner has access to all the test examples.

Original: In all our experimental settings, we follow the inductive setting where each query example is evaluated independently.
Revised: In all our experimental settings, we follow the inductive setting where each query example is evaluated independently.

Original: For general adversarial training method (AT), we choose sample 4 additional episodes per sampled episode.
Revised: For general adversarial training method (AT), we choose sample 4 additional episodes per sampled episode.

Original: For a batch of size 8, there would be 32 additional episodes with a total of 40 episodes.
Revised: For a batch of size 8, there would be 32 additional episodes with a total of 40 episodes.

Original: 8 episodes with the highest loss are selected from this pool for optimizing the meta-learner.
Revised: 8 episodes with the highest loss are selected from this pool for optimizing the meta-learner.

Original: For our adversarial curriculum learning method (ACT), we choose the set of 8 episodes with the lowest loss for the ﬁrst 30 epochs of meta-training and choose the set of 8 episodes during the last 30 epochs of meta-training.
Revised: For our adversarial curriculum learning method (ACT), we choose the set of 8 episodes with the lowest loss for the ﬁrst 30 epochs of meta-training and choose the set of 8 episodes during the last 30 epochs of meta-training.

Original: Across all our experimental settings, we use the same number of shots during meta-training and meta-testing to match the training and testing conditions.
Revised: Across all our experimental settings, we use the same number of shots during meta-training and meta-testing to match the training and testing conditions.

Original: B. F EW - SHOT D ATASETS
Revised: B. F EW - SHOT D ATASETS

Original: mini-ImageNet .
Revised: mini-ImageNet .

Original: The mini-ImageNet dataset is introduced by (Vinyals et al., 2016) and is a standard few-shot classiﬁcation benchmark.
Revised: The mini-ImageNet dataset is introduced by (Vinyals et al., 2016) and is a standard few-shot classiﬁcation benchmark.

Original: This dataset consists of 100 classes which are chosen from ILSVRC-2012 (Deng et al., 2009).
Revised: This dataset consists of 100 classes which are chosen from ILSVRC-2012 (Deng et al., 2009).

Original: The 100 classes are split into 64, 16, and 20 classes for meta-training, meta-validation and meta-testing.
Revised: The 100 classes are split into 64, 16, and 20 classes for meta-training, meta-validation and meta-testing.

Original: Each class has 600 images resulting in a total of 60000 images for the entire dataset.
Revised: Each class has 600 images resulting in a total of 60000 images for the entire dataset.

Original: tieredImageNet.
Revised: tieredImageNet.

Original: The tieredImageNet dataset is a larger and more challenging few-shot learning benchmark.
Revised: The tieredImageNet dataset is a larger and more challenging few-shot learning benchmark.

Original: It consists of 608 classes in total which are subclasses of ILSVRC-2012 (Deng et al., 2009).
Revised: It consists of 608 classes in total which are subclasses of ILSVRC-2012 (Deng et al., 2009).

Original: The total number of classes are split as 351, 97 and 160 classes for meta-training, metavalidation and meta-testing respectively.
Revised: The total number of classes are split as 351, 97 and 160 classes for meta-training, metavalidation and meta-testing respectively.

Original: These classes are selected such that there is minimum semantic similarity between the different splits, which makes this dataset challenging.
Revised: These classes are selected such that there is minimum semantic similarity between the different splits, which makes this dataset challenging.

Original: CIFAR-FS.
Revised: CIFAR-FS.

Original: The CIFAR-FS dataset (Bertinetto et al., 2018) is curated from CIFAR-100 and comprises of 100 classes in total with each class consisting of 600 images.
Revised: The CIFAR-FS dataset (Bertinetto et al., 2018) is curated from CIFAR-100 and comprises of 100 classes in total with each class consisting of 600 images.

Original: The classes are randomly split into 64, 16 and 20 for meta-training, meta-validation and meta-testing respectively.
Revised: The classes are randomly split into 64, 16 and 20 for meta-training, meta-validation and meta-testing respectively.

Original: Amongst the three datasets, the images are of size 84x84 for mini-ImageNet and tieredImageNet, while for CIFAR-FS, the images are of size 32x32.
Revised: Amongst the three datasets, the images are of size 84x84 for mini-ImageNet and tieredImageNet, while for CIFAR-FS, the images are of size 32x32.

Original: C G LOBAL F ORGETTING E VENTS
Revised: C G LOBAL F ORGETTING E VENTS

Original: To gain further insights into global forgetting events, we select 5 hard and easy episodes from each of the few-shot datasets and track their accuracy during the course of meta-training.
Revised: To gain further insights into global forgetting events, we select 5 hard and easy episodes from each of the few-shot datasets and track their accuracy during the course of meta-training.

Original: We use prototypical networks as the base meta-learner for our analysis.
Revised: We use prototypical networks as the base meta-learner for our analysis.

Original: We ﬁnd and validate in Fig.
Revised: We ﬁnd and validate in Fig.

Original: (10), that our analysis on global forgetting (as shown in the main paper) holds true for more number of hard episodes.
Revised: (10), that our analysis on global forgetting (as shown in the main paper) holds true for more number of hard episodes.

Original: Additionally, we also choose 25 hard and easy episodes to compute the mean of their maximum accuracy reached during meta-training and the accuracy at the end of training.
Revised: Additionally, we also choose 25 hard and easy episodes to compute the mean of their maximum accuracy reached during meta-training and the accuracy at the end of training.

Original: We ﬁnd that for hard episodes on an average, the gap between the maximum accuracy reached during training and the ﬁnal accuracy is large.
Revised: We ﬁnd that for hard episodes on an average, the gap between the maximum accuracy reached during training and the ﬁnal accuracy is large.

Original: However, for easy episodes we ﬁnd this gap to be narrow.
Revised: However, for easy episodes we ﬁnd this gap to be narrow.

Original: This large gap for hard episodes signiﬁes they they undergo catastrophic forgetting during the course of meta-training.
Revised: This large gap for hard episodes signiﬁes they they undergo catastrophic forgetting during the course of meta-training.

Original: of the maximum accuracy reached during the course of meta-training and the accuracy at the end of meta-training.
Revised: of the maximum accuracy reached during the course of meta-training and the accuracy at the end of meta-training.

Original: For hard episodes, we observe a substantial gap denoting global forgetting.
Revised: For hard episodes, we observe a substantial gap denoting global forgetting.

Original: D O N T RANSFERABILITY OF E PISODES
Revised: D O N T RANSFERABILITY OF E PISODES

Original: Recent and concurrent work such as (Arnold et al., 2021) shows that episode hardness transfer across different architectures and meta-learners.
Revised: Recent and concurrent work such as (Arnold et al., 2021) shows that episode hardness transfer across different architectures and meta-learners.

Original: In this section, we revisit their analysis and take a closer look at the transferability of episodes across various meta-learners and architectures to obtain certain new insights.
Revised: In this section, we revisit their analysis and take a closer look at the transferability of episodes across various meta-learners and architectures to obtain certain new insights.

Original: To this end, we compute the Pearson and Spearman correlations of the losses (hardness) incurred by episodes across different architectures and meta-learners for each few-shot dataset.
Revised: To this end, we compute the Pearson and Spearman correlations of the losses (hardness) incurred by episodes across different architectures and meta-learners for each few-shot dataset.

Original: Across all the three few-shot datasets, we ﬁnd that the order of transferability across meta-learners and architectures is similar.
Revised: Across all the three few-shot datasets, we ﬁnd that the order of transferability across meta-learners and architectures is similar.

Original: For example, episodes transfer the best consistently between ResNetMetaOptNet and ResNet-R2D2 across all the datasets, while episodes transfer the worst between
Revised: For example, episodes transfer the best consistently between ResNetMetaOptNet and ResNet-R2D2 across all the datasets, while episodes transfer the worst between

Original: ResNet-R2D2 and Conv-Proto.
Revised: ResNet-R2D2 and Conv-Proto.

Original: This observation makes intuitive sense as the meta-training strategies for both R2D2 and MetaOptNet are more similar for than R2D2 and prototypical networks.
Revised: This observation makes intuitive sense as the meta-training strategies for both R2D2 and MetaOptNet are more similar for than R2D2 and prototypical networks.

Original: Amongst the datasets, we ﬁnd that transfer of episode hardness between meta-learners and architectures is much more effective for tieredImageNet than mini-ImageNet.
Revised: Amongst the datasets, we ﬁnd that transfer of episode hardness between meta-learners and architectures is much more effective for tieredImageNet than mini-ImageNet.

Original: For example, out of thecombinations for transferability of episodes, 3 of them have a correlation of more than 0.90 in case of tieredImageNet.
Revised: For example, out of thecombinations for transferability of episodes, 3 of them have a correlation of more than 0.90 in case of tieredImageNet.

Original: E M ORE ON V ISUAL S EMANTICS OF H ARD E PISODES
Revised: E M ORE ON V ISUAL S EMANTICS OF H ARD E PISODES

Original: In this section, we provide more instances of hard episodes.
Revised: In this section, we provide more instances of hard episodes.

Original: In particular, we ﬁnd that episodes also become hard when there are closely related ﬁne-grained categories in the episode.
Revised: In particular, we ﬁnd that episodes also become hard when there are closely related ﬁne-grained categories in the episode.

Original: For example, in an episode consisting of dogs from the classes Malamute and Dalmatian , we notice that the underlying meta-learner often gets confused between the two ﬁne-grained categories leading to erroneous predictions on the query examples.
Revised: For example, in an episode consisting of dogs from the classes Malamute and Dalmatian , we notice that the underlying meta-learner often gets confused between the two ﬁne-grained categories leading to erroneous predictions on the query examples.

Original: Presence of similar ﬁne-grained categories in episodes is therefore one failure mode of existing
Revised: Presence of similar ﬁne-grained categories in episodes is therefore one failure mode of existing

Original: -learning methods
Revised: -learning methods

Original: F D ISTRIBUTION OF L OSSES FOR CIFAR-FS AND TIERED I MAGE N ET
Revised: F D ISTRIBUTION OF L OSSES FOR CIFAR-FS AND TIERED I MAGE N ET

Original: G F ORGETTING EVENTS FOR R2D2 AND M ETA O PT N ET
Revised: G F ORGETTING EVENTS FOR R2D2 AND M ETA O PT N ET

Revised: H H ARDNESS OF OOD EPISODES

Revised: Episode hardness and out-of-distribution (OOD) episodes have a strong connection between them.

Revised: We emphasise that the primary aim of our paper is to analyse episode hardness when the episodes during meta-testing are drawn from the same task distribution as meta-training.

Revised: However, we note that episodes can be hard for a meta-learner when the meta-test episodes are drawn from a different distribution.

Revised: To analyse the hardness of OOD episodes, we ﬁrst curate a new meta-testing set where the support examples in the new episodes are slightly perturbed.

Revised: We use the standard corruptions used previously to generate the ImageNet-C dataset (Hendrycks & Dietterich, 2019) in order to create new episodes where the support examples are perturbed.

Revised: Precisely, for every support example, we randomly choose from a set of perturbations: (None, Gaussian Noise, Shot Noise, Impulse Noise,

Revised: Blur, Frost, Fog, Brightness, Contrast, Elastic) .

Revised: This creates a new test episode which is OOD with respect to the episodes encountered during meta-training.

Revised: From Fig.

Revised: (18), we ﬁnd that the query losses incurred for such perturbed or OOD episodes are higher than the in-distribution episodes.

Revised: Notably, there is an overlap in the distribution of losses for the in-distribution and OOD episodes, which might indicate that some of the hard in-distribution episodes are actually outliers or OOD.

Revised: An end-to-end study of the susceptibility of meta-learners to OOD episodes (and also training strategies to improve on OOD episodes) is an interesting research direction and an important direction for future work.