Original: IEPT: I NSTANCE -L EVEL AND E PISODE -L EVEL P RETEXT T ASKS FOR F EW -S HOT L EARNING
Revised: IEPT: I NSTANCE -L EVEL AND E PISODE -L EVEL P RETEXT T ASKS FOR F EW -S HOT L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The need of collecting large quantities of labeled training data for each new task has limited the usefulness of deep neural networks.
Revised: The need of collecting large quantities of labeled training data for each new task has limited the usefulness of deep neural networks.

Original: Given data from a set of source tasks, this limitation can be overcome using two transfer learning approaches: few-shot learning (FSL) and self-supervised learning (SSL).
Revised: Given data from a set of source tasks, this limitation can be overcome using two transfer learning approaches: few-shot learning (FSL) and self-supervised learning (SSL).

Original: The former aims to learn ‘how to learn’ by designing learning episodes using source tasks to simulate the challenge of solving the target new task with few labeled samples.
Revised: The former aims to learn ‘how to learn’ by designing learning episodes using source tasks to simulate the challenge of solving the target new task with few labeled samples.

Original: In contrast, the latter exploits an annotation-free pretext task across all source tasks in order to learn generalizable feature representations.
Revised: In contrast, the latter exploits an annotation-free pretext task across all source tasks in order to learn generalizable feature representations.

Original: In this work, we propose a novel
Revised: In this work, we propose a novel

Original: Instance-level and Episode-level Pretext Task (IEPT) framework that seamlessly integrates SSL into FSL.
Revised: Instance-level and Episode-level Pretext Task (IEPT) framework that seamlessly integrates SSL into FSL.

Original: Speciﬁcally, given an FSL episode, we ﬁrst apply geometric transformations to each instance to generate extended episodes.
Revised: Speciﬁcally, given an FSL episode, we ﬁrst apply geometric transformations to each instance to generate extended episodes.

Original: At the instancelevel, transformation recognition is performed as per standard SSL.
Revised: At the instancelevel, transformation recognition is performed as per standard SSL.

Original: Importantly, at the episode-level, two SSL-FSL hybrid learning objectives are devised: (1) The consistency across the predictions of an FSL classiﬁer from different extended episodes is maximized as an episode-level pretext task.
Revised: Importantly, at the episode-level, two SSL-FSL hybrid learning objectives are devised: (1) The consistency across the predictions of an FSL classiﬁer from different extended episodes is maximized as an episode-level pretext task.

Original: The features extracted from each instance across different episodes are integrated to construct a single
Revised: The features extracted from each instance across different episodes are integrated to construct a single

Original: FSL classiﬁer for meta-learning.
Revised: FSL classiﬁer for meta-learning.

Original: Extensive experiments show that our proposed model (i.e., FSL with IEPT) achieves the new state-of-the-art.
Revised: Extensive experiments show that our proposed model (i.e., FSL with IEPT) achieves the new state-of-the-art.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep convolutional neural networks (CNNs) (Krizhevsky et al., 2012;
Revised: Deep convolutional neural networks (CNNs) (Krizhevsky et al., 2012;

Original: He et al., 2016b; Huang et al., 2017) have seen tremendous successes in a wide range of application ﬁelds, especially in visual recognition.
Revised: He et al., 2016b; Huang et al., 2017) have seen tremendous successes in a wide range of application ﬁelds, especially in visual recognition.

Original: However, the powerful learning ability of CNNs depends on a large amount of manually labeled training data.
Revised: However, the powerful learning ability of CNNs depends on a large amount of manually labeled training data.

Original: In practice, for many visual recognition tasks, sufﬁcient manual annotation is either too costly to collect or not feasible (e.g., for rare object classes).
Revised: In practice, for many visual recognition tasks, sufﬁcient manual annotation is either too costly to collect or not feasible (e.g., for rare object classes).

Original: This has severely limited the usefulness of CNNs for real-world application scenarios.
Revised: This has severely limited the usefulness of CNNs for real-world application scenarios.

Original: Attempts have been made recently to mitigate such a limitation from two distinct perspectives, resulting in two popular research lines, both of which aim to transfer knowledge learned from the data of a set of source tasks to a new target one: few-shot learning (FSL) and self-supervised learning (SSL).
Revised: Attempts have been made recently to mitigate such a limitation from two distinct perspectives, resulting in two popular research lines, both of which aim to transfer knowledge learned from the data of a set of source tasks to a new target one: few-shot learning (FSL) and self-supervised learning (SSL).

Original: FSL (Fei-Fei et al., 2006; Vinyals et al., 2016;
Revised: FSL (Fei-Fei et al., 2006; Vinyals et al., 2016;

Original: Finn et al., 2017; Snell et al., 2017; Sung et al., 2018) typically takes a ‘learning to learn’ or meta-learning paradigm.
Revised: Finn et al., 2017; Snell et al., 2017; Sung et al., 2018) typically takes a ‘learning to learn’ or meta-learning paradigm.

Original: That is, it aims to learn an algorithm for learning from few labeled samples, which generalizes well across any tasks.
Revised: That is, it aims to learn an algorithm for learning from few labeled samples, which generalizes well across any tasks.

Original: To that end, it adopts  an episodic training strategy – the source tasks are arranged into learning episodes, each of which contains n classes and k labeled samples per class to simulate the setting for the target task.
Revised: To that end, it adopts [Figure] an episodic training strategy – the source tasks are arranged into learning episodes, each of which contains n classes and k labeled samples per class to simulate the setting for the target task

Original: Partof the CNN model (e.g., feature extraction subnet, classiﬁcation layers, or parameter initialization) is then meta-learned for rapid adaptation to new tasks.
Revised: Part the CNN model (e.g., feature extraction subnet, classiﬁcation layers, or parameter initialization) is then meta-learned for rapid adaptation to new tasks.

Original: In contrast, SSL (Doersch et al., 2015; Noroozi & Favaro, 2016; Iizuka et al., 2016; Doersch &
Revised: In contrast, SSL (Doersch et al., 2015; Noroozi & Favaro, 2016; Iizuka et al., 2016; Doersch &

Original: Zisserman, 2017; Noroozi et al., 2018) does not require the source data to be annotated.
Revised: Zisserman, 2017; Noroozi et al., 2018) does not require the source data to be annotated.

Original: Instead, it exploits an annotation-free pretext task on the source task data in the hope that a task-generalizable feature representation can be learned from the source tasks for easy adoption or adaptation in a target task.
Revised: Instead, it exploits an annotation-free pretext task on the source task data in the hope that a task-generalizable feature representation can be learned from the source tasks for easy adoption or adaptation in a target task.

Original: Such a pretext task gets its self-supervised signal at the per-instance level.
Revised: Such a pretext task gets its self-supervised signal at the per-instance level.

Original: Examples include rotation and context prediction (Gidaris et al., 2018; Doersch et al., 2015), jigsaw solving (Noroozi &
Revised: Examples include rotation and context prediction (Gidaris et al., 2018; Doersch et al., 2015), jigsaw solving (Noroozi &

Original: Favaro, 2016), and colorization (Iizuka et al., 2016; Larsson et al., 2016).
Revised: Favaro, 2016), and colorization (Iizuka et al., 2016; Larsson et al., 2016).

Original: Since these pretext tasks are class-agnostic, solving them leads to the learning of transferable knowledge.
Revised: Since these pretext tasks are class-agnostic, solving them leads to the learning of transferable knowledge.

Original: [Figure]

Original: Since both FSL and SSL aim to reduce the need of collecting a large amount of labeled training data for a target task by transferring knowledge from a set of source tasks, it is natural to consider combining them in a single framework.
Revised: Since both FSL and SSL aim to reduce the need of collecting a large amount of labeled training data for a target task by transferring knowledge from a set of source tasks, it is natural to consider combining them in a single framework.

Original: Indeed, two recent works (Gidaris et al., 2019; Su et al., 2020) proposed to integrate SSL into FSL by adding an auxiliary SSL pretext task in an FSL model.
Revised: Indeed, two recent works (Gidaris et al., 2019; Su et al., 2020) proposed to integrate SSL into FSL by adding an auxiliary SSL pretext task in an FSL model.

Original: It showed that the SSL learning objective is complementary to that of FSL and combining them leads to improved FSL performance.
Revised: It showed that the SSL learning objective is complementary to that of FSL and combining them leads to improved FSL performance.

Original: However, in (Gidaris et al., 2019; Su et al., 2020), SSL is combined with FSL in a superﬁcial way: it is only taken as a separate auxiliary task for each single training instance and has no effect on the episodic training pipeline of the FSL model.
Revised: However, in (Gidaris et al., 2019; Su et al., 2020), SSL is combined with FSL in a superﬁcial way: it is only taken as a separate auxiliary task for each single training instance and has no effect on the episodic training pipeline of the FSL model.

Original: Importantly, by ignoring the class labels of samples, the instance-level SSL learning objective is weak on its own.
Revised: Importantly, by ignoring the class labels of samples, the instance-level SSL learning objective is weak on its own.

Original: Since meta-learning across episodes is the essence of most contemporary FSL models, we argue that adding instance-level SSL pretext tasks alone fails to exploit fully the complementarity of the aforementioned FSL and SSL, for which a closer and deeper integration is needed.
Revised: Since meta-learning across episodes is the essence of most contemporary FSL models, we argue that adding instance-level SSL pretext tasks alone fails to exploit fully the complementarity of the aforementioned FSL and SSL, for which a closer and deeper integration is needed.

Original: To that end, in this paper we propose a novel Instance-level and Episode-level Pretext Task (IEPT) framework for few-shot recognition.
Revised: To that end, in this paper we propose a novel Instance-level and Episode-level Pretext Task (IEPT) framework for few-shot recognition.

Original: Apart from adding an instance-level pretext SSL task as in (Gidaris et al., 2019; Su et al., 2020), we introduce two episode-level SSL-FSL hybrid learning objectives for seamless SSL-FSL integration.
Revised: Apart from adding an instance-level pretext SSL task as in (Gidaris et al., 2019; Su et al., 2020), we introduce two episode-level SSL-FSL hybrid learning objectives for seamless SSL-FSL integration.

Original: Concretely, as illustrated in Figure 1, our full model has three additional learning objectives (besides the standard FSL one): (1) Different rotation transformations are applied to each original few-shot episode to generate a set of extended episodes, where each image has a rotation label for the instance-level pretext task (i.e., to predict the rotation label).
Revised: Concretely, as illustrated in Figure 1, our full model has three additional learning objectives (besides the standard FSL one): (1) Different rotation transformations are applied to each original few-shot episode to generate a set of extended episodes, where each image has a rotation label for the instance-level pretext task (i.e., to predict the rotation label).

Original: (2) The consistency across the predictions of an FSL classiﬁer from different extended episodes is maximized as an episode-level pretext task.
Revised: (2) The consistency across the predictions of an FSL classiﬁer from different extended episodes is maximized as an episode-level pretext task.

Original: For each training image, the rotation transformation does not change its semantic content and hence its class label; the FSL classiﬁer predictions across different extended episodes thus should be consistent, hence the consistency regularization objective.
Revised: For each training image, the rotation transformation does not change its semantic content and hence its class label; the FSL classiﬁer predictions across different extended episodes thus should be consistent, hence the consistency regularization objective.

Original: (3) The correlation of features across instances from these extended episodes is modeled by a transformer-based attention module, optimizing the fusion of the features of each instance/image and its various rotation-transformed versions mainly for task adaptation during meta-testing.
Revised: (3) The correlation of features across instances from these extended episodes is modeled by a transformer-based attention module, optimizing the fusion of the features of each instance/image and its various rotation-transformed versions mainly for task adaptation during meta-testing.

Original: Importantly, with these three new learning objectives introduced in IEPT, any meta-learning based FSL model can now beneﬁt more from SSL by fully exploiting their complementarity.
Revised: Importantly, with these three new learning objectives introduced in IEPT, any meta-learning based FSL model can now beneﬁt more from SSL by fully exploiting their complementarity.

Original: Our main contributions are three-fold: (1) For the ﬁrst time, we propose both instance-level and episode-level pretext tasks (IEPT) for integrating SSL into FSL.
Revised: Our main contributions are: (1) For the ﬁrst time, we propose both instance-level and episode-level pretext tasks (IEPT) for integrating SSL into FSL.

Original: The episode-level pretext task enables episodic training of SSL and hence closer integration of SSL with FSL.
Revised: The episode-level pretext task enables episodic training of SSL and hence closer integration of SSL with FSL.

Original: In addition to these pretext tasks, FSL further beneﬁts from SSL by integrating features extracted from various rotationtransformed versions of the original training instances.
Revised: In addition to these pretext tasks, FSL further beneﬁts from SSL by integrating features extracted from various rotation-transformed versions of the original training instances.

Original: The optimal way of feature integration is learned by a transformer-based attention module, which is mainly designed for task adaptation during meta-testing.
Revised: The optimal way of feature integration is learned by a transformer-based attention module, which is mainly designed for task adaptation during meta-testing.

Original: (3) Extensive experiments show that our model(i.e., FSL with IEPT) achieves the new state-of-the-art.
Revised: (3) Extensive experiments show that the proposed model achieves the new state-of-the-art.

Original: The code will be released soon.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Few-Shot Learning.
Revised: Few-Shot Learning.

Original: The recent FSL studies are dominated by meta-learning based methods.
Revised: The recent FSL studies are dominated by meta-learning based methods.

Original: They can be divided into three groups: (1) Metric-based methods (Vinyals et al., 2016;
Revised: They can be divided into three groups: (1) Metric-based methods (Vinyals et al., 2016;

Original: Snell et al., 2017;
Revised: Snell et al., 2017;

Original: Sung et al., 2018; Allen et al., 2019;
Revised: Sung et al., 2018; Allen et al., 2019;

Original: Xing et al., 2019; Li et al., 2019a;b; Wu et al., 2019; Ye et al., 2020; Afrasiyabi et al., 2020; Liu et al., 2020; Zhang et al., 2020) aim to learn the distance metric between feature embeddings.
Revised: Xing et al., 2019; Li et al., 2019a;b; Wu et al., 2019; Ye et al., 2020; Afrasiyabi et al., 2020; Liu et al., 2020; Zhang et al., 2020) aim to learn the distance metric between feature embeddings.

Original: The focus of these methods is often on meta-learning of a feature-extraction CNN, whilst the classiﬁers used are of simple form such as a nearest-neighbor classiﬁer.
Revised: The focus of these methods is often on meta-learning of a feature-extraction CNN, whilst the classiﬁers used are of simple form such as a nearest-neighbor classiﬁer.

Original: (2) Optimization-based methods (Finn et al., 2017; Ravi & Larochelle, 2017; Rusu et al., 2019; Lee et al., 2019) learn to optimize the model rapidly given a few labeled samples per class in the new task.
Revised: (2) Optimization-based methods (Finn et al., 2017; Ravi & Larochelle, 2017; Rusu et al., 2019; Lee et al., 2019) learn to optimize the model rapidly given a few labeled samples per class in the new task.

Original: (3) Model-based methods (Santoro et al., 2016; Munkhdalai & Yu, 2017; Mishra et al., 2018) focus on designing either speciﬁc model structures or parameters capable of rapid updating.
Revised: (3) Model-based methods (Santoro et al., 2016; Munkhdalai & Yu, 2017; Mishra et al., 2018) focus on designing either speciﬁc model structures or parameters capable of rapid updating.

Original: Apart from these three groups of methods, other FSL methods have attempted feature hallucination (Schwartz et al., 2018; Hariharan & Girshick, 2017; Gao et al., 2018; Wang et al., 2018; Zhang et al., 2019;
Revised: Apart from these three groups of methods, other FSL methods have attempted feature hallucination (Schwartz et al., 2018; Hariharan & Girshick, 2017; Gao et al., 2018; Wang et al., 2018; Zhang et al., 2019;

Original: Tsutsui et al., 2019) which generates additional samples from the given few shots for network ﬁnetuning, and parameter predicting (Qiao et al., 2018; Qi et al., 2018; Gidaris & Komodakis, 2019; 2018) which learns to predict part of the parameters of a network given few samples of new classes for quick adaptation.
Revised: Tsutsui et al., 2019) which generates additional samples from the given few shots for network ﬁnetuning, and parameter predicting (Qiao et al., 2018; Qi et al., 2018; Gidaris & Komodakis, 2019; 2018) which learns to predict part of the parameters of a network given few samples of new classes for quick adaptation.

Original: In this work, we adopt the metric-based Prototypical Network (ProtoNet) (Snell et al., 2017) as the basic FSL classiﬁer for the main instantiation of our IEPT framework due to its simplicity and popularity.
Revised: In this work, we adopt the metric-based Prototypical Network (ProtoNet) (Snell et al., 2017) as the basic FSL classiﬁer for the main instantiation of our IEPT framework due to its simplicity and popularity.

Original: However, we show that any meta-learning based FSL method can be combined with our IEPT (see results in Figure 2(c)).
Revised: However, we show that any meta-learning based FSL method can be combined with our IEPT (see results in Figure 2(c)).

Original: Self-Supervised Learning.
Revised: Self-Supervised Learning.

Original: In SSL, it is assumed that the source task data is label-free and a pretext task is designed to provide self-supervision signals at the instance-level.
Revised: In SSL, it is assumed that the source task data is label-free and a pretext task is designed to provide self-supervision signals at the instance-level.

Original: Existing SSL approaches differ mainly in the pretext task design.
Revised: Existing SSL approaches differ mainly in the pretext task design.

Original: These include predicting the rotation angle (Gidaris et al., 2018) and the context of image patch (Doersch et al., 2015; Nathan Mundhenk et al., 2018), jigsaw solving (Noroozi & Favaro, 2016; Noroozi et al., 2018) (i.e. shufﬂing and then reordering image patch), and performing images reversion (Iizuka et al., 2016; Pathak et al., 2016; Larsson et al., 2016).
Revised: These include predicting the rotation angle (Gidaris et al., 2018) and the context of image patch (Doersch et al., 2015; Nathan Mundhenk et al., 2018), jigsaw solving (Noroozi & Favaro, 2016; Noroozi et al., 2018) (i.e. shufﬂing and then reordering image patch), and performing images reversion (Iizuka et al., 2016; Pathak et al., 2016; Larsson et al., 2016).

Original: SSL has been shown to be beneﬁcial to various down-steam tasks such as semantic object matching (Novotny et al., 2018), object segmentation (Ji et al., 2019) and object detection (Doersch & Zisserman, 2017) by learning transferable feature presentations for these tasks.
Revised: SSL has been shown to be beneﬁcial to various down-steam tasks such as semantic object matching (Novotny et al., 2018), object segmentation (Ji et al., 2019) and object detection (Doersch & Zisserman, 2017) by learning transferable feature presentations for these tasks.

Original: Integrating Self-Supervised Learning into Few-Shot Learning.
Revised: Integrating Self-Supervised Learning into Few-Shot Learning.

Original: To the best of our knowledge, only two recent works (Gidaris et al., 2019; Su et al., 2020) have attempted combining SSL with FSL.
Revised: To the best of our knowledge, only two recent works (Gidaris et al., 2019; Su et al., 2020) have attempted combining SSL with FSL.

Original: However, the integration of SSL into FSL is often shallow: the original FSL training pipeline is intact; in the meantime, an additional loss on each image w.r.t.
Revised: However, the integration of SSL into FSL is often shallow: the original FSL training pipeline is intact; in the meantime, an additional loss on each image w.r.t.

Original: a self-supervised signal like the rotation angle or relative patch location is introduced.
Revised: a self-supervised signal like the rotation angle or relative patch location is introduced.

Original: With pretext tasks solely at the instance level, combining the two approaches (i.e., SSL and FSL) can only be superﬁcial without fully exploiting the episodic training pipeline unique to FSL.
Revised: With pretext tasks solely at the instance level, combining the two approaches (i.e., SSL and FSL) can only be superﬁcial without fully exploiting the episodic training pipeline unique to FSL.

Original: Different from (Gidaris et al., 2019; Su et al., 2020), we introduce an episode-level pretext task to integrate SSL into the episodic training in FSL fully.
Revised: Different from (Gidaris et al., 2019; Su et al., 2020), we introduce an episode-level pretext task to integrate SSL into the episodic training in FSL fully.

Original: Speciﬁcally, the consistency across the predictions of an FSL classiﬁer from different extended episodes is maximized to reﬂect the fact that various rotation transformations should not alter the class-label prediction.
Revised: Speciﬁcally, the consistency across the predictions of an FSL classiﬁer from different extended episodes is maximized to reﬂect the fact that various rotation transformations should not alter the class-label prediction.

Original: Moreover, features of each instance and its various rotation-transformed versions are now fused for
Revised: Moreover, features of each instance and its various rotation-transformed versions are now fused for

Original: FSL classiﬁcation, to integrate SSL with FSL for the supervised classiﬁcation task.
Revised: FSL classiﬁcation, to integrate SSL with FSL for the supervised classiﬁcation task.

Original: Our experimental results show that thanks to the closer integration of SSL and FSL, our IEPT clearly outperforms (Gidaris et al., 2019; Su et al., 2020) (see Table 1).
Revised: Our experimental results show that thanks to the closer integration of SSL and FSL, our IEPT clearly outperforms (Gidaris et al., 2019; Su et al., 2020) (see Table 1).

Original:  M ETHODOLOGY 3. P RELIMINARY
Revised:  M ETHODOLOGY 3. P RELIMINARY

Original: Problem Setting.
Revised: Problem Setting.

Original: Given an n -way k -shot FSL task sampled from a test set D t , to imitate the test setting, an FSL model is typically trained in an episodic way.
Revised: Given an n -way k -shot FSL task sampled from a test set D t , to imitate the test setting, an FSL model is typically trained in an episodic way.

Original: That is, n -way k -shot episodes are randomly sampled from a training set D s , where the class label space of D s has no overlap with that of D t .
Revised: That is, n -way k -shot episodes are randomly sampled from a training set D s , where the class label space of D s has no overlap with that of D t .

Original: Each episode E e contains a support set S e and a query set Q e .
Revised: Each episode E e contains a support set S e and a query set Q e .

Original: Concretely, we ﬁrst randomly sample a set of n classes C e from the training set, and then generate S e and Q e by sampling k support samples and q query samples from each class in C e , respectively.
Revised: Concretely, we ﬁrst randomly sample a set of n classes C e from the training set, and then generate S e and Q e by sampling k support samples and q query samples from each class in C e , respectively.

Original: Formally, we have
Revised: Formally, we have

Original: S e = { ( x i , y i ) | y i
Revised: S e = { ( x i , y i ) | y i

Original: ∈
Revised: ∈

Original: C e , i = 1 , ..., n × k } and Q e = { ( x i , y i )
Revised: C e , i = 1 , ..., n × k } and Q e = { ( x i , y i )

Original: | y i ∈
Revised: | y i ∈

Original: C e , i = 1 , ..., n × q } , where
Revised: C e , i = 1 , ..., n × q } , where

Original: S e (cid:84) Q e = ∅ .
Revised: S e (cid:84) Q e = ∅ .

Original: For simplicity, we denote l k = n × k and l q = n × q .
Revised: For simplicity, we denote l k = n × k and l q = n × q .

Original: In the meta-training stage, the training process has an inner and an outer loop in each episode: in the inner loop, the model is updated using S e ; its performance is then evaluated on the query set Q e in the outer loop to update the model parameters or algorithm that one wants to meta-learn.
Revised: In the meta-training stage, the training process has an inner and an outer loop in each episode: in the inner loop, the model is updated using S e ; its performance is then evaluated on the query set Q e in the outer loop to update the model parameters or algorithm that one wants to meta-learn.

Original: Basic FSL Classiﬁer.
Revised: Basic FSL Classiﬁer.

Original: We employ ProtoNet (Snell et al., 2017) as the basic FSL model.
Revised: We employ ProtoNet (Snell et al., 2017) as the basic FSL model.

Original: This model has a feature-extraction CNN and a simple non-parametric classiﬁer.
Revised: This model has a feature-extraction CNN and a simple non-parametric classiﬁer.

Original: The parameter of the feature extractor is to be meta-learned.
Revised: The parameter of the feature extractor is to be meta-learned.

Original: Concretely, in the inner loop of an episode, ProtoNet ﬁxes the feature extractor and computes the mean feature embedding for each class as follows:
Revised: Concretely, in the inner loop of an episode, ProtoNet ﬁxes the feature extractor and computes the mean feature embedding for each class as follows:

Original: [Equation]
Revised: [Equation]

Original: where class c ∈ C e , f φ is a feature extractor with learnable parameters φ , and I is the indicator function.
Revised: where class c ∈ C e , f φ is a feature extractor with learnable parameters φ , and I is the indicator function.

Original: By computing the distance between the feature embedding of each query sample and that of the corresponding class, the loss function used to meta-learn φ in the outer loop is deﬁned as:
Revised: By computing the distance between the feature embedding of each query sample and that of the corresponding class, the loss function used to meta-learn φ in the outer loop is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where d ( · , · ) denotes a distance function (e.g., the l 2 distance).
Revised: where d ( · , · ) denotes a distance function (e.g., the l 2 distance).

Original: 3. P RETEXT T ASKS IN IEPT
Revised: 3. P RETEXT T ASKS IN IEPT

Original: The schematic of our IEPT is illustrated in Figure 1.
Revised: The schematic of our IEPT is illustrated in Figure 1.

Original: We ﬁrst deﬁne a set of 2D-rotation operators
Revised: We ﬁrst deﬁne a set of 2D-rotation operators

Original: G = { g r | r = 0 , ..., R − 1 } , where g r means the operator of rotating the image by r *90 degrees and R is the total number of rotations ( R = 4 in our implementation).
Revised: G = { g r | r = 0 , ..., R − 1 } , where g r means the operator of rotating the image by r *90 degrees and R is the total number of rotations ( R = 4 in our implementation).

Original: Given an original episode
Revised: Given an original episode

Original: E e = {S e , Q e } as described in Sec.
Revised: E e = {S e , Q e } as described in Sec.

Original: 3.1, we utilize the 2D-rotation operators from G in turn to transform each image in E e .
Revised: 3.1, we utilize the 2D-rotation operators from G in turn to transform each image in E e .

Original: This results in a set of R extended episodes (including the original one) E
Revised: This results in a set of R extended episodes (including the original one) E

Original: = {{S re , Q re }| r = 0 , ..., R − 1 } , where S re = { ( x i , y i , r ) | y i
Revised: = {{S re , Q re }| r = 0 , ..., R − 1 } , where S re = { ( x i , y i , r ) | y i

Original: ∈
Revised: ∈

Original: C e , i = 1 , ..., l k } and
Revised: C e , i = 1 , ..., l k } and

Original: Q re = { ( x i , y i , r ) | y i ∈
Revised: Q re = { ( x i , y i , r ) | y i ∈

Original: C e , i = 1 , ..., l q } .
Revised: C e , i = 1 , ..., l q } .

Original: Now each episode is denoted as E re = { ( x i , y i , r ) | y i ∈
Revised: Now each episode is denoted as E re = { ( x i , y i , r ) | y i ∈

Original: C e , i = 1 , ..., l k , l k + 1 , ..., l k + l q } , where the ﬁrst l k samples are from S re and the rest from Q re .
Revised: C e , i = 1 , ..., l k , l k + 1 , ..., l k + l q } , where the ﬁrst l k samples are from S re and the rest from Q re .

Original: Note that {S 0 e , Q 0 e } is the original episode {S e , Q e } .
Revised: Note that {S 0 e , Q 0 e } is the original episode {S e , Q e } .

Original: With the rotation transformations, each sample ( x i , y i , r i ) in E carries a class label
Revised: With the rotation transformations, each sample ( x i , y i , r i ) in E carries a class label

Original: y i for supervised learning (from the inherent class) and a label r i from the rotation operator for self-supervised learning.
Revised: y i for supervised learning (from the inherent class) and a label r i from the rotation operator for self-supervised learning.

Original: After generating the set of extended episodes
Revised: After generating the set of extended episodes

Original: E , the feature extractor f φ is applied to each image x i in E .
Revised: E , the feature extractor f φ is applied to each image x i in E .

Original: On these episodes, we design two self-supervised pretext tasks, one at the instance-level and the other episode-level.
Revised: On these episodes, we design two self-supervised pretext tasks, one at the instance-level and the other episode-level.

Original: Instance-Level Pretext Task.
Revised: Instance-Level Pretext Task.

Original: The instance-level task is to recognize different rotation transformations.
Revised: The instance-level task is to recognize different rotation transformations.

Original: The idea is that if the model to be meta-learned here (i.e., f φ ) can be used to distinguish different transformations, it must understand the canonical poses of objects (e.g., animals have legs touching the ground and trees have leaves on top), a vital part of class-agnostic and thus transferable knowledge.
Revised: The idea is that if the model to be meta-learned here (i.e., f φ ) can be used to distinguish different transformations, it must understand the canonical poses of objects (e.g., animals have legs touching the ground and trees have leaves on top), a vital part of class-agnostic and thus transferable knowledge.

Original: With the self-supervised rotation label r i , we consider the mapping: f θ rot : x i (cid:55)→ r i for each instance ( x i , y i , r i ) ∈ E , where f θ rot is a rotation classiﬁer with learnable parameters θ rot .
Revised: With the self-supervised rotation label r i , we consider the mapping: f θ rot : x i (cid:55)→ r i for each instance ( x i , y i , r i ) ∈ E , where f θ rot is a rotation classiﬁer with learnable parameters θ rot .

Original: Given the input pair ( x i , r i ) , the total instance-level rotation loss is a cross-entropy loss:
Revised: Given the input pair ( x i , r i ) , the total instance-level rotation loss is a cross-entropy loss:

Original: [Equation]
Revised: [Equation]

Original: where [ f θ rot ( f φ ( x i ))]
Revised: where [ f θ rot ( f φ ( x i ))]

Original: ∈ R R is the rotation scoring vector and [ · ] r means taking the r -th element.
Revised: ∈ R R is the rotation scoring vector and [ · ] r means taking the r -th element.

Original: Episode-Level Pretext Task.
Revised: Episode-Level Pretext Task.

Original: We design the episode-level task based on a simple principle: although different extended episodes contain images with different rotation transformations, these transformations do not change their class labels.
Revised: We design the episode-level task based on a simple principle: although different extended episodes contain images with different rotation transformations, these transformations do not change their class labels.

Original: Consequently, the FSL classiﬁer should produce consistent probability distributions for each instance across different extended episodes.
Revised: Consequently, the FSL classiﬁer should produce consistent probability distributions for each instance across different extended episodes.

Original: Such consistency can be measured using the Kullback–Leibler (KL) divergence.
Revised: Such consistency can be measured using the Kullback–Leibler (KL) divergence.

Original: Formally, for each extended episode {S re , Q re } in E , we ﬁrst deﬁne the probability distribution of FSL classiﬁcation over the query set Q re as P re =
Revised: Formally, for each extended episode {S re , Q re } in E , we ﬁrst deﬁne the probability distribution of FSL classiﬁcation over the query set Q re as P re =

Original: [ p r 1 ; · · · ; p rl q ] ∈ R l q × n , where p ri ∈ R n is the probability distribution for x i in Q re with its c -th element [ p ri ] c ( c = 1 , ..., n ) being:
Revised: [ p r 1 ; · · · ; p rl q ] ∈ R l q × n , where p ri ∈ R n is the probability distribution for x i in Q re with its c -th element [ p ri ] c ( c = 1 , ..., n ) being:

Original: [Equation]
Revised: [Equation]

Original: The above probability is computed as in Sec.
Revised: The above probability is computed as in Sec.

Original: 3.1 and the class embedding h rc is obtained from S re .
Revised: 3.1 and the class embedding h rc is obtained from S re .

Original: The mean probability distribution of the R extended episodes is thus given by:
Revised: The mean probability distribution of the R extended episodes is thus given by:

Original: [Equation]
Revised: [Equation]

Original: The total episode-level consistency regularization loss is computed with the KL divergence loss:
Revised: The total episode-level consistency regularization loss is computed with the KL divergence loss:

Original: [Equation]
Revised: [Equation]

Original: where mean( · ) is an element-wise averaging function.
Revised: where mean( · ) is an element-wise averaging function.

Original: 3. I NTEGRATED FSL T ASK
Revised: 3. I NTEGRATED FSL T ASK

Original: The two tasks introduced so far are self-supervised tasks without using the class labels in the query set.
Revised: The two tasks introduced so far are self-supervised tasks without using the class labels in the query set.

Original: Now we describe how in the supervised classiﬁcation task, the extended episodes can be used.
Revised: Now we describe how in the supervised classiﬁcation task, the extended episodes can be used.

Original: Given the set of extended episodes E , we denote the feature set of E as E emb , where E emb =
Revised: Given the set of extended episodes E , we denote the feature set of E as E emb , where E emb =

Original: { f φ ( x i ) | ( x i , y i , r ) ∈
Revised: { f φ ( x i ) | ( x i , y i , r ) ∈

Original: E r e , r = 0 , · · · , R − 1 , i = 1 , ..., l k + l q } .
Revised: E re , r = 0 , · · · , R − 1 , i = 1 , ..., l k + l q } .

Original: Note that each extended episode in E corresponds to one speciﬁc rotation transformation of the same set of images from the original episode
Revised: Note that each extended episode in E corresponds to one speciﬁc rotation transformation of the same set of images from the original episode

Original: E e .
Revised: E e .

Original: Therefore, in order to capture the correlation among instances with different transformations and learn how best combine them to form the class mean for meta-learning, an instance attention module is deployed w.r.t.
Revised: Therefore, in order to capture the correlation among instances with different transformations and learn how best combine them to form the class mean for meta-learning, an instance attention module is deployed w.r.t.

Original: each image in E e (i.e., all images are assumed to be independent).
Revised: each image in E e (i.e., all images are assumed to be independent).

Original: Speciﬁcally, based on E emb , we construct the feature tensor F ∈ R ( l k + l q ) × R × d , where d is the feature dimension.
Revised: Speciﬁcally, based on E emb , we construct the feature tensor F ∈ R ( l k + l q ) × R × d , where d is the feature dimension.

Original: We then adopt a transformer to obtain the integrated representation for FSL classiﬁcation.
Revised: We then adopt a transformer to obtain the integrated representation for FSL classiﬁcation.

Original: The transformer architecture is based on a self-attention mechanism, as in (Vaswani et al., 2017).
Revised: The transformer architecture is based on a self-attention mechanism, as in (Vaswani et al., 2017).

Original: It receives the triplet input ( F, F, F ) as ( Q, K, V )
Revised: It receives the triplet input ( F, F, F ) as ( Q, K, V )

Original: (Query, Key, and Value, respectively).
Revised: (Query, Key, and Value, respectively).

Original: With F ( i ) being the i -th row of F (w.r.t.
Revised: With F ( i ) being the i -th row of F (w.r.t.

Original: the i -th image in E e ), the attentive module is deﬁned as:
Revised: the i -th image in E e ), the attentive module is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where d K = d , and W Q , W K , W V represent the parameters of three fully-connected layers respectively (the parameters of the integration transformer are collected as θ int ).
Revised: where d K = d , and W Q , W K , W V represent the parameters of three fully-connected layers respectively (the parameters of the integration transformer are collected as θ int ).

Original: Note that the key and value are computed from each image and its augmented versions, i.e., they are computed independently without using inter-image correlation.
Revised: Note that the key and value are computed from each image and its augmented versions, i.e., they are computed independently without using inter-image correlation.

Original: With the attentive feature F att ∈ R ( l k + l q ) × R × d , the integrated representation F integ =
Revised: With the attentive feature F att ∈ R ( l k + l q ) × R × d , the integrated representation F integ =

Original: [ F S ; F Q ] ∈ R ( l k + l q ) × Rd ( F S and F Q are respectively for the support set and query set) is given by:
Revised: [ F S ; F Q ] ∈ R ( l k + l q ) × Rd ( F S and F Q are respectively for the support set and query set) is given by:

Original: [Equation]
Revised: [Equation]

Original: where ﬂatten( · ) denotes ﬂattening F att along the last two dimensions, i.e., concatenating the attentive features from different extended episodes for the corresponding images.
Revised: where ﬂatten( · ) denotes ﬂattening F att along the last two dimensions, i.e., concatenating the attentive features from different extended episodes for the corresponding images.

Original: The integrated representation is then inputted to the FSL classiﬁer to deﬁne the FSL classiﬁcation loss:
Revised: The integrated representation is then inputted to the FSL classiﬁer to deﬁne the FSL classiﬁcation loss:

Original: [Equation]
Revised: [Equation]

Original: where the class embedding h fc = 1 k · (cid:80) l k i =1 F Si · I ( y i
Revised: where the class embedding h fc = 1 k · (cid:80) l k i =1 F Si · I ( y i

Original: = c ) is computed on the support set.
Revised: = c ) is computed on the support set.

Original: Note that the integrated FSL task actually acts as an alternative to prediction averaging.
Revised: Note that the integrated FSL task actually acts as an alternative to prediction averaging.

Original: 3. T OTAL L OSS
Revised: 3. T OTAL L OSS

Original: The total training loss for our full model consists of the self-supervised losses from the pretext tasks and the supervised losses from the FSL tasks.
Revised: The total training loss for our full model consists of the self-supervised losses from the pretext tasks and the supervised losses from the FSL tasks.

Original: In this work, in addition to L integ in Eq.
Revised: In this work, in addition to L integ in Eq.

Original: (10), another supervised FSL loss L aux is also used (see Figure 1).
Revised: (10), another supervised FSL loss L aux is also used (see Figure 1).

Original: L aux is the average FSL classiﬁcation loss over the extended episodes.
Revised: L aux is the average FSL classiﬁcation loss over the extended episodes.

Original: Formally, it can be written as:
Revised: Formally, it can be written as:

Original: [Equation]
Revised: [Equation]

Original: Therefore, the total loss L total for training our full model is given as follows:
Revised: Therefore, the total loss L total for training our full model is given as follows:

Original: [Equation]
Revised: [Equation]

Original: where w 1 , w 2 , w 3 are the loss weight hyperparameters.
Revised: where w 1 , w 2 , w 3 are the loss weight hyperparameters.

Original: 3. I NFERENCE
Revised: 3. I NFERENCE

Original: During the test stage, we only exploit the integrated representation F integ for the ﬁnal FSL prediction.
Revised: During the test stage, we only exploit the integrated representation F integ for the ﬁnal FSL prediction.

Original: The predicted class label for x i ∈ Q e can be computed with Eq.
Revised: The predicted class label for x i ∈ Q e can be computed with Eq.

Original: (10) as:
Revised: (10) as:

Original: [Equation]
Revised: [Equation]

Original: 3. F ULL IEPT A LGORITHM
Revised: 3. F ULL IEPT A LGORITHM

Original: For easy reproduction, we present the full algorithm for FSL with IEPT in Algorithm 1.
Revised: For easy reproduction, we present the full algorithm for FSL with IEPT in Algorithm 1.

Original: Once learned, with the learned ψ , we can perform the inference over the test episodes with Eq.
Revised: Once learned, with the learned ψ , we can perform the inference over the test episodes with Eq.

Original: The code and trained models will be released soon.

Original: Algorithm 
Revised: Algorithm 1 FSL with IEPT

Original: Input [Table] [Table]

Original: return

Original: [Table]

Original:  E XPERIMENTS 4. E XPERIMENTAL S ETUP
Revised:  E XPERIMENTS 4. E XPERIMENTAL S ETUP

Original: Datasets.
Revised: Datasets.

Original: Two widely-used FSL datasets are selected: mini ImageNet (Vinyals et al., 2016) and tiered ImageNet (Ren et al., 2018).
Revised: Two widely-used FSL datasets are selected: mini ImageNet (Vinyals et al., 2016) and tiered ImageNet (Ren et al., 2018).

Original: The ﬁrst dataset consists of a total number of 100 classes ( images per class) and the train/validation/test split is set to 64/16/20 classes as in (Ravi & Larochelle, 2017).
Revised: The ﬁrst dataset consists of a total number of 100 classes ( images per class) and the train/validation/test split is set to 64/16/20 classes as in (Ravi & Larochelle, 2017).

Original: The second dataset is a larger dataset including 608 classes totally (nearly 1,200 images per class), which is split into 351/97/160 classes for train/validation/test.
Revised: The second dataset is a larger dataset including 608 classes totally (nearly 1,200 images per class), which is split into 351/97/160 classes for train/validation/test.

Original: Both datasets are subsets sampled from ImageNet (Russakovsky et al., 2015). 
Revised: Both datasets are subsets sampled from ImageNet (Russakovsky et al., 2015). [Table]

Original: Feature Extractors.
Revised: Feature Extractors.

Original: For fair comparison with published results, our IEPT adopts three widely-used feature extractors: Conv4-

Revised: For fair comparison with published results, our IEPT adopts three widely-used feature extractors: Conv4-

Original: (Vinyals et al., 2016), Conv4-512, and ResNet-12 (He et al., 2016a).
Revised: (Vinyals et al., 2016), Conv4-512, and ResNet-12 (He et al., 2016a).

Original: Particularly, Conv4-512 is almost the same as Conv4-64 except having a different channel size of the last convolution layer.
Revised: Particularly, Conv4-512 is almost the same as Conv4-64 except having a different channel size of the last convolution layer.

Original: To speed up the training process, as in many previous works (Ye et al., 2020;
Revised: To speed up the training process, as in many previous works (Ye et al., 2020;

Original: Zhang et al., 2020; Simon et al., 2020), we pretrain all the feature extractors on the training split of each dataset for our IEPT.
Revised: Zhang et al., 2020; Simon et al., 2020), we pretrain all the feature extractors on the training split of each dataset for our IEPT.

Original: Following (He et al., 2016a), we use the temperature scaling skill during the training phase.
Revised: Following (He et al., 2016a), we use the temperature scaling skill during the training phase.

Original: On both datasets, the input image size is 84 × 84 .
Revised: On both datasets, the input image size is 84 × 84 .

Original: The output feature dimensions of Conv4-64, Conv4-512, and ResNet-12 are 64, 512, and [Table]
Revised: The output feature dimensions of Conv4-64, Conv4-512, and ResNet-12 are 64, 512, and 640, respectively.

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: We take the 5-way 5-shot (or 1-shot) FSL evaluation setting [Table] as in previous works.
Revised: We take the 5-way 5-shot (or 1-shot) FSL evaluation setting, as in previous works.

Original: We randomly sample 2,000 episodes from the test split and report the mean classiﬁcation accuracy (top-1, %) as well as the 95% conﬁdence interval.
Revised: We randomly sample 2,000 episodes from the test split and report the mean classiﬁcation accuracy (top-1, %) as well as the 95% conﬁdence interval.

Original: Since the 
Revised: Since the integration transformer copes with each sample independently, we take a strict non-transductive setting during evaluation.

Original: Implementation Details.
Revised: Implementation Details.

Original: PyTorch is used for our implementation.
Revised: PyTorch is used for our implementation.

Original: We utilize the Adam optimizer (Kingma & Ba, 2015) for Conv4-64 & Conv4-512 and the SGD optimizer for ResNet-12 to train our
Revised: We utilize the Adam optimizer (Kingma & Ba, 2015) for Conv4-64 & Conv4-512 and the SGD optimizer for ResNet-12 to train our

Original: IEPT model.
Revised: IEPT model.

Original: The hyperparameters of our IEPT model are selected according to the performance on the validation split.
Revised: The hyperparameters of our IEPT model are selected according to the performance on the validation split.

Original: We will release the code soon.
Revised: We will release the code soon.

Original: 4. M AIN R ESULTS
Revised: 4. M AIN R ESULTS

Original: Comparison to State-of-the-Arts.
Revised: Comparison to State-of-the-Arts.

Original: We compare our IEPT with two groups of baselines: (1) Recent
Revised: We compare our IEPT with two groups of baselines: (1) Recent

Original: SSL-based FSL methods (Gidaris et al., 2019; Su et al., 2020); (2) Representative/latest FSL methods (w/o SSL) (Snell et al., 2017; Finn et al., 2017; Lee et al., 2019; Ravichandran et al., 2019; Simon et al., 2020; Zhang et al., 2020;
Revised: SSL-based FSL methods (Gidaris et al., 2019; Su et al., 2020); (2) Representative/latest FSL methods (w/o SSL) (Snell et al., 2017; Finn et al., 2017; Lee et al., 2019; Ravichandran et al., 2019; Simon et al., 2020; Zhang et al., 2020;

Original: Ye et al., 2020; Liu et al., 2020).
Revised: Ye et al., 2020; Liu et al., 2020).

Revised: The comparative results for 5-way

Original: The comparative results for 5-way 1/5-shot FSL are shown in Table 1.
Revised: [Table] [Table] [Figure] 1/5-shot FSL are shown in Table 1.

Original: We have the following observations: (1) When compared with the representative/latest FSL methods (w/o SSL), our IEPT achieves the best performance on all datasets and under all settings, validating the effectiveness of SSL with IEPT for FSL.
Revised: We have the following observations: (1) When compared with the representative/latest FSL methods (w/o SSL), our IEPT achieves the best performance on all datasets and under all settings, validating the effectiveness of SSL with IEPT for FSL.

Original: Our IEPT also clearly outperforms the two SSL-based FSL methods (Gidaris et al., 2019;
Revised: Our IEPT also clearly outperforms the two SSL-based FSL methods (Gidaris et al., 2019;

Original: Su et al., 2020) which only use instance-level pretext tasks, demonstrating the importance of closer/episode-level integration of
Revised: Su et al., 2020) which only use instance-level pretext tasks, demonstrating the importance of closer/episode-level integration of

Original: SSL into FSL.
Revised: SSL into FSL.

Original: The improvements achieved by our IEPT over ProtoNet range from 2% to 5%.
Revised: The improvements achieved by our IEPT over ProtoNet range from 2% to 5%.

Original: Since our IEPT takes ProtoNet as the baseline, the obtained margins provide direct evidence that SSL brings signiﬁcant beneﬁts to FSL.
Revised: Since our IEPT takes ProtoNet as the baseline, the obtained margins provide direct evidence that SSL brings signiﬁcant beneﬁts to FSL.

Original: Note that our IEPT is also shown to be effective under both the ﬁne-grained FSL and cross-domain FSL settings in Sec.
Revised: Note that our IEPT is also shown to be effective under both the ﬁne-grained FSL and cross-domain FSL settings in Sec.

Original: 4.3 (see Table 3).
Revised: 4.3 (see Table 3).

Original: [Table]

Original: [Figure]

Original: Ablation Study.
Revised: Ablation Study.

Original: Our full IEPT model is trained with four losses (see Eq.
Revised: Our full IEPT model is trained with four losses (see Eq.

Original: (12)), including two self-supervised losses and two supervised losses: the episode-level SSL loss L epis , the instance-level
Revised: (12)), including two self-supervised losses and two supervised losses: the episode-level SSL loss L epis , the instance-level

Original: SSL loss L inst , the auxiliary FSL loss L aux and the integrated FSL loss L integ .
Revised: SSL loss L inst , the auxiliary FSL loss L aux and the integrated FSL loss L integ .

Original: To demonstrate the contribution of each loss, we present the ablation study results for our full IEPT model in Table 2, where Conv4-64 is used as the backbone.
Revised: To demonstrate the contribution of each loss, we present the ablation study results for our full IEPT model in Table 2, where Conv4-64 is used as the backbone.

Original: We start with L integ and then add the additional three losses successively.
Revised: We start with L integ and then add the additional three losses successively.

Original: It can be observed that the performance of our model continuously increases when more losses are used, indicating that each loss contributes to the ﬁnal performance.
Revised: It can be observed that the performance of our model continuously increases when more losses are used, indicating that each loss contributes to the ﬁnal performance.

Original: 4. F URTHER E VALUATIONS
Revised: 4. F URTHER E VALUATIONS

Original: Different Combination Methods over Episodes.
Revised: Different Combination Methods over Episodes.

Original: We have introduced a transformer-based attention module to fuse the features of each instance from all extended episodes (and an integrated episode can be obtained) for the supervised classiﬁcation task (see Sec. 3.3).
Revised: We have introduced a transformer-based attention module to fuse the features of each instance from all extended episodes (and an integrated episode can be obtained) for the supervised classiﬁcation task (see Sec. 3.3).

Original: In this experiment, we compare it with two alternative ways of across-episode integration: (1) Averaging extended episodes: the extended episodes are directly fused for FSL classiﬁcation; (2) Averaging all episodes: the extended episodes as well as the integrated episode are fused for FSL classiﬁcation.
Revised: In this experiment, we compare it with two alternative ways of across-episode integration: (1) Averaging extended episodes: the extended episodes are directly fused for FSL classiﬁcation; (2) Averaging all episodes: the extended episodes as well as the integrated episode are fused for FSL classiﬁcation.

Original: We present the comparative results on mini ImageNet in Figure 2(a).
Revised: We present the comparative results on mini ImageNet in Figure 2(a).

Original: For comprehensive comparison, the results of FSL with each single extended episode are also reported.
Revised: For comprehensive comparison, the results of FSL with each single extended episode are also reported.

Original: We can observe that: (1) The performance of ‘Episode 0 ◦ ’ is the highest among the four baselines (i.e., FSL with single extended episode), perhaps because the feature extractor is pretrained on the original images without rotation transformations.
Revised: We can observe that: (1) The performance of ‘Episode 0 ◦ ’ is the highest among the four baselines (i.e., FSL with single extended episode), perhaps because the feature extractor is pretrained on the original images without rotation transformations.

Original: (2) FSL by averaging extended episodes (i.e., ‘Averaging extended episodes’) indeed improves each of the four  baselines.
Revised: (2) FSL by averaging extended episodes (i.e., ‘Averaging extended episodes’) indeed improves each of the four [Table] baselines.

Original: (3) FSL with integrated episode (i.e., ‘Integrated episode’) is superior to FSL by simply averaging extended episodes.
Revised: (3) FSL with integrated episode (i [Table] e., ‘Integrated episode’) is superior to FSL by simply averaging extended episodes.

Original: (4) Comparing ‘Integrated episode’ with ‘Averaging all episodes’, the performance of FSL with integrated episode is more stable across different settings, furthering validating the usefulness of our across-episode integration.
Revised: (4) Comparing ‘Integrated episode’ with ‘Averaging all episodes’, the performance of FSL with integrated episode is more stable across different settings, furthering validating the usefulness of our across-episode integration.

Original: Overall, the episode-integration module is indeed effective in FSL with self-supervision.
Revised: Overall, the episode-integration module is indeed effective in FSL with self-supervision.

Original: This is also supported by the visualization results in Figure 3 (see more visualization results in Appendices A.3 & A.4). [Table]
Revised: This is also supported by the visualization results presented in Appendices A.3 & A.4.

Original: Different Number of Extended Episodes.
Revised: Different Number of Extended Episodes.

Original: In all the above experiments, the number of the extended episodes R is set to 4 (rotation by 0 ◦ , 90 ◦ , 180 ◦ , 270 ◦ ).
Revised: In all the above experiments, the number of the extended episodes R is set to 4 (rotation by 0 ◦ , 90 ◦ , 180 ◦ , 270 ◦ ).

Original: Figure 2(b) shows the impact of the value of
Revised: Figure 2(b) shows the impact of the value of

Original: R .
Revised: R .

Original: Note that when R = 1 , our IEPT model is equivalent to ProtoNet which is without self-supervision.
Revised: Note that when R = 1 , our IEPT model is equivalent to ProtoNet which is without self-supervision.

Original: It can be seen that the performance of our model consistently grows when R increases from 1 to 4.
Revised: It can be seen that the performance of our model consistently grows when R increases from 1 to 4.

Original: Additionally, the study on exploiting other pretext tasks for our IEPT is presented in Appendix A.1.
Revised: Additionally, the study on exploiting other pretext tasks for our IEPT is presented in Appendix A.1.

Original: Different Basic FSL Classiﬁers.
Revised: Different Basic FSL Classiﬁers.

Original: As mentioned in Sec.
Revised: As mentioned in Sec.

Original: 3.1, we adopt ProtoNet as the basic FSL classiﬁer due to its scalability and simplicity.
Revised: 3.1, we adopt ProtoNet as the basic FSL classiﬁer due to its scalability and simplicity.

Original: To further show the effectiveness of our IEPT when other basic FSL classiﬁers are used, we provide the results obtained by our IEPT using ProtoNet, FEAT, and IMP for FSL in Figure 2(c).
Revised: To further show the effectiveness of our IEPT when other basic FSL classiﬁers are used, we provide the results obtained by our IEPT using ProtoNet, FEAT, and IMP for FSL in Figure 2(c).

Original: It can be clearly observed that our IEPT leads to an improvement of about 1-4% over each basic FSL method (ProtoNet, FEAT, or IMP), indicating that our IEPT can be applied to improve a variety of popular FSL methods.
Revised: It can be clearly observed that our IEPT leads to an improvement of about 1-4% over each basic FSL method (ProtoNet, FEAT, or IMP), indicating that our IEPT can be applied to improve a variety of popular FSL methods.

Original: Comparative Results for Fine-Grained FSL and Cross-Domain FSL.
Revised: Comparative Results for Fine-Grained FSL and Cross-Domain FSL.

Original: To evaluate our IEPT algorithm under the ﬁne-grained FSL and cross-domain FSL settings, we conduct experiments on
Revised: To evaluate our IEPT algorithm under the ﬁne-grained FSL and cross-domain FSL settings, we conduct experiments on

Original: CUB (Wah et al., 2011) and mini ImageNet → CUB, respectively.
Revised: CUB (Wah et al., 2011) and mini ImageNet → CUB, respectively.

Original: For ﬁne-grained FSL on CUB, following (Ye et al., 2020), we randomly split the dataset into 100 training classes, 50 validation classes, and 50 test classes.
Revised: For ﬁne-grained FSL on CUB, following (Ye et al., 2020), we randomly split the dataset into 100 training classes, 50 validation classes, and 50 test classes.

Original: For cross-domain FSL on mini ImageNet → CUB, the 100 training classes are from mini ImageNet; the 50 validation classes and 50 test classes (using the aforementioned split for ﬁne-grained FSL) are from CUB.
Revised: For cross-domain FSL on mini ImageNet → CUB, the 100 training classes are from mini ImageNet; the 50 validation classes and 50 test classes (using the aforementioned split for ﬁne-grained FSL) are from CUB.

Original: Under both settings, we use Conv4-64 as the feature extractor.
Revised: Under both settings, we use Conv4-64 as the feature extractor.

Original: The 5-way 1/5-shot FSL results are shown in Table 3.
Revised: The 5-way 1/5-shot FSL results are shown in Table 3.

Original: Our IEPT clearly achieves the best results, yielding 1–3 % improvements over the second-best FEAT.
Revised: Our IEPT clearly achieves the best results, yielding 1–3 % improvements over the second-best FEAT.

Original: This shows the effectiveness of our IEPT under both ﬁne-grained and cross-domain settings.
Revised: This shows the effectiveness of our IEPT under both ﬁne-grained and cross-domain settings.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have proposed a novel Instance-level and Episode-level Pretext Task (IEPT) framework for integrating SSL into FSL.
Revised: We have proposed a novel Instance-level and Episode-level Pretext Task (IEPT) framework for integrating SSL into FSL.

Original: For the ﬁrst time, we have introduced an episode-level pretext task for
Revised: For the ﬁrst time, we have introduced an episode-level pretext task for

Original: FSL with self-supervision, in addition to the conventional instance-level pretext task.
Revised: FSL with self-supervision, in addition to the conventional instance-level pretext task.

Original: Moreover, we have also developed an episode extension-integration framework by introducing an integration transformer module to fully exploit the extended episodes for FSL.
Revised: Moreover, we have also developed an episode extension-integration framework by introducing an integration transformer module to fully exploit the extended episodes for FSL.

Original: Extensive experiments on two benchmarks demonstrate that the proposed model (i.e., FSL with IEPT) achieves the new state-of-theart.
Revised: Extensive experiments on two benchmarks demonstrate that the proposed model (i.e., FSL with IEPT) achieves the new state-of-theart.

Original: Our ongoing research directions include: exploring other episode-level pretext tasks for FSL with self-supervision, and applying FSL with self-supervision to other vision problems. 
Revised: Our ongoing research directions include: exploring other episode-level pretext tasks for FSL with self-supervision, and applying FSL with self-supervision to other vision problems. A CKNOWLEDGMENTS

Revised: This work was supported in part by National Natural Science Foundation of China (61976220 and 61832017), Beijing Outstanding Young Scientist Program (BJJWZYJH012019100020098), Open

Revised: Project Program Foundation of Key Laboratory of Opto-Electronics Information Processing, Chinese

Revised: Academy of Sciences (OEIP-O-202006), and Alibaba Innovative Research (AIR) Program.

Original: A A PPENDIX A. C OMPARISON AMONG D IFFERENT SSL S TRATEGIES
Revised: A A PPENDIX A. C OMPARISON AMONG D IFFERENT SSL S TRATEGIES

Original: To generate the extended episodes in IEPT, we apply four rotation transformations (i.e. rotation by 0 ◦ , 90 ◦ , 180 ◦ , 270 ◦ ) to each image.
Revised: To generate the extended episodes in IEPT, we apply four rotation transformations (i.e. rotation by 0 ◦ , 90 ◦ , 180 ◦ , 270 ◦ ) to each image.

Original: It makes sense to explore whether other self-supervised strategies are also effective for our IEPT.
Revised: It makes sense to explore whether other self-supervised strategies are also effective for our IEPT.

Original: To this end, we exploit shufﬂing image patches (see Figure 4) for self-supervised learning (SSL).
Revised: To this end, we exploit shufﬂing image patches (see Figure 3) for self-supervised learning (SSL).

Original: Speciﬁcally, we divide each image into 2*2 patches and reorganize the patch orders to obtain a shufﬂing label.
Revised: Speciﬁcally, we divide each image into 2*2 patches and reorganize the patch orders to obtain a shufﬂing label.

Original: To compare with the rotation strategy fairly, we choose only four shufﬂing orders: (1, 2, 3, 4), (2, 3, 4, 1), (3, 4, 1, 2) and (4, 1, 2, 3).
Revised: To compare with the rotation strategy fairly, we choose only four shufﬂing orders: (1, 2, 3, 4), (2, 3, 4, 1), (3, 4, 1, 2) and (4, 1, 2, 3).

Original: Note that the (1, 2, 3, 4) shufﬂing order equals to the original image.
Revised: Note that the (1, 2, 3, 4) shufﬂing order equals to the original image.

Original: Similar to the rotation strategy, a fully-connected layer is utilized to recognize the shufﬂing order.
Revised: Similar to the rotation strategy, a fully-connected layer is utilized to recognize the shufﬂing order.

Original: The comparative results are shown in Table 4.
Revised: The comparative results are shown in Table 4.

Original: We can see that both IEPT with shufﬂing and IEPT with rotation achieve better performance than the original
Revised: We can see that both IEPT with shufﬂing and IEPT with rotation achieve better performance than the original

Original: ProtoNet.
Revised: ProtoNet.

Original: Particularly, IEPT with shufﬂing yields 1-3% and 3-4% improvements under 5-shot and 1-shot, respectively.
Revised: Particularly, IEPT with shufﬂing yields 1-3% and 3-4% improvements under 5-shot and 1-shot, respectively.

Original: This clearly shows the effectiveness of our IEPT for FSL even when different
Revised: This clearly shows the effectiveness of our IEPT for FSL even when different

Original: SSL strategies are used to deﬁne the pretext tasks.
Revised: SSL strategies are used to deﬁne the pretext tasks.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: A. C OMPARISON AMONG D IFFERENT I NTEGRATION A PPROACHES
Revised: A. C OMPARISON AMONG D IFFERENT I NTEGRATION A PPROACHES

Original: We employ the integration transformer to ﬁnd the intrinsic correlation of various rotation-transformed instances.
Revised: We employ the integration transformer to ﬁnd the intrinsic correlation of various rotation-transformed instances.

Original: The transformer architecture is based on a self-attention mechanism.
Revised: The transformer architecture is based on a self-attention mechanism.

Original: Concretely, it receives the feature sets of extended episodes as input Q , K and V .
Revised: Concretely, it receives the feature sets of extended episodes as input Q , K and V .

Original: Further, it matches each query in Q with a list of keys in K and returns the weighted sum of corresponding values.
Revised: Further, it matches each query in Q with a list of keys in K and returns the weighted sum of corresponding values.

Original: To show the importance of the transformer module, we compare it with two other integration approaches (i
Revised: To show the importance of the transformer module, we compare it with two other integration approaches (i

Original: [Table]
Revised: [Table]

Original: e. concatenating and averaging) to integrate the features of extended episodes.
Revised: e. concatenating and averaging) to integrate the features of extended episodes.

Original: The comparative results in Table 5 demonstrate that the integration transformer consistently performs better than the simply concatenating/averaging approaches.
Revised: The comparative results in Table 5 demonstrate that the integration transformer consistently performs better than the simply concatenating/averaging approaches.

Original: This suggests that the attention-based integration transformer is a better choice for designing the integration module.
Revised: This suggests that the attention-based integration transformer is a better choice for designing the integration module.

Original: [Table]
Revised: [Table]

Original: A. F EATURE V ISUALIZATIONS OFM ORE T EST E PISODES
Revised: A. F EATURE V ISUALIZATIONS OF T EST E PISODES

Original: In addition to the feature visualizations of test episodes in Figure 3, we provide more feature visualizations in Figure 5.
Revised: We provide the feature visualizations of test episodes in Figure 4.

Original: It can be seen that an integrated episode (the last one in each row) clearly has a better cluster data structure than the corresponding four extended episodes (the ﬁrst four ones in each row).
Revised: It can be seen that an integrated episode (the last one in each row) clearly has a better cluster data structure than the corresponding four extended episodes (the ﬁrst four ones in each row).

Original: This indicates that our transformer-based across-episode integration is indeed effective for few-shot classiﬁcation with self-supervision.
Revised: This indicates that our transformer-based across-episode integration is indeed effective for few-shot classiﬁcation with self-supervision.

Original: A. A TTENTION VISUALIZATION FOR T EST E PISODES
Revised: A. A TTENTION VISUALIZATION OF T EST E PISODES

Original: We present attention map visualization of two test episodes (left and right) in Figure 6.
Revised: We present attention map visualization of two test episodes (left and right) in Figure 5.

Original: Each average attention map is computed by averaging the attention map of all instances of a certain class.
Revised: Each average attention map is computed by averaging the attention map of all instances of a certain class.

Original: We can observe that: (1) The average attention maps from different classes vary signiﬁcantly, showing that the diverse semantics of different classes can be reﬂected by our attention-based integration transformer.
Revised: We can observe that: (1) The average attention maps from different classes vary signiﬁcantly, showing that the diverse semantics of different classes can be reﬂected by our attention-based integration transformer.

Original: (2) When the classes of two episodes overlap (e.g., ‘triﬂe’ and ‘dalmatian’), the average attention maps of an overlapped class from two episodes are similar, illustrating that our attention-based integration transformer can well capture the semantics of classes across episodes.
Revised: (2) When the classes of two episodes overlap (e.g., ‘triﬂe’ and ‘dalmatian’), the average attention maps of an overlapped class from two episodes are similar, illustrating that our attention-based integration transformer can well capture the semantics of classes across episodes.

Original: Attention: trifle cuirass bookshop nematode dalmatian
Revised: Attention: trifle cuirass bookshop nematode dalmatian

Original: Attention: dalmatian theater curtain trifle vase ant
Revised: Attention: dalmatian theater curtain trifle vase ant

Original: A. C OMPARISON WITH S IMPLE B ASELINE FOR SSL+FSL
Revised: A. C OMPARISON WITH S IMPLE B ASELINE FOR SSL+FSL

Original: We compare our IEPT with a simple baseline that trains the model with L aux + L inst and then makes inference by just averaging the outputs of different extended episodes.
Revised: We compare our IEPT with a simple baseline that trains the model with L aux + L inst and then makes inference by just averaging the outputs of different extended episodes.

Original: The results are shown in
Revised: The results are shown in

Original: Table 6.
Revised: Table 6.

Original: It can be observed that the performance of our IEPT is much more effective than that of simple integration, due to the extra use of L integ + L epis for FSL.
Revised: It can be observed that the performance of our IEPT is much more effective than that of simple integration, due to the extra use of L integ + L epis for FSL.

Original: [Table]
Revised: [Table]

Original: . D IFFERENT A LTERNATIVES OF S ELF -S UPERVISED L OSSES
Revised: . D IFFERENT A LTERNATIVES OF S ELF -S UPERVISED L OSSES

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we provide further ablation study regarding different alternatives of L epis and L inst .
Revised: we provide further ablation study regarding different alternatives of L epis and L inst .

Original: For the episode-level self-supervised loss L epis , we compare our implementation (using the KL loss between each distribution and the mean distribution) with that using a pairwise KL loss (i
Revised: For the episode-level self-supervised loss L epis , we compare our implementation (using the KL loss between each distribution and the mean distribution) with that using a pairwise KL loss (i

Original: [Table]
Revised: [Table]

Original: e., the KL loss between each pair of distributions).
Revised: e., the KL loss between each pair of distributions).

Original: For the instance-level self-supervised loss L inst , we compare our implementation (using the rotation prediction loss) with the recent self-supervised learning technique (Chen et al., 2020).
Revised: For the instance-level self-supervised loss L inst , we compare our implementation (using the rotation prediction loss) with the recent self-supervised learning technique (Chen et al., 2020).

Original: We observe that our implementation achieves slight performance improvements over those using the pairwise KL loss or the contrastive learning loss.
Revised: We observe that our implementation achieves slight performance improvements over those using the pairwise KL loss or the contrastive learning loss.

Original: [Table]
Revised: [Table]

Original: A. A PPLICATION OF IEPT TO O PTIMIZATION -B ASED M ETHOD MAML
Revised: A. A PPLICATION OF IEPT TO O PTIMIZATION -B ASED M ETHOD MAML

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we show the results obtained by applying our IEPT to the optimization-based model
Revised: we show the results obtained by applying our IEPT to the optimization-based model

Original: MAML (Finn et al., 2017)
Revised: MAML (Finn et al., 2017)

Original: [Table]
Revised: [Table]

Original: We use Conv4-64 as the feature extractor.
Revised: We use Conv4-64 as the feature extractor.

Original: We can see that our IEPT brings 0.7%-2.1% improvements to MAML.
Revised: We can see that our IEPT brings 0.7%-2.1% improvements to MAML.

Original: This further shows the ﬂexibility (as well as effectiveness) of our IEPT for FSL
Revised: This further shows the ﬂexibility (as well as effectiveness) of our IEPT for FSL

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: A. H YPER - PARAMETER S ENSITIVITY T EST
Revised: A. H YPER - PARAMETER S ENSITIVITY T EST

Original: We select the hyper-parameters w 1 , w 2 and w 3 from the candidate set { 0 . 1 , 0 .
Revised: We select the hyper-parameters w 1 , w 2 and w 3 from the candidate set { 0 . 1 , 0 .

Original: and show the hyper-parameter analysis results in Figure 7.
Revised: and show the hyper-parameter analysis results in Figure 6.

Original: We ﬁnd that the performance of our IEPT is relatively stable.
Revised: We ﬁnd that the performance of our IEPT is relatively stable.

Original: Concretely, the performance of our IEPT is not sensitive to w 1 and w 2 with proper values, but too large w 1 (i.e. w 1 = 10 . 0 ) tends to cause obvious degradation, perhaps because the FSL task is biased by the rotation prediction loss
Revised: Concretely, the performance of our IEPT is not sensitive to w 1 and w 2 with proper values, but too large w 1 (i.e. w 1 = 10 . 0 ) tends to cause obvious degradation, perhaps because the FSL task is biased by the rotation prediction loss

Original: [Figure]
Revised: [Figure]