Original: T O L EARN E FFECTIVE F EATURES : U NDERSTANDING THE T ASK -S PECIFIC A DAPTATION OF MAML
Revised: T O L EARN E FFECTIVE F EATURES : U NDERSTANDING THE T ASK -S PECIFIC A DAPTATION OF MAML

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Meta learning, an effective way for learning unseen tasks with few samples, is an important research area in machine learning.
Revised: Meta learning, an effective way for learning unseen tasks with few samples, is an important research area in machine learning.

Original: Model Agnostic MetaLearning (MAML)
Revised: Model Agnostic MetaLearning (MAML)

Original: (Finn et al.
Revised: (Finn et al.

Original: (2017)) is one of the most well-known gradientbased meta learning algorithms, that learns the meta-initialization through the inner and outer optimization loop.
Revised: (2017)) is one of the most well-known gradientbased meta learning algorithms, that learns the meta-initialization through the inner and outer optimization loop.

Original: The inner loop is to perform fast adaptation in several gradient update steps with the support datapoints, while the outer loop to generalize the updated model to the query datapoints.
Revised: The inner loop is to perform fast adaptation in several gradient update steps with the support datapoints, while the outer loop to generalize the updated model to the query datapoints.

Original: Recently, it has been argued that instead of rapid learning and adaptation, the learned meta-initialization through MAML has already absorbed the high-quality features prior, where the task-speciﬁc head at training facilitates the feature learning.
Revised: Recently, it has been argued that instead of rapid learning and adaptation, the learned meta-initialization through MAML has already absorbed the high-quality features prior, where the task-speciﬁc head at training facilitates the feature learning.

Original: In this work, we investigate the impact of the task-speciﬁc adaptation of MAML and discuss the general formula for other gradient-based and metric-based meta-learning approaches.
Revised: In this work, we investigate the impact of the task-speciﬁc adaptation of MAML and discuss the general formula for other gradient-based and metric-based meta-learning approaches.

Original: From our analysis, we further devise the Random Decision Planes (RDP) algorithm to ﬁnd a suitable linear classiﬁer without any gradient descent step and the
Revised: From our analysis, we further devise the Random Decision Planes (RDP) algorithm to ﬁnd a suitable linear classiﬁer without any gradient descent step and the

Original: Meta Contrastive Learning (MCL) algorithm to exploit the inter-samples relationship instead of the expensive inner-loop adaptation.
Revised: Meta Contrastive Learning (MCL) algorithm to exploit the inter-samples relationship instead of the expensive inner-loop adaptation.

Original: We conduct sufﬁcient experiments on various datasets to explore our proposed algorithms.
Revised: We conduct sufﬁcient experiments on various datasets to explore our proposed algorithms.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Few-shot learning, aiming to learn from few labelled examples, is a great challenge for modern machine learning systems.
Revised: Few-shot learning, aiming to learn from few labelled examples, is a great challenge for modern machine learning systems.

Original: Meta learning, an effective way for tracking this challenge, enables the model to learn general knowledge across a distribution of tasks.
Revised: Meta learning, an effective way for tracking this challenge, enables the model to learn general knowledge across a distribution of tasks.

Original: Various ideas of meta learning have been proposed to address the few-shot problems.
Revised: Various ideas of meta learning have been proposed to address the few-shot problems.

Original: Gradient-based meta learning (Finn et al.
Revised: Gradient-based meta learning (Finn et al.

Original: (2017); Nichol et al.
Revised: (2017); Nichol et al.

Original: (2018)) learns the meta-parameters that can be quickly adapted to new tasks by few gradient descent steps.
Revised: (2018)) learns the meta-parameters that can be quickly adapted to new tasks by few gradient descent steps.

Original: Metric-based meta learning (Koch et al.
Revised: Metric-based meta learning (Koch et al.

Original: ; Vinyals et al.
Revised: ; Vinyals et al.

Original: Snell et al.
Revised: Snell et al.

Original: (2017)) proposes to learn a metric space by comparing different datapoints.
Revised: (2017)) proposes to learn a metric space by comparing different datapoints.

Original: Memorybased meta learning (Santoro et al.
Revised: Memorybased meta learning (Santoro et al.

Original: (2016)) can rapidly assimilate new data and leverage the stored information to make predictions.
Revised: (2016)) can rapidly assimilate new data and leverage the stored information to make predictions.

Original: Model Agnostic Meta-Learning (MAML)
Revised: Model Agnostic Meta-Learning (MAML)

Original: (Finn et al.
Revised: (Finn et al.

Original: (2017)) is one of the most well-known gradient-based meta learning algorithms, that learns the meta-initialization parameters through the inner optimization loop and the outer optimization loop.
Revised: (2017)) is one of the most well-known gradient-based meta learning algorithms, that learns the meta-initialization parameters through the inner optimization loop and the outer optimization loop.

Original: For a given task, the inner loop is to perform fast adaptation in several gradient descent steps with the support datapoints, while the outer loop to generalize the updated model to the query datapoints.
Revised: For a given task, the inner loop is to perform fast adaptation in several gradient descent steps with the support datapoints, while the outer loop to generalize the updated model to the query datapoints.

Original: With the learned meta-initialization, the model can be quickly adapted to the unseen tasks with few labelled samples.
Revised: With the learned meta-initialization, the model can be quickly adapted to the unseen tasks with few labelled samples.

Original: Following the
Revised: Following the

Original: MAML algorithm, many signiﬁcant variants (Finn et al.
Revised: MAML algorithm, many signiﬁcant variants (Finn et al.

Original: ; Rusu et al.
Revised: ; Rusu et al.

Original: ; Oreshkin et al.
Revised: ; Oreshkin et al.

Original: (2018); Bertinetto et al.
Revised: (2018); Bertinetto et al.

Original: ; Lee et al.
Revised: ; Lee et al.

Original: (2019b)) are studied under the few-shot setting.
Revised: (2019b)) are studied under the few-shot setting.

Original: To understand how the MAML works, Raghu et al.
Revised: To understand how the MAML works, Raghu et al.

Original: (2019) conduct a series of experiments and claim that rather than rapid learning and adaptation, the learned meta-initialization has already absorbed the high-quality features prior, thus the representations after ﬁne-tuning are almost the same for the coming unseen tasks.
Revised: (2019) conduct a series of experiments and claim that rather than rapid learning and adaptation, the learned meta-initialization has already absorbed the high-quality features prior, thus the representations after ﬁne-tuning are almost the same for the coming unseen tasks.

Original: Also, the task speciﬁc head of MAML at training facilitates the learning of better features.
Revised: Also, the task speciﬁc head of MAML at training facilitates the learning of better features.

Original: In this paper, we further design more representative experiments and present a formal argument to explain the importance of the task speciﬁc adaptation.
Revised: In this paper, we further design more representative experiments and present a formal argument to explain the importance of the task speciﬁc adaptation.

Original: Actually, the multistep task-speciﬁc adaptation, making the body and head have similar classiﬁcation capabilities, can provide better gradient descent direction for the features learning of body.
Revised: Actually, the multistep task-speciﬁc adaptation, making the body and head have similar classiﬁcation capabilities, can provide better gradient descent direction for the features learning of body.

Original: We also notice that for both the gradient-based methods (e.g. MAML (Finn et al.
Revised: We also notice that for both the gradient-based methods (e.g. MAML (Finn et al.

Original: , MetaOpNet (Lee et al.
Revised: , MetaOpNet (Lee et al.

Original: (2019b)))
Revised: (2019b)))

Original: and metric-based methods (e.g. Prototypical Networks (Snell et al.
Revised: and metric-based methods (e.g. Prototypical Networks (Snell et al.

Original: that attempt to learn a task-speciﬁc head using the support datapoints, the adaptation is a common mode for features learning of body but varied in different methods.
Revised: that attempt to learn a task-speciﬁc head using the support datapoints, the adaptation is a common mode for features learning of body but varied in different methods.

Original: Based on our analysis, we ﬁrst propose a new training paradigm to ﬁnd a decision plane (linear classiﬁer) for guidance with no gradient descent step during the inner loop and get more supporting conclusions.
Revised: Based on our analysis, we ﬁrst propose a new training paradigm to ﬁnd a decision plane (linear classiﬁer) for guidance with no gradient descent step during the inner loop and get more supporting conclusions.

Original: Moreover, we devise another training paradigm that removes the inner loop and trains the model with only the query datapoints.
Revised: Moreover, we devise another training paradigm that removes the inner loop and trains the model with only the query datapoints.

Original: Speciﬁcally, inspired by contrastive representation learning (Oord et al.
Revised: Speciﬁcally, inspired by contrastive representation learning (Oord et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: ; He et al.
Revised: ; He et al.

Original: , we exploit the inter-samples relationship of query set to ﬁnd a guidance for the body across different tasks.
Revised: , we exploit the inter-samples relationship of query set to ﬁnd a guidance for the body across different tasks.

Original: This meta contrastive learning algorithm even achieves competitive results comparable to some state-of-the-art methods.
Revised: This meta contrastive learning algorithm even achieves competitive results comparable to some state-of-the-art methods.

Original: In total, our contributions can be listed as follows:
Revised: In total, our contributions can be listed as follows:

Original: We present sufﬁcient experiments and formal argument to explore the impact of the taskspeciﬁc adaptation for body features learning and discuss the general formula for other gradient-based and metric-based meta-learning approaches.
Revised: We present sufﬁcient experiments and formal argument to explore the impact of the taskspeciﬁc adaptation for body features learning and discuss the general formula for other gradient-based and metric-based meta-learning approaches.

Original: We devise a training algorithm to obtain a decision plane with no gradient descent step during the inner loop, named as Random Decision Planes (RDP), and get more supporting conclusions.
Revised: We devise a training algorithm to obtain a decision plane with no gradient descent step during the inner loop, named as Random Decision Planes (RDP), and get more supporting conclusions.

Original: Unlike prior gradient-based methods, we propose the Meta Contrastive Learning (MCL) algorithm to exploit the inter-samples relations instead of training a task-speciﬁc head during the inner loop.
Revised: Unlike prior gradient-based methods, we propose the Meta Contrastive Learning (MCL) algorithm to exploit the inter-samples relations instead of training a task-speciﬁc head during the inner loop.

Original: Even without the task-speciﬁc adaptation for guidance, our algorithm still achieve better results with even less computation costs.
Revised: Even without the task-speciﬁc adaptation for guidance, our algorithm still achieve better results with even less computation costs.

Original: We empirically shows the effectiveness of the proposed algorithm with different backbones on four benchmark datasets: miniImageNet (Vinyals et al.
Revised: We empirically shows the effectiveness of the proposed algorithm with different backbones on four benchmark datasets: miniImageNet (Vinyals et al.

Original: , tieredImageNet (Ren et al.
Revised: , tieredImageNet (Ren et al.

Original: , CIFAR-FS (Bertinetto et al.
Revised: , CIFAR-FS (Bertinetto et al.

Original: and FC100 (Oreshkin et al.
Revised: and FC100 (Oreshkin et al.

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: MAML (Finn et al.
Revised: MAML (Finn et al.

Original: (2017)) is a highly inﬂuential gradient-based meta learning algorithm for fewshot learning.
Revised: (2017)) is a highly inﬂuential gradient-based meta learning algorithm for fewshot learning.

Original: The amazing experiment results on several public few-shot datasets have proved its effectiveness.
Revised: The amazing experiment results on several public few-shot datasets have proved its effectiveness.

Original: Following the core idea of MAML, there are numerous works to handle the data insufﬁciency problem in few-shot learning.
Revised: Following the core idea of MAML, there are numerous works to handle the data insufﬁciency problem in few-shot learning.

Original: Some works (Oreshkin et al.
Revised: Some works (Oreshkin et al.

Original: ; Vuorio et al.
Revised: ; Vuorio et al.

Original: (2019)) introduce the task-dependent representations via conditioning the feature extractor on the speciﬁc task to improve the performance.
Revised: (2019)) introduce the task-dependent representations via conditioning the feature extractor on the speciﬁc task to improve the performance.

Original: Sun et al.
Revised: Sun et al.

Original: (2019) also employ the meta-learned scaling and shifting parameters for transferring from another large-scale dataset.
Revised: (2019) also employ the meta-learned scaling and shifting parameters for transferring from another large-scale dataset.

Original: Others (Grant et al.
Revised: Others (Grant et al.

Original: Finn et al.
Revised: Finn et al.

Original: ; Lee et al.
Revised: ; Lee et al.

Original: (2019a)) study this problem from the perspective of Bayesian approach.
Revised: (2019a)) study this problem from the perspective of Bayesian approach.

Original: Unlike prior methods, we provide two training paradigms, one with no gradient descent step during the inner loop and another removing the inner loop and exploiting the inter-sample relations for training.
Revised: Unlike prior methods, we provide two training paradigms, one with no gradient descent step during the inner loop and another removing the inner loop and exploiting the inter-sample relations for training.

Original: Recent works also explore the key factors that makes the meta-learned model perform better than others at few-shot tasks.
Revised: Recent works also explore the key factors that makes the meta-learned model perform better than others at few-shot tasks.

Original: Chen et al.
Revised: Chen et al.

Original: (2019) discovers that a deeper backbone has a large effect on the success of meta learning algorithm, while Goldblum et al.
Revised: (2019) discovers that a deeper backbone has a large effect on the success of meta learning algorithm, while Goldblum et al.

Original: (2020) ﬁnds that the meta learning tends to cluster object classes more tightly in feature space for those methods that ﬁx the backbone during the inner loop (Bertinetto et al.
Revised: (2020) ﬁnds that the meta learning tends to cluster object classes more tightly in feature space for those methods that ﬁx the backbone during the inner loop (Bertinetto et al.

Original: ; Rusu et al.
Revised: ; Rusu et al.

Original: A very recent work (Raghu et al.
Revised: A very recent work (Raghu et al.

Original: (2019)) argues that the meta-trained model can be applied to new task due to the high-quality features prior learned by the meta-initialized parameters rather than rapid learning.
Revised: (2019)) argues that the meta-trained model can be applied to new task due to the high-quality features prior learned by the meta-initialized parameters rather than rapid learning.

Original: In this paper, we further study the impact of the task-speciﬁc adaptation for feature learning.
Revised: In this paper, we further study the impact of the task-speciﬁc adaptation for feature learning.

Original: Based on the analysis, we devise two algorithms, Random Decision Planes (RDP) and Meta Contrastive Learning (MCL) requiring less computation cost but still with competitive performance.
Revised: Based on the analysis, we devise two algorithms, Random Decision Planes (RDP) and Meta Contrastive Learning (MCL) requiring less computation cost but still with competitive performance.

Original:  M ODEL -A GNOSTIC M ETA L EARNING (MAML)
Revised:  M ODEL -A GNOSTIC M ETA L EARNING (MAML)

Original: The MAML aims to learn the meta-initialized parameters θ for the coming unseen tasks through the inner optimization loop and the outer optimization loop.
Revised: The MAML aims to learn the meta-initialized parameters θ for the coming unseen tasks through the inner optimization loop and the outer optimization loop.

Original: Under the N -way- K -shot setting, for a task T b sampled from the task distribution P ( T ) , we have a support set of N × K examples T s b and
Revised: Under the N -way- K -shot setting, for a task T b sampled from the task distribution P ( T ) , we have a support set of N × K examples T s b and

Original: [Table]
Revised: [Table]

Original: a query set T qb , where N is the number of sampled class and K is the number of instances for each class.
Revised: a query set T qb , where N is the number of sampled class and K is the number of instances for each class.

Original: During the inner loop, with the support set T sb , we perform fast adaptation in several gradient descent steps and obtain the task-speciﬁc parameters θ tT b where t is the number of gradient descent steps, given by:
Revised: During the inner loop, with the support set T sb , we perform fast adaptation in several gradient descent steps and obtain the task-speciﬁc parameters θ tT b where t is the number of gradient descent steps, given by:

Original: [Equation]
Revised: [Equation]

Original: (1) where α is the step size for inner loop and L T
Revised: (1) where α is the step size for inner loop and L T

Original: sb ( θ t − 1 T b ) denoted as the loss on the support set T sb after t − 1 steps.
Revised: sb ( θ t − 1 T b ) denoted as the loss on the support set T sb after t − 1 steps.

Original: With the query set T qb , we compute the meta loss on the task-speciﬁc parameters θ tT b and backward to update the meta-initialized parameters θ , given by
Revised: With the query set T qb , we compute the meta loss on the task-speciﬁc parameters θ tT b and backward to update the meta-initialized parameters θ , given by

Original: [Equation]
Revised: [Equation]

Original: where β is the learning rate and B is the number of sampled tasks in a batch.
Revised: where β is the learning rate and B is the number of sampled tasks in a batch.

Original:  I MPACT OF TASK - SPECIFIC ADAPTATION 4. T HE MULTI - STEP TASK - SPECIFIC ADAPTATION IS IMPORTANT .
Revised:  I MPACT OF TASK - SPECIFIC ADAPTATION 4. T HE MULTI - STEP TASK - SPECIFIC ADAPTATION IS IMPORTANT .

Original: To explore the effectiveness of MAML, Raghu et al.
Revised: To explore the effectiveness of MAML, Raghu et al.

Original: (2019) have conducted sufﬁcient experiments, indicating that the network body (the representation layers) has already absorbed the high-quality features prior.
Revised: (2019) have conducted sufﬁcient experiments, indicating that the network body (the representation layers) has already absorbed the high-quality features prior.

Original: During meta-testing, instead of ﬁne tuning on the network head (the classiﬁer), simply building the prototypes with the support set can achieve comparable performance to MAML.
Revised: During meta-testing, instead of ﬁne tuning on the network head (the classiﬁer), simply building the prototypes with the support set can achieve comparable performance to MAML.

Original: Raghu et al.
Revised: Raghu et al.

Original: (2019) also shows that the task speciﬁcity of head at training can facilitate feature learning and ensure good representation learning in the network body.
Revised: (2019) also shows that the task speciﬁcity of head at training can facilitate feature learning and ensure good representation learning in the network body.

Original: In our work, we show that besides the task speciﬁcity of head, the multi-step adaptation is also essential, and further study the role of network body and head during meta-training.
Revised: In our work, we show that besides the task speciﬁcity of head, the multi-step adaptation is also essential, and further study the role of network body and head during meta-training.

Original: We devise several methods using different training regimes: (1) Multi-Task , where all the tasks simply share one common head and the model is trained in a traditional way without inner loop adaptation; (2) Multi-Head , where different tasks are equipped with different heads for task speciﬁcity and the model is trained in a traditional way without inner loop adaptation; (3) Almost No Inner Loop (ANIL) , where the network body is ﬁxed during the inner loop; (4) Body Outer Loop, Head Inner Loop (BOHI) , where the network body is updated only by the outer loop and the head is adapted only during the inner loop, making the head’s meta-initialized parameters unchanged.
Revised: We devise several methods using different training regimes: (1) Multi-Task , where all the tasks simply share one common head and the model is trained in a traditional way without inner loop adaptation; (2) Multi-Head , where different tasks are equipped with different heads for task speciﬁcity and the model is trained in a traditional way without inner loop adaptation; (3) Almost No Inner Loop (ANIL) , where the network body is ﬁxed during the inner loop; (4) Body Outer Loop, Head Inner Loop (BOHI) , where the network body is updated only by the outer loop and the head is adapted only during the inner loop, making the head’s meta-initialized parameters unchanged.

Original: More algorithms’ details can be found in Appendix B, and implementation details can be found in Appendix C.1.
Revised: More algorithms’ details can be found in Appendix B, and implementation details can be found in Appendix C.1.

Original: Following Raghu et al.
Revised: Following Raghu et al.

Original: (2019), we employ the cosine similarities between prototypes and the query datapoints to evaluate the quality of features learned.
Revised: (2019), we employ the cosine similarities between prototypes and the query datapoints to evaluate the quality of features learned.

Original: As Table 1 shows, even equipped with taskspeciﬁc head, the Multi-Head training still performs worse than the standard MAML algorithm by
Revised: As Table 1 shows, even equipped with taskspeciﬁc head, the Multi-Head training still performs worse than the standard MAML algorithm by

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: a large margin, indicating the multi-step adaptation of MAML is helpful for features learning.
Revised: a large margin, indicating the multi-step adaptation of MAML is helpful for features learning.

Original: The results of Multi-Head and Multi-Task show the importance of multi-step task-speciﬁc adaptation.
Revised: The results of Multi-Head and Multi-Task show the importance of multi-step task-speciﬁc adaptation.

Original: As the results shown in Table
Revised: As the results shown in Table

Original: [Figure]
Revised: [Figure]

Original: the ANIL training remains effective comparable to the standard
Revised: the ANIL training remains effective comparable to the standard

Original: MAML algorithm, indicating that the task-speciﬁc adaptation of network body is unnecessary to learn good features.
Revised: MAML algorithm, indicating that the task-speciﬁc adaptation of network body is unnecessary to learn good features.

Original: More interestingly, the BOHI training that keeps the meta-initialization of head unchanged even performs better than MAML, further demonstrating that good features learning depends on the multi-step task-speciﬁc adaptation of head during inner loop more than updating the meta-initialization of head in outer loop.
Revised: More interestingly, the BOHI training that keeps the meta-initialization of head unchanged even performs better than MAML, further demonstrating that good features learning depends on the multi-step task-speciﬁc adaptation of head during inner loop more than updating the meta-initialization of head in outer loop.

Original: Also, the ANIL and BOHI have similar performance, indicating that compared with learned prior knowledge in head, the inner loop adaptation, as a guidance, contributes more to the features learning.
Revised: Also, the ANIL and BOHI have similar performance, indicating that compared with learned prior knowledge in head, the inner loop adaptation, as a guidance, contributes more to the features learning.

Original: More experimental results can be found in
Revised: More experimental results can be found in

Original: Appendix C.2.
Revised: Appendix C.2.

Original: 4. W HY IS MULTI - STEP TASK - SPECIFIC ADAPTATION IMPORTANT ?
Revised: 4. W HY IS MULTI - STEP TASK - SPECIFIC ADAPTATION IMPORTANT ?

Original: Having observed that the MAML algorithm outperforms the Multi-Task training by a large margin and the multi-step task-speciﬁc adaptation is important for features learning, we extend our analysis to explore the reason why the inner loop adaptation is essential for MAML at different stages of meta training.
Revised: Having observed that the MAML algorithm outperforms the Multi-Task training by a large margin and the multi-step task-speciﬁc adaptation is important for features learning, we extend our analysis to explore the reason why the inner loop adaptation is essential for MAML at different stages of meta training.

Original: Speciﬁcally, we freeze the initialized MAML model and model at 5,000 iterations, sample validation tasks from the task distribution, and record the test accuracy of model in different inner loop steps.
Revised: Speciﬁcally, we freeze the initialized MAML model and model at 5,000 iterations, sample validation tasks from the task distribution, and record the test accuracy of model in different inner loop steps.

Original: Both the body accuracy based on prototypes construction and head accuracy based on ﬁne-tuning are given in Figure 1 and Figure 2, where “Task ID” stands for different tasks.
Revised: Both the body accuracy based on prototypes construction and head accuracy based on ﬁne-tuning are given in Figure 1 and Figure 2, where “Task ID” stands for different tasks.

Original: As the results shows, at different stages of meta training, the head accuracy increases signiﬁcantly in the ﬁrst few adaptation steps since the model has learnt the correspondence between sample and label.
Revised: As the results shows, at different stages of meta training, the head accuracy increases signiﬁcantly in the ﬁrst few adaptation steps since the model has learnt the correspondence between sample and label.

Original: However, at the beginning of training, there is only a small improvement on the body accuracy after ﬁrst adaptation step.
Revised: However, at the beginning of training, there is only a small improvement on the body accuracy after ﬁrst adaptation step.

Original: In Figure 2, as the model converges, the body accuracy even 
Revised: In Figure 2, as the model converges, the body accuracy even decreases in the ﬁrst few adaptation steps.

Revised: In the following steps, with the task-speciﬁc adaptation of head, the network body then learns better representations, further demonstrating that the multi-

Revised: Network Body f θ , Learning Rate β , Task Distribution P ( T )

Original: Perform the Gram-Schmidt method on random metrices to get the classiﬁer set P
Revised: Perform the Gram-Schmidt method on random metrices to get the classiﬁer set P

Original: = { W i } n p i
Revised: = { W i } n p i

Original: =
Revised: =

Original: while not done do
Revised: while not done do

Original: Sample a batch of tasks { T b } Bb =1 , where T b ∼ P ( T ) for b ∈ { 1 , ..., B } do
Revised: Sample a batch of tasks { T b } Bb =1 , where T b ∼ P ( T ) for b ∈ { 1 , ..., B } do

Original: Sample the support set T sb =
Revised: Sample the support set T sb =

Original: { ( x si , y si ) } N × K i = and query set T qb =
Revised: { ( x si , y si ) } N × K i = and query set T qb =

Original: { ( x qi , y qi ) } N × K i = from task T b .
Revised: { ( x qi , y qi ) } N × K i = from task T b .

Original: for each sample x in { T sb , T qb } do z
Revised: for each sample x in { T sb , T qb } do z

Original: [Equation]
Revised: [Equation]

Original: end for deﬁne CrossEntropyLoss
Revised: end for deﬁne CrossEntropyLoss

Original: ( H , D ) as the cross entropy loss on the features representations set D with head H .
Revised: ( H , D ) as the cross entropy loss on the features representations set D with head H .

Original: [Equation]
Revised: [Equation]

Original: end for
Revised: end for

Original: [Equation]
Revised: [Equation]

Original: Bb [Equation] end while decreases in the ﬁrst few adaptation steps. In the following steps, with the task-speciﬁc adaptation of head, the network body then learns better representations, further demonstrating that the multistep task-speciﬁc adaptation, making the body and head have similar classiﬁcation capabilities, can be regarded as a guidance to provide better gradient descent direction for the feature learning of body.
Revised: end while step task-speciﬁc adaptation, making the body and head have similar classiﬁcation capabilities, can be regarded as a guidance to provide better gradient descent direction for the feature learning of body.

Original: To understand this intuitive argument better, we consider a sample ( x , y ) for few-shot classiﬁcation where the cross entropy loss is employed, formulated as:
Revised: To understand this intuitive argument better, we consider a sample ( x , y ) for few-shot classiﬁcation where the cross entropy loss is employed, formulated as:

Original: [Equation]
Revised: [Equation]

Original: where { w 1 , w 2 , ..., w k } is the weights of the classiﬁer head, h is the body representation of x .
Revised: where { w 1 , w 2 , ..., w k } is the weights of the classiﬁer head, h is the body representation of x .

Original: The gradients of loss L c with respect to the body representation h are denoted by,
Revised: The gradients of loss L c with respect to the body representation h are denoted by,

Original: [Equation]
Revised: [Equation]

Original: where ¯ w is exactly the weighted average of the weights { w 1 , w 2 , ..., w k } .
Revised: where ¯ w is exactly the weighted average of the weights { w 1 , w 2 , ..., w k } .

Original: As shown in Equation 4, a reasonable direction for the network body to minimize the target loss L c is to make the representation h closer to the corresponding class weight w y , given by h = h
Revised: As shown in Equation 4, a reasonable direction for the network body to minimize the target loss L c is to make the representation h closer to the corresponding class weight w y , given by h = h

Original: + λ ( w y − ¯ w ) .
Revised: + λ ( w y − ¯ w ) .

Original: As the model converges, in the ﬁrst few adaptation steps, there is a signiﬁcant margin between the performance of head and body, and the classiﬁer weights contain little knowledge about correspondence between samples and labels and differences between different classes.
Revised: As the model converges, in the ﬁrst few adaptation steps, there is a signiﬁcant margin between the performance of head and body, and the classiﬁer weights contain little knowledge about correspondence between samples and labels and differences between different classes.

Original: With the low-performance head, this updating rule for body may lead to a decline in the quality of features, which also explains why the simpler BOHI, ANIL even performs better than MAML in Table 1.
Revised: With the low-performance head, this updating rule for body may lead to a decline in the quality of features, which also explains why the simpler BOHI, ANIL even performs better than MAML in Table 1.

Original: After several adaptation steps during the inner loop, the body then receives the useful guidance for features learning from the taskspeciﬁc head since w y can better express its corresponding class.
Revised: After several adaptation steps during the inner loop, the body then receives the useful guidance for features learning from the taskspeciﬁc head since w y can better express its corresponding class.

Original: The formulation above shows that the multi-step task-speciﬁc adaptation, making the body and head have similar classiﬁcation capabilities, can provide better gradient descent direction for the features learning of body.
Revised: The formulation above shows that the multi-step task-speciﬁc adaptation, making the body and head have similar classiﬁcation capabilities, can provide better gradient descent direction for the features learning of body.

Original: 4. T ASK - SPECIFIC ADAPTATION IN OTHER META - LEARNING ALGORITHMS
Revised: 4. T ASK - SPECIFIC ADAPTATION IN OTHER META - LEARNING ALGORITHMS

Original: Having noticed that the multi-step task-speciﬁc adaptation of MAML, which promotes the performance of head, can facilitate the features learning of body.
Revised: Having noticed that the multi-step task-speciﬁc adaptation of MAML, which promotes the performance of head, can facilitate the features learning of body.

Original: It works similarly for other gradientbased methods that use end-to-end ﬁne-tuning, such as Reptile (Nichol et al.
Revised: It works similarly for other gradientbased methods that use end-to-end ﬁne-tuning, such as Reptile (Nichol et al.

Original: In the case of meta-learning methods that ﬁx the network body and only update the head during the inner loop, such as MetaOpNet (Lee et al. (2019b)) and R2-D2 (Bertinetto et al.
Revised: In the case of meta-learning methods that ﬁx the network body and only update the head during the inner loop, such as MetaOpNet (Lee et al. (2019b)) and R2-D2 (Bertinetto et al.

Original: , the convex optimization of head also aims to provide a classiﬁer with better classiﬁcation capabilities.
Revised: , the convex optimization of head also aims to provide a classiﬁer with better classiﬁcation capabilities.

Original: For metric-based 
Revised: For metric-based methods, such as Prototypical Networks (Snell et al.

Original: [Figure]
Revised: (2017)), the adaptation of head is actually conducted through the nearest neighbor algorithm. In conclusion, the adaptation is a common mode but

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: methods, such as Prototypical Networks (Snell et al.

Original: (2017)), the adaptation of head is actually conducted through the nearest neighbor algorithm.

Original: In conclusion, the adaptation is a common mode but varied in different methods.

Original: These meta-learning algorithms reveal a general formula that the inner loop is for building a task-speciﬁc head that matches the classiﬁcation capabilities of body and the outer loop for task-independent features learning.

Original:  T HE R ANDOM D ECISION P LANES A LGORITHM
Revised:  T HE R ANDOM D ECISION P LANES A LGORITHM

Original: As discussed above, the multi-step adaptation based on gradient descent during the inner loop aims to provide guidance for features learning of body.
Revised: As discussed above, the multi-step adaptation based on gradient descent during the inner loop aims to provide guidance for features learning of body.

Original: From this consideration, we suppose that if a suitable linear classiﬁer is given, the feature learning can be facilitated even without gradient descent during the inner loop.
Revised: From this consideration, we suppose that if a suitable linear classiﬁer is given, the feature learning can be facilitated even without gradient descent during the inner loop.

Original: From this consideration, we devise such an algorithm named Random Decision
Revised: From this consideration, we devise such an algorithm named Random Decision

Original: Planes (RDP), where a classiﬁer is chosen from a predeﬁned set P according to the target loss on the support set.
Revised: Planes (RDP), where a classiﬁer is chosen from a predeﬁned set P according to the target loss on the support set.

Original: The predeﬁned set of classiﬁer P consists of n p different orthonormal matrices that are generated through the Gram-Schmidt method from random matrices.
Revised: The predeﬁned set of classiﬁer P consists of n p different orthonormal matrices that are generated through the Gram-Schmidt method from random matrices.

Original: During the inner loop, without gradient descent, we directly choose a most suitable classiﬁer as the network head which minimizes the cross entropy loss on the support set.
Revised: During the inner loop, without gradient descent, we directly choose a most suitable classiﬁer as the network head which minimizes the cross entropy loss on the support set.

Original: In the outer loop, we compute the loss based on the chosen head and run backward to update the network body.
Revised: In the outer loop, we compute the loss based on the chosen head and run backward to update the network body.

Original: A formal description of RDP is presented in Algorithm 1.
Revised: A formal description of RDP is presented in Algorithm 1.

Original: The implementation details can be found in Appendix C.1.
Revised: The implementation details can be found in Appendix C.1.

Original: The overall evaluation results on three datasets are presented in Table 2.
Revised: The overall evaluation results on three datasets are presented in Table 2.

Original: Note that we also remove the head and construct the prototypes from the body network f θ for predictions during meta-testing.
Revised: Note that we also remove the head and construct the prototypes from the body network f θ for predictions during meta-testing.

Original: The proposed RDP algorithm performs comparably to the standard MAML method on three datasets, especially on the FC100 dataset.
Revised: The proposed RDP algorithm performs comparably to the standard MAML method on three datasets, especially on the FC100 dataset.

Original: Without any task-speciﬁc adaptation for the network body, a best performing classiﬁer chosen from a set of randomly generated subspaces can also be a guidance to facilitate the features learning, further suggesting that a head with better classiﬁcation capabilities, is key factor to learn good representations even if the chosen approximate head performs worse than a gradient-based head, and the main purpose of task-speciﬁc adaptation is to adjust the lowperformance head for features learning of body.
Revised: Without any task-speciﬁc adaptation for the network body, a best performing classiﬁer chosen from a set of randomly generated subspaces can also be a guidance to facilitate the features learning, further suggesting that a head with better classiﬁcation capabilities, is key factor to learn good representations even if the chosen approximate head performs worse than a gradient-based head, and the main purpose of task-speciﬁc adaptation is to adjust the lowperformance head for features learning of body.

Original: Input: Network Body f θ , Projection Layer g φ , Learning Rate β ,
Revised: Input: Network Body f θ , Projection Layer g φ , Learning Rate β ,

Original: Constant τ , Task Distribution P ( T ) while not done do
Revised: Constant τ , Task Distribution P ( T ) while not done do

Original: Sample batch
Revised: Sample batch

Original: ) for
Revised: ) for

Original: ., B } do
Revised: ., B } do

Original: Sample the query
Revised: Sample the query

Original: Ni for i ∈ { 1 , ..., 2 N } do z
Revised: Ni for i ∈ { 1 , ..., 2 N } do z

Original: i
Revised: i

Original: =
Revised: =

Original: g
Revised: g

Original: φ ( f θ ( x qi )) end for for i ∈ { 1 , ..., 2 N } and j ∈ { 1 , ..., 2 N } do s i,j = z (cid:62)
Revised: φ ( f θ ( x qi )) end for for i ∈ { 1 , ..., 2 N } and j ∈ { 1 , ..., 2 N } do s i,j = z (cid:62)

Original: i z j / ( (cid:107) z
Revised: i z j / ( (cid:107) z

Original: i (cid:107)(cid:107)
Revised: i (cid:107)(cid:107)

Original: z j (cid:107) ) end for deﬁne l ( i, j )
Revised: z j (cid:107) ) end for deﬁne l ( i, j )

Original: =
Revised: =

Original: − log( exp( s i,j /τ ) (cid:80) 

Revised: − log( exp( s i,j /τ ) (cid:80)

Original: Nk
Revised: Nk

Original: =
Revised: =

Original: [ k (cid:54)
Revised: [ k (cid:54)

Original: =
Revised: =

Original: i ] exp( s i,k /τ ) )
Revised: i ] exp( s i,k /τ ) )

Original: L b = 12 N (cid:80) Nk
Revised: L b = 12 N (cid:80) Nk

Original: =

Revised: =

Original: [ l (2 k − 1 , 2 k ) + l (2 k, 2 k − 1)] end for θ = θ − β ∇ θ 1 B (cid:80) Bb =
Revised: [ l (2 k − 1 , 2 k ) + l (2 k, 2 k − 1)] end for θ = θ − β ∇ θ 1 B (cid:80) Bb =

Original: 1 L b φ = φ − β ∇ φ 1 B (cid:80) Bb =
Revised: 1 L b φ = φ − β ∇ φ 1 B (cid:80) Bb =

Original: 1 L b end while
Revised: 1 L b end while

Original: Also, we conduct experiments to explore the impact of the number of decision planes.
Revised: Also, we conduct experiments to explore the impact of the number of decision planes.

Original: Results are shown in Figure 3 on two datasets.
Revised: Results are shown in Figure 3 on two datasets.

Original: With a small set of decision planes, it can be more difﬁcult to ﬁnd a suitable head to guide the features learning, while with enough decision planes, the performance then reaches the upper limit.
Revised: With a small set of decision planes, it can be more difﬁcult to ﬁnd a suitable head to guide the features learning, while with enough decision planes, the performance then reaches the upper limit.

Original:  T HE M ETA C ONTRASTIVE L EARNING A LGORITHM
Revised:  T HE M ETA C ONTRASTIVE L EARNING A LGORITHM

Original: We have already seen that the multi-step task-speciﬁc adaptation to improve the classiﬁer head can essentially facilitate the features learning of body.
Revised: We have already seen that the multi-step task-speciﬁc adaptation to improve the classiﬁer head can essentially facilitate the features learning of body.

Original: In total, prior gradient-based methods based on the cross-entropy loss proposes to learn the correspondence between samples and assigned labels for different tasks, thus requiring the task-speciﬁc adaptation for the classiﬁer head during inner loop.
Revised: In total, prior gradient-based methods based on the cross-entropy loss proposes to learn the correspondence between samples and assigned labels for different tasks, thus requiring the task-speciﬁc adaptation for the classiﬁer head during inner loop.

Original: Since the task-speciﬁc head also serves for features learning of body, we wonder if we can remove the inner loop or adaptation, and make full use of the labels information in other way to be a guidance for features learning.
Revised: Since the task-speciﬁc head also serves for features learning of body, we wonder if we can remove the inner loop or adaptation, and make full use of the labels information in other way to be a guidance for features learning.

Original: From this consideration and inspired by recent works (Chen et al.
Revised: From this consideration and inspired by recent works (Chen et al.

Original: ; He et al.
Revised: ; He et al.

Original: about self-supervised contrastive learning, we further devise the
Revised: about self-supervised contrastive learning, we further devise the

Original: Meta Contrastive Learning (MCL) algorithm that directly removes the inner loop and exploits the inter-sample relationship with only the query set.
Revised: Meta Contrastive Learning (MCL) algorithm that directly removes the inner loop and exploits the inter-sample relationship with only the query set.

Original: Speciﬁcally, rather than using cross entropy loss for task-speciﬁc adaptation, we simply impose that normalized representations from the same class are closer together than representations from different classes.
Revised: Speciﬁcally, rather than using cross entropy loss for task-speciﬁc adaptation, we simply impose that normalized representations from the same class are closer together than representations from different classes.

Original: For N -way few-shot learning, we sample two examples per class to build the query set.
Revised: For N -way few-shot learning, we sample two examples per class to build the query set.

Original: Next, for a given anchor example, the meta contrastive loss pulls it closer to the point of same class while pushes the anchor farther away from the negative examples of other classes.
Revised: Next, for a given anchor example, the meta contrastive loss pulls it closer to the point of same class while pushes the anchor farther away from the negative examples of other classes.

Original: Following Chen et al.
Revised: Following Chen et al.

Original: (2020), we also employ a small neural network projection layer that maps the body features to the space where contrastive loss is applied.
Revised: (2020), we also employ a small neural network projection layer that maps the body features to the space where contrastive loss is applied.

Original: A formal description of MCL is presented in Algorithm 2.
Revised: A formal description of MCL is presented in Algorithm 2.

Original: The implementation details can be found in Appendix C.1.
Revised: The implementation details can be found in Appendix C.1.

Original: During meta-testing, we discard the projection layer g φ and construct the prototypes from the body network f θ for predictions.
Revised: During meta-testing, we discard the projection layer g φ and construct the prototypes from the body network f θ for predictions.

Original: The overall evaluation results on the MiniImageNet, TieredImageNet and FC100 datasets are presented in Table 3.
Revised: The overall evaluation results on the MiniImageNet, TieredImageNet and FC100 datasets are presented in Table 3.

Original: Note that TADAM (Oreshkin et al.
Revised: Note that TADAM (Oreshkin et al.

Original: (2018)) employs a extra task embedding network (TEN) block to predict element-wise scale and shift vectors, and
Revised: (2018)) employs a extra task embedding network (TEN) block to predict element-wise scale and shift vectors, and

Original: MetaOpNet (Lee et al.
Revised: MetaOpNet (Lee et al.

Original: (2019b)) proposes to learn a linear support vector machine (SVM) as classiﬁer head during the inner loop.
Revised: (2019b)) proposes to learn a linear support vector machine (SVM) as classiﬁer head during the inner loop.

Original: Unlike those methods, our MCL method is arguably simpler.
Revised: Unlike those methods, our MCL method is arguably simpler.

Original: By exploiting the relationship between different samples, we are able to remove the inner loop which contains a complex adaptation process, and devise a contrastive loss to train the network body directly.
Revised: By exploiting the relationship between different samples, we are able to remove the inner loop which contains a complex adaptation process, and devise a contrastive loss to train the network body directly.

Original: As the results shows, our method outperforms almost previous well-designed methods and
Revised: As the results shows, our method outperforms almost previous well-designed methods and

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: also achieves results comparable to MetaOpNet.
Revised: also achieves results comparable to MetaOpNet.

Original: More experimental results and time-efﬁciency analysis can be found in Appendix C.2 and C.3.
Revised: More experimental results and time-efﬁciency analysis can be found in Appendix C.2 and C.3.

Original: We also study the impact of the projection layer g φ .
Revised: We also study the impact of the projection layer g φ .

Original: Figure 4 shows the evaluation results with different output dimensions.
Revised: Figure 4 shows the evaluation results with different output dimensions.

Original: Note that “None” means that there is no projection layer for loss computation.
Revised: Note that “None” means that there is no projection layer for loss computation.

Original: As the results show, for a deeper ResNet12 backbone, the projection layer facilitates the features learning a lot ( > 6% for 5-shot, > 5% for 1-shot).
Revised: As the results show, for a deeper ResNet12 backbone, the projection layer facilitates the features learning a lot ( > 6% for 5-shot, > 5% for 1-shot).

Original: We conjecture that the projection layer is trained to extract task-speciﬁc information useful for the contrastive loss, while the body representations h learns more general information.
Revised: We conjecture that the projection layer is trained to extract task-speciﬁc information useful for the contrastive loss, while the body representations h learns more general information.

Original: More analysis can be found in Appendix C.4.
Revised: More analysis can be found in Appendix C.4.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, based on the hypothesis that feature reuse is the dominant factor for the success of
Revised: In this paper, based on the hypothesis that feature reuse is the dominant factor for the success of

Original: MAML algorithm, we further study the impact of task-speciﬁc adaptation and devise several training regimes including BOHI, Multi-Head and so on.
Revised: MAML algorithm, we further study the impact of task-speciﬁc adaptation and devise several training regimes including BOHI, Multi-Head and so on.

Original: Also, we provide a more formal argument from the perspective of gradient descent optimization.
Revised: Also, we provide a more formal argument from the perspective of gradient descent optimization.

Original: Based on analysis above, we ﬁnd that the multistep task-speciﬁc adaptation, making the body and head have similar classiﬁcation capabilities, can provide better gradient descent direction for the features learning of body.
Revised: Based on analysis above, we ﬁnd that the multistep task-speciﬁc adaptation, making the body and head have similar classiﬁcation capabilities, can provide better gradient descent direction for the features learning of body.

Original: We further connect our results to other meta-learning algorithm, showing the adaptation is a common mode but varied in different methods.
Revised: We further connect our results to other meta-learning algorithm, showing the adaptation is a common mode but varied in different methods.

Original: From our consideration, we devise the RDP algorithm where a suitable linear classiﬁer is chosen without gradient descent and get more supporting conclusions.
Revised: From our consideration, we devise the RDP algorithm where a suitable linear classiﬁer is chosen without gradient descent and get more supporting conclusions.

Original: We also build the
Revised: We also build the

Original: MCL algorithm that removes the inner loop and exploit the inter-sample relationship, and achieve results comparable to some state-of-the-art methods.
Revised: MCL algorithm that removes the inner loop and exploit the inter-sample relationship, and achieve results comparable to some state-of-the-art methods.

Original: A F EW - SHOT I MAGE C LASSIFICATION D ATASETS
Revised: A F EW - SHOT I MAGE C LASSIFICATION D ATASETS

Original: In this section, we introduce four benchmark datasets often used for few-shot image classiﬁcation: the miniImageNet (Vinyals et al.
Revised: In this section, we introduce four benchmark datasets often used for few-shot image classiﬁcation: the miniImageNet (Vinyals et al.

Original: , tieredImageNet (Ren et al.
Revised: , tieredImageNet (Ren et al.

Original: CIFAR-FS (Bertinetto et al.
Revised: CIFAR-FS (Bertinetto et al.

Original: and FC100 (Oreshkin et al.
Revised: and FC100 (Oreshkin et al.

Original: The miniImageNet (Vinyals et al.
Revised: The miniImageNet (Vinyals et al.

Original: dataset is standard benchmark for few-shot image classiﬁcation, comprises 100 classes randomly chosen from the original ImageNet (Russakovsky et al.
Revised: dataset is standard benchmark for few-shot image classiﬁcation, comprises 100 classes randomly chosen from the original ImageNet (Russakovsky et al.

Original: (2015)) dataset, where 64 classes is used for meta-training, 16 classes for meta-validation andclasses for meta-testing.
Revised: (2015)) dataset, where 64 classes is used for meta-training, 16 classes for meta-validation andclasses for meta-testing.

Original: Each class contains 600 images of size 84 × 84.
Revised: Each class contains 600 images of size 84 × 84.

Original: Since the original class splits are unavailable, we use the commonly-used split proposed in Ravi & Larochelle (2016).
Revised: Since the original class splits are unavailable, we use the commonly-used split proposed in Ravi & Larochelle (2016).

Original: The tieredImageNet (Ren et al.
Revised: The tieredImageNet (Ren et al.

Original: dataset is another larger subset of ImageNet (Russakovsky et al.
Revised: dataset is another larger subset of ImageNet (Russakovsky et al.

Original: This dataset contains 608 classes that are grouped into 34 high-level categories, where 20 categories (351 classes) are used for meta-training, 6 categories (97 classes) for meta-validation and 8 categories(160 classes) for meta-testing.
Revised: This dataset contains 608 classes that are grouped into 34 high-level categories, where 20 categories (351 classes) are used for meta-training, 6 categories (97 classes) for meta-validation and 8 categories(160 classes) for meta-testing.

Original: All images are also size of 84 × 84.
Revised: All images are also size of 84 × 84.

Original: The CIFAR-FS (Bertinetto et al.
Revised: The CIFAR-FS (Bertinetto et al.

Original: dataset is a few-shot image classiﬁcation benchmark, consisting of all 100 classes from CIFAR-100 (Krizhevsky et al.
Revised: dataset is a few-shot image classiﬁcation benchmark, consisting of all 100 classes from CIFAR-100 (Krizhevsky et al.

Original: These classes are randomly split into 64, 16, and 20 separately for meta-training, meta-validation and meta-testing.
Revised: These classes are randomly split into 64, 16, and 20 separately for meta-training, meta-validation and meta-testing.

Original: Each class contains 600 images of size 32 × 32.
Revised: Each class contains 600 images of size 32 × 32.

Original: The FC100 (Oreshkin et al.
Revised: The FC100 (Oreshkin et al.

Original: dataset is another benchmark derived from CIFAR100 (Krizhevsky et al.
Revised: dataset is another benchmark derived from CIFAR100 (Krizhevsky et al.

Original: This dataset comprises 100 classes that are grouped into 20 highlevel categories, where 12 categories (60 classes) are used for meta 4 categories (20 classes for  images 
Revised: This dataset comprises 100 classes that are grouped into 20 highlevel categories, where 12 categories (60 classes) are used for meta-training, 4 categories (20 classes) for meta-validation and 4 categories (20 classes) for meta-testing. Each class contains 600 images of size 32 × 32.

Original: [Table]
Revised: [Table]

Original: B M ORE D ETAILS ABOUT A LGORITHMS
Revised: B M ORE D ETAILS ABOUT A LGORITHMS

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we provide further details about the training regimes and algorithms mentioned above.
Revised: we provide further details about the training regimes and algorithms mentioned above.

Original: Note that we denote the meta parameters of the network as θ in previous sections.
Revised: Note that we denote the meta parameters of the network as θ in previous sections.

Original: Considering the network is composed of the body (feature extractor) and head (classiﬁer)
Revised: Considering the network is composed of the body (feature extractor) and head (classiﬁer)

Original: [Table]
Revised: [Table]

Original: we further rewrite θ as θ =
Revised: we further rewrite θ as θ =

Original: [ θ f , θ c ]
Revised: [ θ f , θ c ]

Original: [Table]
Revised: [Table]

Original: where θ f , θ c is the parameters of body and head respectively.
Revised: where θ f , θ c is the parameters of body and head respectively.

Original: For a given task
Revised: For a given task

Original: T b = { T sb
Revised: T b = { T sb

Original: [Table]
Revised: [Table]

Original: T qb }
Revised: T qb }

Original: [Table]
Revised: [Table]

Original: the meta-initialization updating of MAML can be expressed as follows:
Revised: the meta-initialization updating of MAML can be expressed as follows:

Original: [Equation]
Revised: [Equation]

Original: where α is the step size of the inner loop, β is the learning rate.
Revised: where α is the step size of the inner loop, β is the learning rate.

Original: In our work, we devise several methods using different training regimes including Multi-Task, Multi-Head, Almost
Revised: In our work, we devise several methods using different training regimes including Multi-Task, Multi-Head, Almost

Original: No Inner
Revised: No Inner

Original: Loop (ANIL) and Body Outer Loop, Head Inner Loop (BOHI) to study the role of network body and head during meta-training.
Revised: Loop (ANIL) and Body Outer Loop, Head Inner Loop (BOHI) to study the role of network body and head during meta-training.

Original: The updating rules of Multi-Task can be expressed as follows
Revised: The updating rules of Multi-Task can be expressed as follows

Original: [Equation]
Revised: [Equation]

Original: For different tasks, the Multi-head has different heads for task speciﬁcity, given by
Revised: For different tasks, the Multi-head has different heads for task speciﬁcity, given by

Original: [Equation]
Revised: [Equation]

Original: where θ T b c is the speciﬁc parameters for task T b .
Revised: where θ T b c is the speciﬁc parameters for task T b .

Original: The network body is ﬁxed during the inner loop for
Revised: The network body is ﬁxed during the inner loop for

Original: ANIL, given by
Revised: ANIL, given by

Original: [Equation]
Revised: [Equation]

Original: For BOHI, the network body is updated only by the outer loop and the head is adapted only during the inner loop, making the head’s meta-initialized parameters unchanged, given by
Revised: For BOHI, the network body is updated only by the outer loop and the head is adapted only during the inner loop, making the head’s meta-initialized parameters unchanged, given by

Original: [Equation]
Revised: [Equation]

Original: C M ORE E XPERIMENTAL D ETAILS C. I MPLEMENTATION D ETAILS
Revised: C M ORE E XPERIMENTAL D ETAILS C. I MPLEMENTATION D ETAILS

Original: For all training regimes, RDP and MCL, we use the Adam optimizer with weight decay of 5e-4 and the learning rate is set to 1e-3.
Revised: For all training regimes, RDP and MCL, we use the Adam optimizer with weight decay of 5e-4 and the learning rate is set to 1e-3.

Original: For 4-layer convolution network with 64 ﬁlters, we ﬂatten the output feature map of the network body, and obtain 1600-d features for miniImageNet and tieredImageNet, while 256-d features for CIFAR-FS and FC100.
Revised: For 4-layer convolution network with 64 ﬁlters, we ﬂatten the output feature map of the network body, and obtain 1600-d features for miniImageNet and tieredImageNet, while 256-d features for CIFAR-FS and FC100.

Original: For ResNet12 network, we employ a global max pooling layer on the output feature map of the network body, and obtain 512-d features for four public datasets.
Revised: For ResNet12 network, we employ a global max pooling layer on the output feature map of the network body, and obtain 512-d features for four public datasets.

Original: During meta-training, we adopt horizontal ﬂip, random crop and color (brightness,
Revised: During meta-training, we adopt horizontal ﬂip, random crop and color (brightness,

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: contrast, and saturation) jitter data augmentation as proposed in Gidaris & Komodakis (2018); Qiao et al.
Revised: contrast, and saturation) jitter data augmentation as proposed in Gidaris & Komodakis (2018); Qiao et al.

Original: We train all models 100 epochs and take 500 batches per epoch.
Revised: We train all models 100 epochs and take 500 batches per epoch.

Original: For MAML, BOHI and ANIL, both models are trained using 5 gradient steps of size α = 0 .
Revised: For MAML, BOHI and ANIL, both models are trained using 5 gradient steps of size α = 0 .

Original: 01 for Conv

Revised: 01 for Conv

Original: and α = 0 .for
Revised: and α = 0 .for

Original: ResNet12.
Revised: ResNet12.

Original: For the Random Decision Planes algorithm, the number of decision planes n p is set to 64.
Revised: For the Random Decision Planes algorithm, the number of decision planes n p is set to 64.

Original: For the Meta Contrastive Learning (MCL) algorithm, we apply a two-layer nonlinear projection layer with hidden size of 512.
Revised: For the Meta Contrastive Learning (MCL) algorithm, we apply a two-layer nonlinear projection layer with hidden size of 512.

Original: Also, the query datapoints come from 10 different classes for each sampled task
Revised: Also, the query datapoints come from 10 different classes for each sampled task

Original: [Table]
Revised: [Table]

Original: which is helpful for accelerating model convergence.
Revised: which is helpful for accelerating model convergence.

Original: C. M ORE RESULTS FOR BOHI, ANIL, MAML, MCL
Revised: C. M ORE RESULTS FOR BOHI, ANIL, MAML, MCL

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we provide complete experimental results for BOHI, ANIL, MAML and MCL with different backbones on four datasets.
Revised: we provide complete experimental results for BOHI, ANIL, MAML and MCL with different backbones on four datasets.

Original: The complete results on four datasets are presented in Table 4,
Revised: The complete results on four datasets are presented in Table 4,

Original: Table 5, Table 6 and Table 7 respectively.
Revised: Table 5, Table 6 and Table 7 respectively.

Original: The results can further verify our description mentioned above.
Revised: The results can further verify our description mentioned above.

Original: Good features learning depends on the multi-step task-speciﬁc adaptation of head during the inner loop more than updating the meta-initialization of head in outer loop.
Revised: Good features learning depends on the multi-step task-speciﬁc adaptation of head during the inner loop more than updating the meta-initialization of head in outer loop.

Original: With the lowperformance head, the update of body may even lead to a decline in the quality of features.
Revised: With the lowperformance head, the update of body may even lead to a decline in the quality of features.

Original: In addition, the results on four datasets further demonstrate the effectiveness of our proposed MCL algorithm.
Revised: In addition, the results on four datasets further demonstrate the effectiveness of our proposed MCL algorithm.

Original: C. T HE TIME - EFFICIENCY ANALYSIS FOR BOHI, ANIL, MAML, MCL
Revised: C. T HE TIME - EFFICIENCY ANALYSIS FOR BOHI, ANIL, MAML, MCL

Original: It is obvious that ANIL, BOHI and MCL can speeds up training.
Revised: It is obvious that ANIL, BOHI and MCL can speeds up training.

Original: The results about the comparison of computation time are presented in Table 8.
Revised: The results about the comparison of computation time are presented in Table 8.

Original: We implement our methods based on PyTorch and the
Revised: We implement our methods based on PyTorch and the

Original: [Table]
Revised: [Table]

Original: training of models is run on two NVIDIA 1080Ti GPU.
Revised: training of models is run on two NVIDIA 1080Ti GPU.

Original: Notice that our MCL can run much faster than BOHI and ANIL while achieves better evaluation results.
Revised: Notice that our MCL can run much faster than BOHI and ANIL while achieves better evaluation results.

Original: The training speedups also illustrate the signiﬁcant computational beneﬁt of MCL and prove its effectiveness.
Revised: The training speedups also illustrate the signiﬁcant computational beneﬁt of MCL and prove its effectiveness.

Original: [Table]
Revised: [Table]

Original: C. A BOUT THE PROJECTION LAYER OF MCL
Revised: C. A BOUT THE PROJECTION LAYER OF MCL

Original: We have found that with a deeper backbone, the features learning can be facilitated a lot by the projection layer.
Revised: We have found that with a deeper backbone, the features learning can be facilitated a lot by the projection layer.

Original: We further evaluate the quality of features extracted by the network body and the projection layer.
Revised: We further evaluate the quality of features extracted by the network body and the projection layer.

Original: The evaluation results are given in Table 9.
Revised: The evaluation results are given in Table 9.

Original: Even if the contrastive loss is applied to the projection layer, the network body learns better and general representations.
Revised: Even if the contrastive loss is applied to the projection layer, the network body learns better and general representations.

Original: We conjecture that during the meta-training, the projection layer may absorb more task-speciﬁc information while the backbone tends to learn task-independent representations.
Revised: We conjecture that during the meta-training, the projection layer may absorb more task-speciﬁc information while the backbone tends to learn task-independent representations.