Original: MAML IS A N OISY C ONTRASTIVE L EARNER IN C LASSIFICATION
Revised: MAML IS A N OISY C ONTRASTIVE L EARNER IN C LASSIFICATION

Original: A BSTRACT
Revised: A BSTRACT

Original: Model-agnostic meta-learning (MAML) is one of the most popular and widely adopted meta-learning algorithms, achieving remarkable success in various learning problems.
Revised: Model-agnostic meta-learning (MAML) is one of the most popular and widely adopted meta-learning algorithms, achieving remarkable success in various learning problems.

Original: Yet, with the unique design of nested inner-loop and outer-loop updates, which govern the task-specific and meta-model-centric learning, respectively, the underlying learning objective of MAML remains implicit, impeding a more straightforward understanding of it.
Revised: Yet, with the unique design of nested inner-loop and outer-loop updates, which govern the task-specific and meta-model-centric learning, respectively, the underlying learning objective of MAML remains implicit, impeding a more straightforward understanding of it.

Original: In this paper, we provide a new perspective of the working mechanism of MAML.
Revised: In this paper, we provide a new perspective of the working mechanism of MAML.

Original: We discover that MAML is analogous to a meta-learner using a supervised contrastive objective in classification.
Revised: We discover that MAML is analogous to a meta-learner using a supervised contrastive objective in classification.

Original: The query features are pulled towards the support features of the same class and against those of different classes.
Revised: The query features are pulled towards the support features of the same class and against those of different classes.

Original: Such contrastiveness is experimentally verified via an analysis based on the cosine similarity.
Revised: Such contrastiveness is experimentally verified via an analysis based on the cosine similarity.

Original: Moreover, we reveal that vanilla MAML has an undesirable interference term originating from the random initialization and the cross-task interaction.
Revised: Moreover, we reveal that vanilla MAML has an undesirable interference term originating from the random initialization and the cross-task interaction.

Original: We thus propose a simple but effective technique, the zeroing trick, to alleviate the interference.
Revised: We thus propose a simple but effective technique, the zeroing trick, to alleviate the interference.

Original: Extensive experiments are conducted on both mini-ImageNet and Omniglot datasets to validate the consistent improvement brought by our proposed method.
Revised: Extensive experiments are conducted on both mini-ImageNet and Omniglot datasets to validate the consistent improvement brought by our proposed method.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Humans can learn from very few samples.
Revised: Humans can learn from very few samples.

Original: They can readily establish their cognition and understanding of novel tasks, environments, or domains even with very limited experience in the corresponding circumstances.
Revised: They can readily establish their cognition and understanding of novel tasks, environments, or domains even with very limited experience in the corresponding circumstances.

Original: Meta-learning , a subfield of machine learning, aims at equipping machines with such capacity to accommodate new scenarios effectively (Vilalta & Drissi, 2002; Grant et al., 2018).
Revised: Meta-learning , a subfield of machine learning, aims at equipping machines with such capacity to accommodate new scenarios effectively (Vilalta & Drissi, 2002; Grant et al., 2018).

Original: Machines learn to extract task-agnostic information so that their performance on unseen tasks can be improved (Hospedales et al., 2020).
Revised: Machines learn to extract task-agnostic information so that their performance on unseen tasks can be improved (Hospedales et al., 2020).

Original: One highly influential meta-learning algorithm is Model Agnostic Meta-Learning (MAML)
Revised: One highly influential meta-learning algorithm is Model Agnostic Meta-Learning (MAML)

Original: (Finn et al., 2017), which has inspired numerous follow-up extensions (Nichol et al., 2018; Rajeswaran et al., 2019; Liu et al., 2019; Finn et al., 2019; Jamal & Qi, 2019; Javed & White, 2019).
Revised: (Finn et al., 2017), which has inspired numerous follow-up extensions (Nichol et al., 2018; Rajeswaran et al., 2019; Liu et al., 2019; Finn et al., 2019; Jamal & Qi, 2019; Javed & White, 2019).

Original: MAML estimates a set of model parameters such that an adaptation of the model to a new task only requires some updates to those parameters.
Revised: MAML estimates a set of model parameters such that an adaptation of the model to a new task only requires some updates to those parameters.

Original: We take the few-shot classification task as an example to review the algorithmic procedure of MAML.
Revised: We take the few-shot classification task as an example to review the algorithmic procedure of MAML.

Original: A few-shot classification problem refers to classifying samples from some classes (i.e. query data) after seeing a few examples per class (i.e. support data).
Revised: A few-shot classification problem refers to classifying samples from some classes (i.e. query data) after seeing a few examples per class (i.e. support data).

Original: In a meta-learning scenario, we consider a distribution of tasks, where each task is a few-shot classification problem and different tasks have different target classes.
Revised: In a meta-learning scenario, we consider a distribution of tasks, where each task is a few-shot classification problem and different tasks have different target classes.

Original: MAML aims to meta-train the base-model based on training tasks (i.e., the meta-training dataset) and evaluate the performance of the base-model on the testing tasks sampled from a held-out unseen dataset (i.e. the meta-testing dataset).
Revised: MAML aims to meta-train the base-model based on training tasks (i.e., the meta-training dataset) and evaluate the performance of the base-model on the testing tasks sampled from a held-out unseen dataset (i.e. the meta-testing dataset).

Original: In meta-training, MAML follows a bi-level optimization scheme composed of the inner loop and the outer loop, as shown in Appendix A (please refer to Section 2 for detailed definition).
Revised: In meta-training, MAML follows a bi-level optimization scheme composed of the inner loop and the outer loop, as shown in Appendix A (please refer to Section 2 for detailed definition).

Original: In the inner loop (also known as fast adaptation ), the base-model θ is updated to θ ′ using the support set.
Revised: In the inner loop (also known as fast adaptation ), the base-model θ is updated to θ ′ using the support set.

Original: In the outer loop, a loss is evaluated on θ ′ using the query set, and its gradient is computed with respect to θ to update the base-model.
Revised: In the outer loop, a loss is evaluated on θ ′ using the query set, and its gradient is computed with respect to θ to update the base-model.

Original: Since the outer loop requires computing the gradient of gradient (as the update in the inner loop is included in the entire computation graph), it is called second-order MAML (SOMAML).
Revised: Since the outer loop requires computing the gradient of gradient (as the update in the inner loop is included in the entire computation graph), it is called second-order MAML (SOMAML).

Original: To prevent computing the Hessian matrix, Finn et al.
Revised: To prevent computing the Hessian matrix, Finn et al.

Original: (2017) propose first-order
Revised: (2017) propose first-order

Original: MAML
Revised: MAML

Original: FOMAML
Revised: FOMAML

Original: ) that uses
Revised: ) that uses

Original: the
Revised: the

Original: gradient computed
Revised: gradient computed

Original: with
Revised: with

Original: respect
Revised: respect

Original: to the
Revised: to the

Original: inner
Revised: inner

Original: -loop-updated
Revised: -loop-updated

Original: parameters
Revised: parameters

Original: θ
Revised: θ

Original: ′ to
Revised: ′ to

Original: update the base
Revised: update the base

Original: -model.
Revised: -model.

Original: The widely accepted intuition behind MAML is that the models are encouraged to learn generalpurpose representations which are broadly applicable not only to the seen tasks but also to novel tasks (Finn et al., 2017; Raghu et al., 2020;
Revised: The widely accepted intuition behind MAML is that the models are encouraged to learn generalpurpose representations which are broadly applicable not only to the seen tasks but also to novel tasks (Finn et al., 2017; Raghu et al., 2020;

Original: Goldblum et al., 2020).
Revised: Goldblum et al., 2020).

Original: Raghu et al.
Revised: Raghu et al.

Original: (2020) confirm this perspective by showing that during fast adaptation, the majority of changes being made are in the final linear layers.
Revised: (2020) confirm this perspective by showing that during fast adaptation, the majority of changes being made are in the final linear layers.

Original: In contrast, the convolution layers (as the feature encoder) remain almost static.
Revised: In contrast, the convolution layers (as the feature encoder) remain almost static.

Original: This implies that the models trained with MAML learn a good feature representation and that they only have to change the linear mapping from features to outputs during the fast adaptation.
Revised: This implies that the models trained with MAML learn a good feature representation and that they only have to change the linear mapping from features to outputs during the fast adaptation.

Original: Similar ideas of freezing feature extractors during the inner loop have also been explored (Lee et al., 2019; Bertinetto et al., 2019; Liu et al., 2020), and have been held as an assumption in theoretical works (Du et al., 2021; Tripuraneni et al., 2020; Chua et al., 2021).
Revised: Similar ideas of freezing feature extractors during the inner loop have also been explored (Lee et al., 2019; Bertinetto et al., 2019; Liu et al., 2020), and have been held as an assumption in theoretical works (Du et al., 2021; Tripuraneni et al., 2020; Chua et al., 2021).

Original: While this intuition sounds satisfactory, we step further and ask the following fundamental questions: (1) In what sense does MAML guide any model to learn general-purpose representations?
Revised: While this intuition sounds satisfactory, we step further and ask the following fundamental questions: (1) In what sense does MAML guide any model to learn general-purpose representations?

Original: (2) How do the inner and outer loops in the training mechanism of MAML collaboratively prompt to achieve so?
Revised: (2) How do the inner and outer loops in the training mechanism of MAML collaboratively prompt to achieve so?

Original: (3) What is the role of support and query data, and how do they interact with each other?
Revised: (3) What is the role of support and query data, and how do they interact with each other?

Original: In this paper, we answer these questions and give new insights on the working mechanism of MAML, which turns out to be closely connected to supervised contrastive learning (SCL) 2 .
Revised: In this paper, we answer these questions and give new insights on the working mechanism of MAML, which turns out to be closely connected to supervised contrastive learning (SCL) 2 .

Original: Here, we provide a sketch of our analysis in Figure 1.
Revised: Here, we provide a sketch of our analysis in Figure 1.

Original: We consider a setting of (a) a 5-way 1-shot paradigm of few-shot learning, (b) the mean square error (MSE) between the one-hot encoding of groundtruth label and the outputs as the objective function, and (c) MAML with a single inner-loop update.
Revised: We consider a setting of (a) a 5-way 1-shot paradigm of few-shot learning, (b) the mean square error (MSE) between the one-hot encoding of groundtruth label and the outputs as the objective function, and (c) MAML with a single inner-loop update.

Original: At the beginning of the inner loop, we set the linear layer w 0 to zero.
Revised: At the beginning of the inner loop, we set the linear layer w 0 to zero.

Original: Then, the inner loop update of w 0 is equivalent to adding the support features to w 0 .
Revised: Then, the inner loop update of w 0 is equivalent to adding the support features to w 0 .

Original: In the outer loop, the output of a query sample q 1 is actually the inner product between the query feature ϕ ( q 1 ) and all support features (the learning rate is omitted for now).
Revised: In the outer loop, the output of a query sample q 1 is actually the inner product between the query feature ϕ ( q 1 ) and all support features (the learning rate is omitted for now).

Original: As the groundtruth is an one-hot vector, the encoder is trained to either minimize the inner product between the query features and the support features (when they are from different classes, as shown in the green box), or to pull the inner product between the query features and the support features to 1 (when they have the same label, as shown in the red box).
Revised: As the groundtruth is an one-hot vector, the encoder is trained to either minimize the inner product between the query features and the support features (when they are from different classes, as shown in the green box), or to pull the inner product between the query features and the support features to 1 (when they have the same label, as shown in the red box).

Original: Therefore, the inner loop and the outer loop together manifest a SCL objective.
Revised: Therefore, the inner loop and the outer loop together manifest a SCL objective.

Original: Particularly, as the vanilla implementation of MAML uses non-zero (random) initialization for the linear layer, we will show such initialization leads to a noisy SCL objective which would impede the training.
Revised: Particularly, as the vanilla implementation of MAML uses non-zero (random) initialization for the linear layer, we will show such initialization leads to a noisy SCL objective which would impede the training.

Original: In this paper, we firstly review a formal definition of SCL, present a more general case of MAML with cross entropy loss in classification, and show the underlying learning protocol of vanilla MAML as an interfered SCL in Section 2.
Revised: In this paper, we firstly review a formal definition of SCL, present a more general case of MAML with cross entropy loss in classification, and show the underlying learning protocol of vanilla MAML as an interfered SCL in Section 2.

Original: We then experimentally verify the supervised contrastiveness of
Revised: We then experimentally verify the supervised contrastiveness of

Original: MAML and propose to mitigate the interference with our simple but effective technique of the zeroinitialization and zeroing trick (cf.
Revised: MAML and propose to mitigate the interference with our simple but effective technique of the zeroinitialization and zeroing trick (cf.

Original: Section 3).
Revised: Section 3).

Original: In summary, our main contributions are three-fold:
Revised: In summary, our main contributions are three-fold:

Original: • We show MAML is implicitly an SCL algorithm in classification and the noise comes from the randomly initialized linear layer and the cross-task interaction.
Revised: • We show MAML is implicitly an SCL algorithm in classification and the noise comes from the randomly initialized linear layer and the cross-task interaction.

Original: • We verify the inherent contrastiveness of MAML based on the cosine similarity analysis.
Revised: • We verify the inherent contrastiveness of MAML based on the cosine similarity analysis.

Original: • Our experiments show that applying the zeroing trick induces a notable improvement in testing accuracy during training and that that during meta-testing, a pronounced increase in the accuracy occurs when the zeroing trick is applied.
Revised: • Our experiments show that applying the zeroing trick induces a notable improvement in testing accuracy during training and that that during meta-testing, a pronounced increase in the accuracy occurs when the zeroing trick is applied.

Original:  W HY MAML IS I MPLICITLY A N OISY S UPERVISED C ONTRASTIVE A LGORITHM ? 2. P RELIMINARY : S UPERVISED C ONTRASTIVE L EARNING
Revised:  W HY MAML IS I MPLICITLY A N OISY S UPERVISED C ONTRASTIVE A LGORITHM ? 2. P RELIMINARY : S UPERVISED C ONTRASTIVE L EARNING

Original: In this work, we aim to bridge MAML and supervised contrastive learning (SCL) and attribute the success of MAML to SCL’s capacity in learning good representations.
Revised: In this work, we aim to bridge MAML and supervised contrastive learning (SCL) and attribute the success of MAML to SCL’s capacity in learning good representations.

Original: Thus, we would like to introduce SCL briefly.
Revised: Thus, we would like to introduce SCL briefly.

Original: [Figure]
Revised: [Figure]

Original: Thus,
Revised: Thus,

Original: MAML displays the characteristic of supervised contrastiveness.
Revised: MAML displays the characteristic of supervised contrastiveness.

Original: Besides, the support data acts as positive samples when the labels of support and query data match, and vice versa.
Revised: Besides, the support data acts as positive samples when the labels of support and query data match, and vice versa.

Original: Supervised contrastive learning, proposed by Khosla et al.
Revised: Supervised contrastive learning, proposed by Khosla et al.

Original: (2020), is a generalization of several metric learning algorithms, such as triplet loss and N-pair loss (Schroff et al., 2015; Sohn, 2016), and has shown the best performance in classification compared to SimCLR and CrossEntropy.
Revised: (2020), is a generalization of several metric learning algorithms, such as triplet loss and N-pair loss (Schroff et al., 2015; Sohn, 2016), and has shown the best performance in classification compared to SimCLR and CrossEntropy.

Original: In
Revised: In

Original: Khosla et al.
Revised: Khosla et al.

Original: , SCL is described as “contrasts the set of all samples from the same class as positives against the negatives from the remainder of the batch” and “embeddings from the same class are pulled closer together than embeddings from different classes.”
Revised: , SCL is described as “contrasts the set of all samples from the same class as positives against the negatives from the remainder of the batch” and “embeddings from the same class are pulled closer together than embeddings from different classes.”

Original: For a sample s , the label information is leveraged to indicate positive samples (i.e., samples having the same label as sample s ) and negative samples (i.e., samples having different labels to sample s ).
Revised: For a sample s , the label information is leveraged to indicate positive samples (i.e., samples having the same label as sample s ) and negative samples (i.e., samples having different labels to sample s ).

Original: The loss of SCL is designed to increase the similarity (or decrease the metric distance) of embeddings of positive samples and to reduce the similarity (or increase the metric distance) of embeddings of negative samples (Khosla et al., 2020).
Revised: The loss of SCL is designed to increase the similarity (or decrease the metric distance) of embeddings of positive samples and to reduce the similarity (or increase the metric distance) of embeddings of negative samples (Khosla et al., 2020).

Original: In essence, SCL combines supervised learning and contrastive learning and differs from supervised learning in that the loss contains a measurement of the similarity (or distance) between the embedding of a sample and embeddings of its positive/negative sample pairs.
Revised: In essence, SCL combines supervised learning and contrastive learning and differs from supervised learning in that the loss contains a measurement of the similarity (or distance) between the embedding of a sample and embeddings of its positive/negative sample pairs.

Original: Now we give a formal definition of SCL.
Revised: Now we give a formal definition of SCL.

Original: For a set of N samples drawn from a n -class dataset.
Revised: For a set of N samples drawn from a n -class dataset.

Original: Let i ∈ I = { 1 , ..., N } be the index of an arbitrary sample.
Revised: Let i ∈ I = { 1 , ..., N } be the index of an arbitrary sample.

Original: Let A ( i ) =
Revised: Let A ( i ) =

Original: I \ { i } , P
Revised: I \ { i } , P

Original: i ) be the set of indices of all positive samples of sample i , and N ( i )
Revised: i ) be the set of indices of all positive samples of sample i , and N ( i )

Original: =
Revised: =

Original: A ( i ) \ P ( i ) be the set of indices of all negative samples of sample i .
Revised: A ( i ) \ P ( i ) be the set of indices of all negative samples of sample i .

Original: Let z i indicates the embedding of sample i .
Revised: Let z i indicates the embedding of sample i .

Original: Definition 1 Let M sim be a measurement of similarity (e.g., inner product, cosine similarity).
Revised: Definition 1 Let M sim be a measurement of similarity (e.g., inner product, cosine similarity).

Original: Training algorithms that adopt loss of the following form belong to SCL:
Revised: Training algorithms that adopt loss of the following form belong to SCL:

Original: [Equation]
Revised: [Equation]

Original: where c − p,i < 0 and c + n,i > 0 for all n , p and i ; and c is a constant independent of samples.
Revised: where c − p,i < 0 and c + n,i > 0 for all n , p and i ; and c is a constant independent of samples.

Original: We further define that a training algorithm that follows Eq.
Revised: We further define that a training algorithm that follows Eq.

Original: (1) , but with either (a) c + n,i < 0 for some n, i or (b) c is a constant dependent of samples, belongs to noisy SCL .
Revised: (1) , but with either (a) c + n,i < 0 for some n, i or (b) c is a constant dependent of samples, belongs to noisy SCL .

Original: 2. P ROBLEM S ETUP
Revised: 2. P ROBLEM S ETUP

Original: We provide the detailed derivation to show that MAML is implicitly a noisy SCL, where we adopt the few-shot classification as the example application.
Revised: We provide the detailed derivation to show that MAML is implicitly a noisy SCL, where we adopt the few-shot classification as the example application.

Original: In this section, we focus on the meta-training period.
Revised: In this section, we focus on the meta-training period.

Original: Consider drawing a batch of tasks { T 1 , . .
Revised: Consider drawing a batch of tasks { T 1 , . .

Original: , T N batch } from a meta-training task distribution
Revised: , T N batch } from a meta-training task distribution

Original: D .
Revised: D .

Original: Each task T n contains a support set S n and a query set Q n , where S n = { ( s m , t m ) } N way × N shot m = ,
Revised: Each task T n contains a support set S n and a query set Q n , where S n = { ( s m , t m ) } N way × N shot m = ,

Original: Q n = { ( q m , u m ) } N way × N query m = , s m , q m ∈ R N in are data samples, and t m , u m ∈ { 1 , ..., N way } are labels.
Revised: Q n = { ( q m , u m ) } N way × N query m = , s m , q m ∈ R N in are data samples, and t m , u m ∈ { 1 , ..., N way } are labels.

Original: We denote N way the number of classes in each task, and { N shot , N query } respectively the number of support and query samples per class.
Revised: We denote N way the number of classes in each task, and { N shot , N query } respectively the number of support and query samples per class.

Original: The architecture of our base-model comprises of a convolutional encoder ϕ : R N in → R N f (parameterized by φ ), a fully connected linear head w ∈ R N f × N way , and a Softmax output layer, where N f is the dimension of the feature space.
Revised: The architecture of our base-model comprises of a convolutional encoder ϕ : R N in → R N f (parameterized by φ ), a fully connected linear head w ∈ R N f × N way , and a Softmax output layer, where N f is the dimension of the feature space.

Original: We denote the k th column of w as w k .
Revised: We denote the k th column of w as w k .

Original: Note that the base-model parameters θ consist of φ and w .
Revised: Note that the base-model parameters θ consist of φ and w .

Original: As shown in Appendix A, both FOMAML and SOMAML adopt a training strategy comprising the inner loop and the outer loop.
Revised: As shown in Appendix A, both FOMAML and SOMAML adopt a training strategy comprising the inner loop and the outer loop.

Original: At the beginning of a meta-training iteration, we sample N batch tasks.
Revised: At the beginning of a meta-training iteration, we sample N batch tasks.

Original: For each task T n , we perform inner loop updates using the inner loop loss (c.f. Eq.
Revised: For each task T n , we perform inner loop updates using the inner loop loss (c.f. Eq.

Original: (2)) evaluated on the support data, and then evaluate the outer loop loss (c.f. Eq.
Revised: (2)) evaluated on the support data, and then evaluate the outer loop loss (c.f. Eq.

Original: (3)) on the updated base-model using the query data.
Revised: (3)) on the updated base-model using the query data.

Original: In the i th step of the inner loop, the parameters { φ i − 1 , w i − 1 } are updated to { φ i , w i } using the multi-class cross entropy loss evaluated on the support dataset S n as
Revised: In the i th step of the inner loop, the parameters { φ i − 1 , w i − 1 } are updated to { φ i , w i } using the multi-class cross entropy loss evaluated on the support dataset S n as

Original: [Equation]
Revised: [Equation]

Original: After N step inner loop updates, we compute the outer loop loss using the query data Q n :
Revised: After N step inner loop updates, we compute the outer loop loss using the query data Q n :

Original: [Equation]
Revised: [Equation]

Original: Then, we sum up the outer loop losses of all tasks, and perform gradient descent to update the base-model’s initial parameters { φ 0 , w 0 } .
Revised: Then, we sum up the outer loop losses of all tasks, and perform gradient descent to update the base-model’s initial parameters { φ 0 , w 0 } .

Original: To show the supervised contrastiveness entailed in MAML, we adopt an assumption that the E ncoder ϕ is F rozen during the I nner L oop (the EFIL assumption ) and we discuss the validity of the assumption in Section 2.6.
Revised: To show the supervised contrastiveness entailed in MAML, we adopt an assumption that the E ncoder ϕ is F rozen during the I nner L oop (the EFIL assumption ) and we discuss the validity of the assumption in Section 2.6.

Original: Without loss of generality, we consider training models with MAML with
Revised: Without loss of generality, we consider training models with MAML with

Original: N batch =
Revised: N batch =

Original: 1 and N step = 1 , and we discuss the generalized version in Section 2.6.
Revised: 1 and N step = 1 , and we discuss the generalized version in Section 2.6.

Original: For simplicity, the k th element of model output
Revised: For simplicity, the k th element of model output

Original: [Equation]
Revised: [Equation]

Original: s (respectively q ) is denoted as s k (respectively q k ).
Revised: s (respectively q ) is denoted as s k (respectively q k ).

Original: 2. I NNER L OOP AND O UTER L OOP U PDATE OF L INEAR L AYER AND E NCODER
Revised: 2. I NNER L OOP AND O UTER L OOP U PDATE OF L INEAR L AYER AND E NCODER

Original: In this section, we primarily focus on the update of parameters in the case of FOMAML.
Revised: In this section, we primarily focus on the update of parameters in the case of FOMAML.

Original: The full derivation and discussion of SOMAML are provided in Appendix B.
Revised: The full derivation and discussion of SOMAML are provided in Appendix B.

Original: Inner loop update of the linear layer.
Revised: Inner loop update of the linear layer.

Original: In the inner loop, the linear layer w 0 is updated to w 1 with a learning rate η as shown in Eq.
Revised: In the inner loop, the linear layer w 0 is updated to w 1 with a learning rate η as shown in Eq.

Original: (4) in both FOMAML and SOMAML.
Revised: (4) in both FOMAML and SOMAML.

Original: In contrast to the example in Figure 1, the columns of the linear layer are added with the weighted sum of the features extracted from support samples (i.e., support features).
Revised: In contrast to the example in Figure 1, the columns of the linear layer are added with the weighted sum of the features extracted from support samples (i.e., support features).

Original: Compared to w k 0 , w k 1 is pushed towards the support features of the same class (i.e., class k ) with strength of 1 − s k , while being pulled away from the support features of different classes with strength of s k .
Revised: Compared to w k 0 , w k 1 is pushed towards the support features of the same class (i.e., class k ) with strength of 1 − s k , while being pulled away from the support features of different classes with strength of s k .

Original: [Equation]
Revised: [Equation]

Original: Outer loop update of the linear layer.
Revised: Outer loop update of the linear layer.

Original: In the outer loop, w 0 is updated using the query data with a learning rate ρ .
Revised: In the outer loop, w 0 is updated using the query data with a learning rate ρ .

Original: For FOMAML, the final linear layer is updated as follows.
Revised: For FOMAML, the final linear layer is updated as follows.

Original: [Equation]
Revised: [Equation]

Original: Note that the computation of q k requires the inner-loop updated w 1 .
Revised: Note that the computation of q k requires the inner-loop updated w 1 .

Original: Generally speaking, Eq.
Revised: Generally speaking, Eq.

Original: (5) resembles Eq.
Revised: (5) resembles Eq.

Original: It is obvious that, in the outer loop, the query features are added weightedly to the linear layer, and the strength of change relates to the output value.
Revised: It is obvious that, in the outer loop, the query features are added weightedly to the linear layer, and the strength of change relates to the output value.

Original: In other words, after the outer loop update, the linear layer memorizes the query features of current tasks.
Revised: In other words, after the outer loop update, the linear layer memorizes the query features of current tasks.

Original: This can cause a crosstask interference because in the next inner loop there would be additional inner products between the support features of the next tasks and the query features of the current tasks.
Revised: This can cause a crosstask interference because in the next inner loop there would be additional inner products between the support features of the next tasks and the query features of the current tasks.

Original: Outer loop update of the encoder.
Revised: Outer loop update of the encoder.

Original: Using the chain rule, the gradient of the outer loop loss with respect to φ (i.e., the parameters of the encoder) is given by ∂L
Revised: Using the chain rule, the gradient of the outer loop loss with respect to φ (i.e., the parameters of the encoder) is given by ∂L

Original: [Equation]
Revised: [Equation]

Original: φ, w
Revised: φ, w

Original: [Equation]
Revised: [Equation]

Original: ∂ϕ ( s ) ∂φ , where the second term can be neglected when FOMAML is considered.
Revised: ∂ϕ ( s ) ∂φ , where the second term can be neglected when FOMAML is considered.

Original: Below, we take a deeper look at the backpropagated error of one query data ( q, u ) ∼ Q .
Revised: Below, we take a deeper look at the backpropagated error of one query data ( q, u ) ∼ Q .

Original: The full derivation is provided in Appendix B.2.
Revised: The full derivation is provided in Appendix B.2.

Original: [Equation]
Revised: [Equation]

Original: 2. MAML IS A N OISY C ONTRASTIVE L EARNER
Revised: 2. MAML IS A N OISY C ONTRASTIVE L EARNER

Original: Reformulating the outer loop loss for the encoder as a noisy SCL loss.
Revised: Reformulating the outer loop loss for the encoder as a noisy SCL loss.

Original: We can observe from
Revised: We can observe from

Original: Eq.
Revised: Eq.

Original: (6) that the actual loss for the encoder (evaluated on a single query data ( q, u ) ∼ Q ) is as the following.
Revised: (6) that the actual loss for the encoder (evaluated on a single query data ( q, u ) ∼ Q ) is as the following.

Original: [Equation]
Revised: [Equation]

Original: For SOMAML, the range of “stop gradient” in the second term is different:
Revised: For SOMAML, the range of “stop gradient” in the second term is different:

Original: [Equation]
Revised: [Equation]

Original: With these two reformulations, we observe the essential difference between FOMAML and SOMAML is the range of stop gradient.
Revised: With these two reformulations, we observe the essential difference between FOMAML and SOMAML is the range of stop gradient.

Original: We provide detailed discussion and instinctive illustration in
Revised: We provide detailed discussion and instinctive illustration in

Original: Appendix B.5 on how this explains the phenomenon that SOMAML often leads to faster convergence.
Revised: Appendix B.5 on how this explains the phenomenon that SOMAML often leads to faster convergence.

Original: To better deliberate the effect of each term in the reformulated outer loop loss, we define the first term in Eq.
Revised: To better deliberate the effect of each term in the reformulated outer loop loss, we define the first term in Eq.

Original: (7) or Eq.
Revised: (7) or Eq.

Original: (8) as interference term , the second term as noisy contrastive term , and the coefficients − (cid:80) N way j =1 q j s j + s u + q t − 1 t=
Revised: (8) as interference term , the second term as noisy contrastive term , and the coefficients − (cid:80) N way j =1 q j s j + s u + q t − 1 t=

Original: u as contrastive coefficients .
Revised: u as contrastive coefficients .

Original: Understanding the interference term.
Revised: Understanding the interference term.

Original: In the case of j = u , the outer loop loss forces the model to minimize ( q j − 1 ) w j 0 ⊤ ϕ ( q ) .
Revised: In the case of j = u , the outer loop loss forces the model to minimize ( q j − 1 ) w j 0 ⊤ ϕ ( q ) .

Original: This can be problematic because (a) at the beginning of training, wis assigned with random values and (b) w 0 is added with query features of previous tasks as shown in Eq.
Revised: This can be problematic because (a) at the beginning of training, wis assigned with random values and (b) w 0 is added with query features of previous tasks as shown in Eq.

Original: Consequently, ϕ ( q ) is pushed to a direction composed of previous query features or to a random direction, introducing an unnecessary cross-task interference or an initialization interference that slows down the training of the encoder.
Revised: Consequently, ϕ ( q ) is pushed to a direction composed of previous query features or to a random direction, introducing an unnecessary cross-task interference or an initialization interference that slows down the training of the encoder.

Original: Noting that the cross-task interference also occurs at the testing period, since, at testing stage, w 0 is already added with query features of training tasks, which can be an interference to testing tasks.
Revised: Noting that the cross-task interference also occurs at the testing period, since, at testing stage, w 0 is already added with query features of training tasks, which can be an interference to testing tasks.

Original: Understanding the noisy contrastive term.
Revised: Understanding the noisy contrastive term.

Original: When the query and support data have the same label (i.e., u = t ), e.g., class 1 , the contrastive coefficients becomes − (cid:80) N way j =2 q
Revised: When the query and support data have the same label (i.e., u = t ), e.g., class 1 , the contrastive coefficients becomes − (cid:80) N way j =2 q

Original: j s j − q 1 s 1 + s 1 + q 1 − 1 , which is − (cid:80) N way j =2 q j s j − (1 − q 1 )(1 − s 1 ) < 0 .
Revised: j s j − q 1 s 1 + s 1 + q 1 − 1 , which is − (cid:80) N way j =2 q j s j − (1 − q 1 )(1 − s 1 ) < 0 .

Original: This indicates the encoder would be updated to maximize the inner product between ϕ ( q ) and the support features of the same class.
Revised: This indicates the encoder would be updated to maximize the inner product between ϕ ( q ) and the support features of the same class.

Original: However, when the query and support data are in different classes, the sign of the contrastive coefficient can sometimes be negative.
Revised: However, when the query and support data are in different classes, the sign of the contrastive coefficient can sometimes be negative.

Original: The outer loop loss thus cannot well contrast the query features against the support features of different classes, making this loss term not an ordinary SCL loss.
Revised: The outer loop loss thus cannot well contrast the query features against the support features of different classes, making this loss term not an ordinary SCL loss.

Original: To better illustrate the influence of the interference term and the noisy contrastive term, we provide an ablation experiment in Appendix B.7.
Revised: To better illustrate the influence of the interference term and the noisy contrastive term, we provide an ablation experiment in Appendix B.7.

Original: Theorem 1 below formally connects MAML to SCL.
Revised: Theorem 1 below formally connects MAML to SCL.

Original: Theorem 1 With the EFIL assumption, FOMAML is a noisy SCL algorithm.
Revised: Theorem 1 With the EFIL assumption, FOMAML is a noisy SCL algorithm.

Original: With assumptions of (a) EFIL and (b) a single inner-loop update, SOMAML is a noisy SCL algorithm.
Revised: With assumptions of (a) EFIL and (b) a single inner-loop update, SOMAML is a noisy SCL algorithm.

Original: Proof: For FOMAML, both Eq.
Revised: Proof: For FOMAML, both Eq.

Original: (7) (one inner loop update step) and Eq.
Revised: (7) (one inner loop update step) and Eq.

Original: (26) (multiple inner loop update steps) follows Definition 1.
Revised: (26) (multiple inner loop update steps) follows Definition 1.

Original: For SOMAML, Eq.
Revised: For SOMAML, Eq.

Original: (8) follows Definition 1.
Revised: (8) follows Definition 1.

Original: Introduction of the zeroing trick makes Eq.
Revised: Introduction of the zeroing trick makes Eq.

Original: (7) and Eq.
Revised: (7) and Eq.

Original: (8) SCL losses.
Revised: (8) SCL losses.

Original: To tackle the interference term and make the contrastive coefficients more accurate, we introduce the zeroing trick: setting the w 0 to be zero after each outer loop update, as shown in Appendix A. With the zeroing trick, the original outer loop loss (of FOMAML) becomes
Revised: To tackle the interference term and make the contrastive coefficients more accurate, we introduce the zeroing trick: setting the w 0 to be zero after each outer loop update, as shown in Appendix A. With the zeroing trick, the original outer loop loss (of FOMAML) becomes

Original: [Equation]
Revised: [Equation]

Original: For SOMAML, the original outer loop loss becomes
Revised: For SOMAML, the original outer loop loss becomes

Original: [Equation]
Revised: [Equation]

Original: The zeroing trick brings two nontrivial effects: (a) eliminating the interference term in both Eq.
Revised: The zeroing trick brings two nontrivial effects: (a) eliminating the interference term in both Eq.

Original: (7) and Eq.
Revised: (7) and Eq.

Original: (8); (b) making the contrastive coefficients follow SCL.
Revised: (8); (b) making the contrastive coefficients follow SCL.

Original: For (b), since all the predictive values of support data become the same, i.e., s k
Revised: For (b), since all the predictive values of support data become the same, i.e., s k

Original: = 1 N way , the contrastive coefficient becomes q t − 1 t=u , which is negative when the support and query data have the same label, and positive otherwise.
Revised: = 1 N way , the contrastive coefficient becomes q t − 1 t=u , which is negative when the support and query data have the same label, and positive otherwise.

Original: With the zeroing trick, the contrastive coefficient follows the SCL loss, as summarized below.
Revised: With the zeroing trick, the contrastive coefficient follows the SCL loss, as summarized below.

Original: Corollary 1 With mild assumptions of (a) EFIL, (b) a single inner-loop update and (c) training with the zeroing trick (i.e., the linear layer is zeroed at the end of each outer loop), both FOMAML and
Revised: Corollary 1 With mild assumptions of (a) EFIL, (b) a single inner-loop update and (c) training with the zeroing trick (i.e., the linear layer is zeroed at the end of each outer loop), both FOMAML and

Original: SOMAML are SCL algorithms.
Revised: SOMAML are SCL algorithms.

Original: Proof: Both Eq.
Revised: Proof: Both Eq.

Original: (9) and Eq.
Revised: (9) and Eq.

Original: (10) follow Definition 1.
Revised: (10) follow Definition 1.

Original: The introduction of the zeroing trick makes the relationship between MAML and SCL more straightforward.
Revised: The introduction of the zeroing trick makes the relationship between MAML and SCL more straightforward.

Original: Generally speaking, by connecting MAML and SCL, we can better understand other
Revised: Generally speaking, by connecting MAML and SCL, we can better understand other

Original: MAML-based meta-learning studies.
Revised: MAML-based meta-learning studies.

Original: 2. R ESPONSES TO Q UESTIONS IN S ECTION
Revised: 2. R ESPONSES TO Q UESTIONS IN S ECTION

Original: In what sense does MAML guide any model to learn general-purpose representations?
Revised: In what sense does MAML guide any model to learn general-purpose representations?

Original: Under the EFIL assumption, MAML is a noisy SCL algorithm in a classification paradigm.
Revised: Under the EFIL assumption, MAML is a noisy SCL algorithm in a classification paradigm.

Original: The effectiveness of MAML in enabling models to learn general-purpose representations can be attributed to the
Revised: The effectiveness of MAML in enabling models to learn general-purpose representations can be attributed to the

Original: SCL characteristics of MAML.
Revised: SCL characteristics of MAML.

Original: How do the inner and outer loops in the training mechanism of MAML collaboratively prompt to achieve so?
Revised: How do the inner and outer loops in the training mechanism of MAML collaboratively prompt to achieve so?

Original: MAML adopts the inner and outer loops to perform noisy SCL sequentially.
Revised: MAML adopts the inner and outer loops to perform noisy SCL sequentially.

Original: In the inner loop, the features of support data are memorized by w via inner-loop update.
Revised: In the inner loop, the features of support data are memorized by w via inner-loop update.

Original: In the outer loop, the softmax output of the query data thus contains the inner products between the support features and the query feature.
Revised: In the outer loop, the softmax output of the query data thus contains the inner products between the support features and the query feature.

Original: What is the role of support and query data, and how do they interact with each other?
Revised: What is the role of support and query data, and how do they interact with each other?

Original: We show that the original loss in MAML can be reformulated as a loss term containing the inner products of the embedding of the support and query data.
Revised: We show that the original loss in MAML can be reformulated as a loss term containing the inner products of the embedding of the support and query data.

Original: In FOMAML, the support features act as the reference, while the query features are updated to move towards the support features of the same class and against those of the different classes.
Revised: In FOMAML, the support features act as the reference, while the query features are updated to move towards the support features of the same class and against those of the different classes.

Original: 2. G ENERALIZATION OF OUR A NALYSIS
Revised: 2. G ENERALIZATION OF OUR A NALYSIS

Original: In Appendix C, we provide the analysis where N batch ≥ 1 and N step ≥ 1 .
Revised: In Appendix C, we provide the analysis where N batch ≥ 1 and N step ≥ 1 .

Original: For the EFIL assumption, it can hardly be dropped because the behavior of the updated encoder is intractable.
Revised: For the EFIL assumption, it can hardly be dropped because the behavior of the updated encoder is intractable.

Original: Besides, Raghu et al.
Revised: Besides, Raghu et al.

Original: (2020) show that the representations of intermediate layers do not change notably during the inner loop of MAML, and thus it is understood that the main function of the inner loop is to change the final linear layer.
Revised: (2020) show that the representations of intermediate layers do not change notably during the inner loop of MAML, and thus it is understood that the main function of the inner loop is to change the final linear layer.

Original: Furthermore, the EFIL assumption is empirically reasonable, since previous works (Raghu et al., 2020; Lee et al., 2019; Bertinetto et al., 2019; Liu et al., 2020) yield comparable performance while leaving the encoder untouched during the inner loop.
Revised: Furthermore, the EFIL assumption is empirically reasonable, since previous works (Raghu et al., 2020; Lee et al., 2019; Bertinetto et al., 2019; Liu et al., 2020) yield comparable performance while leaving the encoder untouched during the inner loop.

Original: With our analysis, one may notice that MAML is approximately a metric-based few-shot learning algorithm.
Revised: With our analysis, one may notice that MAML is approximately a metric-based few-shot learning algorithm.

Original: From a high-level perspective, under the EFIL assumption, second-order MAML is similar to metric-based few-shot learning algorithms, such as MatchingNet (Vinyals et al., 2016), Prototypical network (Snell et al., 2017), and Relation network (Sung et al., 2018).
Revised: From a high-level perspective, under the EFIL assumption, second-order MAML is similar to metric-based few-shot learning algorithms, such as MatchingNet (Vinyals et al., 2016), Prototypical network (Snell et al., 2017), and Relation network (Sung et al., 2018).

Original: The main difference lies in the metric and the way prototypes are constructed.
Revised: The main difference lies in the metric and the way prototypes are constructed.

Original: Our work follows the setting adopted by
Revised: Our work follows the setting adopted by

Original: MAML, such as using negative LogSoftmax as objective function, but we can effortlessly generalize our analysis to a MSE loss as had been shown in Figure 1.
Revised: MAML, such as using negative LogSoftmax as objective function, but we can effortlessly generalize our analysis to a MSE loss as had been shown in Figure 1.

Original: As a result, our work points out a new research direction in improving MAML by changing the objective functions in the inner and the outer loops, e.g., using MSE for the inner loop but negative LogSoftmax for the outer loop.
Revised: As a result, our work points out a new research direction in improving MAML by changing the objective functions in the inner and the outer loops, e.g., using MSE for the inner loop but negative LogSoftmax for the outer loop.

Original: Besides, in MAML, we often obtain the logits by multiplying the features by the linear weight w .
Revised: Besides, in MAML, we often obtain the logits by multiplying the features by the linear weight w .

Original: Our work implies future direction as to alternatively substitute this inner product operation with other metrics or other similarity measurements such as cosine similarity or negative Euclidean distance.
Revised: Our work implies future direction as to alternatively substitute this inner product operation with other metrics or other similarity measurements such as cosine similarity or negative Euclidean distance.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: In this section, we provide empirical evidence of the supervised contrastiveness of MAML and show that zero-initialization of w 0 , reduction in the initial norm of w 0 , or the application of zeroing trick can speed up the learning profile.
Revised: In this section, we provide empirical evidence of the supervised contrastiveness of MAML and show that zero-initialization of w 0 , reduction in the initial norm of w 0 , or the application of zeroing trick can speed up the learning profile.

Original: This is applicable to both SOMAML and FOMAML.
Revised: This is applicable to both SOMAML and FOMAML.

Original: 3. S ETUP
Revised: 3. S ETUP

Original: We conduct our experiments on the mini-ImageNet dataset (Vinyals et al., 2016; Ravi & Larochelle, 2017) and the Omniglot dataset (Lake et al., 2015).
Revised: We conduct our experiments on the mini-ImageNet dataset (Vinyals et al., 2016; Ravi & Larochelle, 2017) and the Omniglot dataset (Lake et al., 2015).

Original: For the results on the Omniglot dataset, please refer to Appendix E. For the mini-ImageNet, it contains 84 × 84 RGB images of 100 classes from the ImageNet dataset with 600 samples per class.
Revised: For the results on the Omniglot dataset, please refer to Appendix E. For the mini-ImageNet, it contains 84 × 84 RGB images of 100 classes from the ImageNet dataset with 600 samples per class.

Original: We split the dataset into 64, 16 and 20 classes for training, validation, and testing as proposed in (Ravi & Larochelle, 2017).
Revised: We split the dataset into 64, 16 and 20 classes for training, validation, and testing as proposed in (Ravi & Larochelle, 2017).

Original: We do not perform hyperparameter search and thus are not using the validation data.
Revised: We do not perform hyperparameter search and thus are not using the validation data.

Original: For all our experiments of applying
Revised: For all our experiments of applying

Original: MAML into few-shot classification problem, where we adopt two experimental settings: 5-way 1shot and 5-way 5-shot, with the batch size N batch being 4 and 2, respectively (Finn et al., 2017).
Revised: MAML into few-shot classification problem, where we adopt two experimental settings: 5-way 1shot and 5-way 5-shot, with the batch size N batch being 4 and 2, respectively (Finn et al., 2017).

Original: The few-shot classification accuracy is calculated by averaging the results over 400 tasks in the test phase.
Revised: The few-shot classification accuracy is calculated by averaging the results over 400 tasks in the test phase.

Original: For model architecture, optimizer and other experimental details, please refer to Appendix D.1.
Revised: For model architecture, optimizer and other experimental details, please refer to Appendix D.1.

Original: 3. C OSINE S IMILARITY A NALYSIS V ERIFIES THE I MPLICIT C ONTRASTIVENESS IN MAML
Revised: 3. C OSINE S IMILARITY A NALYSIS V ERIFIES THE I MPLICIT C ONTRASTIVENESS IN MAML

Original: In Section 2, we show that the encoder is updated so that the query features are pushed towards the support features of the same class and pulled away from those of different classes.
Revised: In Section 2, we show that the encoder is updated so that the query features are pushed towards the support features of the same class and pulled away from those of different classes.

Original: Here we verify this supervised contrastiveness experimentally.
Revised: Here we verify this supervised contrastiveness experimentally.

Original: Consider a relatively overfitting scenario where there are five classes of images and for each class there are 20 support images and 20 query images.
Revised: Consider a relatively overfitting scenario where there are five classes of images and for each class there are 20 support images and 20 query images.

Original: We fix the support and query set (i.e. the data is not resampled every iteration) to verify the concept that the support features work as positive and negative samples.
Revised: We fix the support and query set (i.e. the data is not resampled every iteration) to verify the concept that the support features work as positive and negative samples.

Original: Channel shuffling is used to avoid the undesirable channel memorization effect (Jamal & Qi, 2019; Rajendran et al., 2020).
Revised: Channel shuffling is used to avoid the undesirable channel memorization effect (Jamal & Qi, 2019; Rajendran et al., 2020).

Original: We train the model using FOMAML and examine how well the encoder can separate the data of different classes in the feature space by measuring the averaged cosine similarities between the features of each class.
Revised: We train the model using FOMAML and examine how well the encoder can separate the data of different classes in the feature space by measuring the averaged cosine similarities between the features of each class.

Original: The results are averaged over 10 random seeds.
Revised: The results are averaged over 10 random seeds.

Original: As shown in the top row of Figure 2, the model trained with MAML learns to separate the features of different classes.
Revised: As shown in the top row of Figure 2, the model trained with MAML learns to separate the features of different classes.

Original: Moreover, the contrast between the diagonal and the off-diagonal entries of the heatmap increases as we remove the initialization interference (by zero-initializing w 0 , shown in the middle row) and remove the cross-task interference (by applying the zeroing trick, shown in the bottom row).
Revised: Moreover, the contrast between the diagonal and the off-diagonal entries of the heatmap increases as we remove the initialization interference (by zero-initializing w 0 , shown in the middle row) and remove the cross-task interference (by applying the zeroing trick, shown in the bottom row).

Original: The result agrees with our analysis that MAML implicitly contains the interference term which can impede the encoder from learning a good feature representation.
Revised: The result agrees with our analysis that MAML implicitly contains the interference term which can impede the encoder from learning a good feature representation.

Original: For experiments on semantically similar classes of images, the result is shown in Section D.3.
Revised: For experiments on semantically similar classes of images, the result is shown in Section D.3.

Original: 3. Z EROING L INEAR L AYER AT T ESTING T IME I NCREASES T ESTING A CCURACY
Revised: 3. Z EROING L INEAR L AYER AT T ESTING T IME I NCREASES T ESTING A CCURACY

Original: Before starting our analysis on benchmark datasets, we note that the cross-task interference can also occur during meta-testing.
Revised: Before starting our analysis on benchmark datasets, we note that the cross-task interference can also occur during meta-testing.

Original: In the meta-testing stage, the base-model is updated in the inner loop using support data S and then the performance is evaluated using query data Q , where S and Q are drawn from a held-out, unseen meta-testing dataset.
Revised: In the meta-testing stage, the base-model is updated in the inner loop using support data S and then the performance is evaluated using query data Q , where S and Q are drawn from a held-out, unseen meta-testing dataset.

Original: Recall that at the end of the outer loop (in meta-training stage), the query features are added weightedly to the linear layer w 0 .
Revised: Recall that at the end of the outer loop (in meta-training stage), the query features are added weightedly to the linear layer w 0 .

Original: In other words, at the beginning of meta-testing, w 0 is already added with the query features of previous training tasks, which can drastically influence the performance on the unseen tasks.
Revised: In other words, at the beginning of meta-testing, w 0 is already added with the query features of previous training tasks, which can drastically influence the performance on the unseen tasks.

Original: To validate this idea, we apply the zeroing trick at meta-testing time (which we refer to zeroing wat the beginning of the meta-testing time) and show such trick increases the testing accuracy of the model trained with FOMAML.
Revised: To validate this idea, we apply the zeroing trick at meta-testing time (which we refer to zeroing wat the beginning of the meta-testing time) and show such trick increases the testing accuracy of the model trained with FOMAML.

Original: As illustrated in Figure 3, compared to directly entering meta-testing (i.e. the subplot at the left), additionally zeroing the linear layer at the beginning of each meta-testing time (i.e. the subplot at the right) increases the testing accuracy of the model whose linear layer is randomly initialized or zero-initialized (denoted by the red and orange curves, respectively).
Revised: As illustrated in Figure 3, compared to directly entering meta-testing (i.e. the subplot at the left), additionally zeroing the linear layer at the beginning of each meta-testing time (i.e. the subplot at the right) increases the testing accuracy of the model whose linear layer is randomly initialized or zero-initialized (denoted by the red and orange curves, respectively).

Original: And the difference in testing performance sustains across the whole training session.
Revised: And the difference in testing performance sustains across the whole training session.

Original: In the following experiments, we evaluate the testing performance only with zeroing the linear layer at the beginning of the meta-testing stage.
Revised: In the following experiments, we evaluate the testing performance only with zeroing the linear layer at the beginning of the meta-testing stage.

Original: By zeroing the linear layer, the potential interference brought by the prior (of the linear layer) is ignored.
Revised: By zeroing the linear layer, the potential interference brought by the prior (of the linear layer) is ignored.

Original: Then, we can fully focus on the capacity of the encoder in learning a good feature representation
Revised: Then, we can fully focus on the capacity of the encoder in learning a good feature representation

Original: [Figure]
Revised: [Figure]

Original: 3. S INGLE I NNER L OOP U PDATE S UFFICES WHEN U SING THE Z EROING T RICK
Revised: 3. S INGLE I NNER L OOP U PDATE S UFFICES WHEN U SING THE Z EROING T RICK

Original: In Eq.
Revised: In Eq.

Original: (4) and Eq.
Revised: (4) and Eq.

Original: (21), we show that the features of the support data are added to the linear layer in the inner loop.
Revised: (21), we show that the features of the support data are added to the linear layer in the inner loop.

Original: Larger number of inner loop update steps can better offset the effect of interference brought by a non-zeroed linear layer.
Revised: Larger number of inner loop update steps can better offset the effect of interference brought by a non-zeroed linear layer.

Original: In other words, when the models are trained with the zeroing trick, a larger number of inner loop updates can not bring any benefit.
Revised: In other words, when the models are trained with the zeroing trick, a larger number of inner loop updates can not bring any benefit.

Original: We validate this intuition in
Revised: We validate this intuition in

Original: Figure 4 under a 5-way 1-shot setting.
Revised: Figure 4 under a 5-way 1-shot setting.

Original: In the original FOMAML, the models trained with a single inner loop update step (denoted as red curve) converge slower than those trained with update step of 7 (denoted as purple curve).
Revised: In the original FOMAML, the models trained with a single inner loop update step (denoted as red curve) converge slower than those trained with update step of 7 (denoted as purple curve).

Original: On the contrary, when the models are trained with the zeroing trick, models with various inner loop update steps converge at the same speed.
Revised: On the contrary, when the models are trained with the zeroing trick, models with various inner loop update steps converge at the same speed.

Original: [Figure]
Revised: [Figure]

Original: However
Revised: However,

Original: 3. E FFECT OF I NITIALIZATION AND THE Z EROING T RICK
Revised: 3. E FFECT OF I NITIALIZATION AND THE Z EROING T RICK

Original: In Eq.
Revised: In Eq.

Original: (7), we observe an interference derived from the historical task features or random initialization.
Revised: (7), we observe an interference derived from the historical task features or random initialization.

Original: We validate our formula by examining the effects of (1) reducing the norm of w 0 at initialization and (2) applying the zeroing trick.
Revised: We validate our formula by examining the effects of (1) reducing the norm of w 0 at initialization and (2) applying the zeroing trick.

Original: From Figure 5, the performance is higher when the initial norm of w 0 is lower.
Revised: From Figure 5, the performance is higher when the initial norm of w 0 is lower.

Original: Compared to random initialization, reducing the norm via down-scaling w 0 by 0 .
Revised: Compared to random initialization, reducing the norm via down-scaling w 0 by 0 .

Original: 7 yields visible differences.
Revised: 7 yields visible differences.

Original: Besides, the testing accuracy of MAML with zeroing trick (the purple curve) outperforms that of original MAML.
Revised: Besides, the testing accuracy of MAML with zeroing trick (the purple curve) outperforms that of original MAML.

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper presents an extensive study to demystify how the seminal MAML algorithm guides the encoder to learn a general-purpose feature representation and how support and query data interact.
Revised: This paper presents an extensive study to demystify how the seminal MAML algorithm guides the encoder to learn a general-purpose feature representation and how support and query data interact.

Original: Our analysis shows that MAML is implicitly a supervised contrastive learner using the support features as positive and negative samples to direct the update of the encoder.
Revised: Our analysis shows that MAML is implicitly a supervised contrastive learner using the support features as positive and negative samples to direct the update of the encoder.

Original: Moreover, we unveil an interference term hidden in MAML originated from the random initialization or cross-task interaction, which can impede the representation learning.
Revised: Moreover, we unveil an interference term hidden in MAML originated from the random initialization or cross-task interaction, which can impede the representation learning.

Original: Driven by our analysis, removing the interference term by a simple zeroing trick renders the model unbiased to seen or unseen tasks.
Revised: Driven by our analysis, removing the interference term by a simple zeroing trick renders the model unbiased to seen or unseen tasks.

Original: Furthermore, we show constant improvements in the training and testing profiles with this zeroing trick, with experiments conducted on the mini-ImageNet and Omniglot datasets.
Revised: Furthermore, we show constant improvements in the training and testing profiles with this zeroing trick, with experiments conducted on the mini-ImageNet and Omniglot datasets.

Original: A PPENDIX A O RIGINAL MAML AND MAML WITH THE Z EROING T RICK
Revised: A PPENDIX A O RIGINAL MAML AND MAML WITH THE Z EROING T RICK

Original: Require : Task distribution D
Revised: Require : Task distribution D

Original: Require : η, ρ : inner loop and outer loop learning rates
Revised: Require : η, ρ : inner loop and outer loop learning rates

Original: Require : Randomly initialized base-model parameters θ 1: while not done do 2:
Revised: Require : Randomly initialized base-model parameters θ 1: while not done do 2:

Original: Sample tasks { T 1 , . . .
Revised: Sample tasks { T 1 , . . .

Original: T N batch } from D 3: for n
Revised: T N batch } from D 3: for n

Original: = 1 , 2 , . .
Revised: = 1 , 2 , . .

Original: . , N batch do 4: { S n , Q n } ← sample from T n 5: θ n = θ 6: for i = 1 , 2 , . . .
Revised: . , N batch do 4: { S n , Q n } ← sample from T n 5: θ n = θ 6: for i = 1 , 2 , . . .

Original: , N step do 7: θ n ← θ n − η ∇ θ n L θ n ,S n 8: end for 9: end for 10:
Revised: , N step do 7: θ n ← θ n − η ∇ θ n L θ n ,S n 8: end for 9: end for 10:

Original: Update θ ← θ − ρ (cid:80) N batch n = ∇ θ L θ n ,Q n 11: end while
Revised: Update θ ← θ − ρ (cid:80) N batch n = ∇ θ L θ n ,Q n 11: end while

Original: Require : Task distribution D
Revised: Require : Task distribution D

Original: Require : η, ρ : inner loop and outer loop learning rates
Revised: Require : η, ρ : inner loop and outer loop learning rates

Original: Require : Randomly initialized base-model parameters θ 1: while not done do 2:
Revised: Require : Randomly initialized base-model parameters θ 1: while not done do 2:

Original: Sample tasks { T 1 , . . .
Revised: Sample tasks { T 1 , . . .

Original: T N batch } from D 3: for n
Revised: T N batch } from D 3: for n

Original: = 1 , 2 , . .
Revised: = 1 , 2 , . .

Original: . , N batch do 4: { S n , Q n } ← sample from T n 5: θ n = θ 6: for i = 1 , 2 , . . .
Revised: . , N batch do 4: { S n , Q n } ← sample from T n 5: θ n = θ 6: for i = 1 , 2 , . . .

Original: , N step do 7: θ n ← θ n − η ∇ θ n L θ n ,S n 8: end for 9: end for 10:
Revised: , N step do 7: θ n ← θ n − η ∇ θ n L θ n ,S n 8: end for 9: end for 10:

Original: Update θ ← θ − ρ (cid:80) N batch n = ∇ θ n L θ n ,Q n 11: end while
Revised: Update θ ← θ − ρ (cid:80) N batch n = ∇ θ n L θ n ,Q n 11: end while

Original: Require : Task distribution D
Revised: Require : Task distribution D

Original: Require : η, ρ : inner loop and outer loop learning rates
Revised: Require : η, ρ : inner loop and outer loop learning rates

Original: Require : Randomly initialized base-model parameters θ 1: Set w ← 0 (the zeroing trick) 2: while not done do 3:
Revised: Require : Randomly initialized base-model parameters θ 1: Set w ← 0 (the zeroing trick) 2: while not done do 3:

Original: Sample tasks { T 1 , . . .
Revised: Sample tasks { T 1 , . . .

Original: T N batch } from D 4: for n
Revised: T N batch } from D 4: for n

Original: = 1 , 2 , . . .
Revised: = 1 , 2 , . . .

Original: do 5: do 8:
Revised: do 5: do 8:

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: 9: end for 10: end for 11:
Revised: 9: end for 10: end for 11:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Set w ← 0 (the zeroing trick) 13: end while
Revised: Set w ← 0 (the zeroing trick) 13: end while

Original: B S UPPLEMENTARY D ERIVATION
Revised: B S UPPLEMENTARY D ERIVATION

Original: In this section
Revised: In this section

Original: [Equation]
Revised: [Equation]

Original: we provide the full generalization and further discussion that supplement the main paper.
Revised: we provide the full generalization and further discussion that supplement the main paper.

Original: We consider the case of N batch = 

Revised: We consider the case of N batch =

Original: and N step = 1 under the EFIL assumption.
Revised: and N step = 1 under the EFIL assumption.

Original: We provide the outer loop update of the linear layer under SOMAML in Section B.1.
Revised: We provide the outer loop update of the linear layer under SOMAML in Section B.1.

Original: Next, we offer the full derivation of the outer loop update of the encoder in Section B.2.
Revised: Next, we offer the full derivation of the outer loop update of the encoder in Section B.2.

Original: Then, we reformulate the outer loop loss for the encoder in both FOMAML and SOMAML in Section B.3 and Section B.4.
Revised: Then, we reformulate the outer loop loss for the encoder in both FOMAML and SOMAML in Section B.3 and Section B.4.

Original: Afterward, we discuss the main difference in FOMAML and SOMAML in detail in Section B.5.
Revised: Afterward, we discuss the main difference in FOMAML and SOMAML in detail in Section B.5.

Original: Finally, we show the performance of the models trained using the reformulated loss in Section B.6.
Revised: Finally, we show the performance of the models trained using the reformulated loss in Section B.6.

Original: B. T HE D ERIVATION OF O UTER L OOP U PDATE FOR THE L INEAR L AYER U SING SOMAML
Revised: B. T HE D ERIVATION OF O UTER L OOP U PDATE FOR THE L INEAR L AYER U SING SOMAML

Original: Here, we provide the complete derivation of the outer loop update for the linear layer.
Revised: Here, we provide the complete derivation of the outer loop update for the linear layer.

Original: Using SO- MAML with support set S and query set Q , the update of the linear layer follows
Revised: Using SO- MAML with support set S and query set Q , the update of the linear layer follows

Original: [Equation]
Revised: [Equation]

Original: We can further simplify Eq.
Revised: We can further simplify Eq.

Original: (11) to Eq.
Revised: (11) to Eq.

Original: (12) with the help of the zeroing trick.
Revised: (12) with the help of the zeroing trick.

Original: [Equation]
Revised: [Equation]

Original: This is because the zeroing trick essentially turns the logits of all support samples to zero, and consequently the predicted probability (softmax) output s m becomes 1 N way for all channel m .
Revised: This is because the zeroing trick essentially turns the logits of all support samples to zero, and consequently the predicted probability (softmax) output s m becomes 1 N way for all channel m .

Original: Therefore, the third term in Eq.
Revised: Therefore, the third term in Eq.

Original: (11) turns out to be zero (c.f. Eq.
Revised: (11) turns out to be zero (c.f. Eq.

Original: The equality of Eq.
Revised: The equality of Eq.

Original: (13) holds since the summation of the (softmax) outputs is one.
Revised: (13) holds since the summation of the (softmax) outputs is one.

Original: [Equation]
Revised: [Equation]

Original: B. T HE F ULL D ERIVATION OF THE O UTER L OOP U PDATE OF THE E NCODER .
Revised: B. T HE F ULL D ERIVATION OF THE O UTER L OOP U PDATE OF THE E NCODER .

Original: As the encoder ϕ is parameterized by φ , the outer loop gradient with respect to φ is given by ∂L
Revised: As the encoder ϕ is parameterized by φ , the outer loop gradient with respect to φ is given by ∂L

Original: [Equation]
Revised: [Equation]

Original: We take a deeper look at the backpropagated error
Revised: We take a deeper look at the backpropagated error

Original: [Equation]
Revised: [Equation]

Original: of the feature of one query data ( q, u ) ∼ Q , based on the following form:
Revised: of the feature of one query data ( q, u ) ∼ Q , based on the following form:

Original: [Equation]
Revised: [Equation]

Original: B. R EFORMULATION OF THE O UTER L OOP L OSS FOR THE E NCODER AS N OISY SCL L OSS .
Revised: B. R EFORMULATION OF THE O UTER L OOP L OSS FOR THE E NCODER AS N OISY SCL L OSS .

Original: We can derive the actual loss (evaluated on a single query data ( q, u ) ∼ Q ) that the encoder uses under FOMAML scheme as follows:
Revised: We can derive the actual loss (evaluated on a single query data ( q, u ) ∼ Q ) that the encoder uses under FOMAML scheme as follows:

Original: [Equation]
Revised: [Equation]

Original: For SOMAML, we need to additionally plug Eq.
Revised: For SOMAML, we need to additionally plug Eq.

Original: (4) into Eq.
Revised: (4) into Eq.

Original: [Equation]
Revised: [Equation]

Original: B. I NTRODUCTION OF THE ZEROING TRICK MAKES E Q . (7) AND E Q . (8) SCL LOSSES
Revised: B. I NTRODUCTION OF THE ZEROING TRICK MAKES E Q . (7) AND E Q . (8) SCL LOSSES

Original: Apply the zeroing trick to Eq.
Revised: Apply the zeroing trick to Eq.

Original: (7) and Eq.
Revised: (7) and Eq.

Original: (8), we can derive the actual loss Eq.
Revised: (8), we can derive the actual loss Eq.

Original: (17) and Eq.
Revised: (17) and Eq.

Original: (18) that the encoder follows.
Revised: (18) that the encoder follows.

Original: [Equation]
Revised: [Equation]

Original: With these two equations, we can observe the essential difference in FOMAML and SOMAML is the range of stopping gradients.
Revised: With these two equations, we can observe the essential difference in FOMAML and SOMAML is the range of stopping gradients.

Original: We would further discuss the implication of different ranges of gradient stopping in Appendix B.5.
Revised: We would further discuss the implication of different ranges of gradient stopping in Appendix B.5.

Original: B. D ISCUSSION ABOUT THE DIFFERENCE BETWEEN FOMAML AND SOMAML
Revised: B. D ISCUSSION ABOUT THE DIFFERENCE BETWEEN FOMAML AND SOMAML

Original: Central to the mystery of MAML is the difference between FOMAML and SOMAML.
Revised: Central to the mystery of MAML is the difference between FOMAML and SOMAML.

Original: Plenty of work is dedicated to approximating or estimating the second-order derivatives in the MAML algorithm in a more computational-efficient or accurate manner (Song et al., 2020;
Revised: Plenty of work is dedicated to approximating or estimating the second-order derivatives in the MAML algorithm in a more computational-efficient or accurate manner (Song et al., 2020;

Original: Rothfuss et al., 2019;
Revised: Rothfuss et al., 2019;

Original: Liu et al., 2019).
Revised: Liu et al., 2019).

Original: With the EFIL assumption and our analysis through connecting SCL to these algorithms, we found that we can better understand the distinction between FOMAML and SOMAML from a novel perspective.
Revised: With the EFIL assumption and our analysis through connecting SCL to these algorithms, we found that we can better understand the distinction between FOMAML and SOMAML from a novel perspective.

Original: To better understand the difference, we can compare Eq.
Revised: To better understand the difference, we can compare Eq.

Original: (7) with Eq.
Revised: (7) with Eq.

Original: (8) or compare Eq.
Revised: (8) or compare Eq.

Original: (9) with Eq.
Revised: (9) with Eq.

Original: To avoid being distracted by the interference terms, we provide the analysis of the latter.
Revised: To avoid being distracted by the interference terms, we provide the analysis of the latter.

Original: The main difference between Eq.
Revised: The main difference between Eq.

Original: (9) and Eq.
Revised: (9) and Eq.

Original: (10) is the range of gradient stopping and we will show that this difference results in a significant distinction in the feature space.
Revised: (10) is the range of gradient stopping and we will show that this difference results in a significant distinction in the feature space.

Original: To begin with, by chain rule, we have ∂L∂φ =
Revised: To begin with, by chain rule, we have ∂L∂φ =

Original: E ( q,u ) ∼ Q
Revised: E ( q,u ) ∼ Q

Original: ∂L∂ϕ
Revised: ∂L∂ϕ

Original: ( q ) ∂ϕ ( q ) ∂φ
Revised: ( q ) ∂ϕ ( q ) ∂φ

Original: + E ( s,t )
Revised: + E ( s,t )

Original: ∼ S ∂L∂ϕ ( s ) ∂ϕ ( s ) ∂φ .
Revised: ∼ S ∂L∂ϕ ( s ) ∂ϕ ( s ) ∂φ .

Original: As we specifically want to know how the encoded features are updated given different losses, we can look at the terms ∂L∂ϕ ( q ) and ∂L∂ϕ ( s ) by differentiating Eq.
Revised: As we specifically want to know how the encoded features are updated given different losses, we can look at the terms ∂L∂ϕ ( q ) and ∂L∂ϕ ( s ) by differentiating Eq.

Original: (9) and Eq.
Revised: (9) and Eq.

Original: (10) with respect to the features of query data q and support data s , respectively.
Revised: (10) with respect to the features of query data q and support data s , respectively.

Original: FOMAML:
Revised: FOMAML:

Original: [Equation]
Revised: [Equation]

Original: SOMAML:
Revised: SOMAML:

Original: [Equation]
Revised: [Equation]

Original: Obviously, as the second equation in Eq (19) is zero, we know that in FOMAML, the update of the encoder does consider the change of the support features.
Revised: Obviously, as the second equation in Eq (19) is zero, we know that in FOMAML, the update of the encoder does consider the change of the support features.

Original: The encoder is updated to move the query features closer to support features of the same class and further to support features of different classes in FOMAML.
Revised: The encoder is updated to move the query features closer to support features of the same class and further to support features of different classes in FOMAML.

Original: On the contrary, we can tell from the above equations that in SOMAML, the encoder is updated to make support features and query features closer if both come from the same class and make support features and query features further if they come from different classes.
Revised: On the contrary, we can tell from the above equations that in SOMAML, the encoder is updated to make support features and query features closer if both come from the same class and make support features and query features further if they come from different classes.

Original: We illustrate the difference in Figure 6.
Revised: We illustrate the difference in Figure 6.

Original: For simplicity, we do not consider the scale of the coefficients but their signs.
Revised: For simplicity, we do not consider the scale of the coefficients but their signs.

Original: The subplot on the left indicates that this FOMAML loss guides the encoder to be updated so that the feature of the query data moves 1) towards the support feature of the same class, and 2) against the support features of the different classes.
Revised: The subplot on the left indicates that this FOMAML loss guides the encoder to be updated so that the feature of the query data moves 1) towards the support feature of the same class, and 2) against the support features of the different classes.

Original: On the other hand, the SOMAML loss guides the encoder to be updated so that 1) when the support data and query data belong to the same class, their features move closer, and otherwise, their features move further.
Revised: On the other hand, the SOMAML loss guides the encoder to be updated so that 1) when the support data and query data belong to the same class, their features move closer, and otherwise, their features move further.

Original: This generally explains why models trained using SOMAML generally converge faster than those trained using
Revised: This generally explains why models trained using SOMAML generally converge faster than those trained using

Original: FOMAML.
Revised: FOMAML.

Original: represent different classes.
Revised: represent different classes.

Original: The solid and hollow arrows indicate the gradient calculated from positive and negative samples, respectively.
Revised: The solid and hollow arrows indicate the gradient calculated from positive and negative samples, respectively.

Original: Note that this is in the feature space, not the weight space.
Revised: Note that this is in the feature space, not the weight space.

Original: B. E XPLICITLY C OMPUTING THE R EFORMULATING L OSS U SING E Q . (7) AND E Q . (8)
Revised: B. E XPLICITLY C OMPUTING THE R EFORMULATING L OSS U SING E Q . (7) AND E Q . (8)

Original: Under the EFIL assumption, we show that MAML can be reformulated as a loss taking noisy SCL form.
Revised: Under the EFIL assumption, we show that MAML can be reformulated as a loss taking noisy SCL form.

Original: Below, we consider a setting of 5-way 1-shot mini-ImageNet few-shot classification task, under the condition of no inner loop update of the encoder.
Revised: Below, we consider a setting of 5-way 1-shot mini-ImageNet few-shot classification task, under the condition of no inner loop update of the encoder.

Original: (This is the assumption that our derivation heavily depends on.
Revised: (This is the assumption that our derivation heavily depends on.

Original: It means that we now only update the encoder in the outer loop.)
Revised: It means that we now only update the encoder in the outer loop.)

Original: We empirically show that explicitly computing the reformulated losses of Eq.
Revised: We empirically show that explicitly computing the reformulated losses of Eq.

Original: (7), Eq.
Revised: (7), Eq.

Original: (17) and Eq.
Revised: (17) and Eq.

Original: (18) yield almost the same curves as MAML (with the EFIL assumption).
Revised: (18) yield almost the same curves as MAML (with the EFIL assumption).

Original: Please note that the reformulated losses are used to update the encoders, for the linear layer w 0 , we explicitly update it using Eq.
Revised: Please note that the reformulated losses are used to update the encoders, for the linear layer w 0 , we explicitly update it using Eq.

Original: Note that although the performance models training using FOMAML, FOMAML with the zeroing trick, and
Revised: Note that although the performance models training using FOMAML, FOMAML with the zeroing trick, and

Original: SOMAML converge to similar testing accuracy, the overall testing performance during the training process is distinct.
Revised: SOMAML converge to similar testing accuracy, the overall testing performance during the training process is distinct.

Original: The results are averaged over three random seeds.
Revised: The results are averaged over three random seeds.

Original: [Figure]
Revised: [Figure]

Original: B. T HE E FFECT OF I NTERFERENCE T ERM AND N OISY C ONTRASTIVE T ERM
Revised: B. T HE E FFECT OF I NTERFERENCE T ERM AND N OISY C ONTRASTIVE T ERM

Original: Reformulating the loss of MAML into a noisy SCL form enables us to further investigate the effects brought by the interference term and the noisy contrastive term, which we presume both effects to be negative.
Revised: Reformulating the loss of MAML into a noisy SCL form enables us to further investigate the effects brought by the interference term and the noisy contrastive term, which we presume both effects to be negative.

Original: To investigate the effect of the interference term, we simply consider the loss  in Eq.
Revised: To investigate the effect of the interference term, we simply consider the loss adopted by firstorder MAML as in Eq.

Original: (7)  with the interference term dropped (denoted as “ITF × ”).
Revised: (7) but with the interference term dropped (denoted as “ n 1 × ”).

Original: As for the noisy contrastive term, the noise comes from the fact that “when the query and support data are in different classes, the sign of the contrastive coefficient can sometimes be negative”, as being discussed in Section 2.4.
Revised: As for the noisy contrastive term, the noise comes from the fact that “when the query and support data are in different classes, the sign of the contrastive coefficient can sometimes be negative”, as being discussed in Section 2.4.

Original: To mitigate this noise, we consider the loss in Eq.
Revised: To mitigate this noise, we consider the loss in Eq.

Original: (7) with the term − ( (cid:80) N way j =1 q j s j )+ s u dropped from the contrastive coefficient, and denote it as “nCTT × ”.
Revised: (7) with the term − ( (cid:80) N way j =1 q j s j ) + s u dropped from the contrastive coefficient, and denote it as “ n 2 × ”.

Original: On the other hand, we also implement a loss with “ITF × , nCTT × ”, which is actually Eq.
Revised: On the other hand, we also implement a loss with “ n 1 × , n 2 × ”, which is actually Eq.

Original: We adopt the same experimental setting as Section B.6.
Revised: We adopt the same experimental setting as Section B.6.

Original: In Figure 8, we show the testing profiles of the original reformulated loss (i.e., the curve in red, labeled as “ITF ✓ , nCTT ✓ ”), dropping the interference term (i.e., the curve in orange, labeled as “ITF × , nCTT ✓ ”), dropping the noisy part of the contrastive term (i.e., the curve in green, labeled as “ITF ✓ , nCTT × ”) or dropping both (i.e., the curve in blue, labeled as “ITF × , nCTT × ”).
Revised: In Figure 8, we show the testing profiles of the original reformulated loss (i.e., the curve in red, labeled as “ n 1 ✓ , n 2 ✓ ”), dropping the interference term (i.e., the curve in orange, labeled as “ n× , n 2 ✓ ”), dropping the noisy part of the contrastive term (i.e., the curve in green, labeled as “ n 1 ✓ , n 2 × ”) or dropping both (i.e., the curve in blue, labeled as “ n 1 × , n 2 × ”).

Original: We can see that either dropping the interference term or dropping dropping the noisy part of contrastive coefficients yield profound benefit.
Revised: We can see that either dropping the interference term or dropping dropping the noisy part of contrastive coefficients yield profound benefit.

Revised: [Figure]

Original: To better understand how noisy is the noisy contrastive term, i.e., how many times the sign of the contrastive coefficient is negative when the query and support data are in different classes, we explicitly record the ratio of the contrastive term being positive or negative.
Revised: To better understand how noisy is the noisy contrastive term, i.e., how many times the sign of the contrastive coefficient is negative when the query and support data are in different classes, we explicitly record the ratio of the contrastive term being positive or negative.

Original: We adopt the same experimental setting as Section B.6.
Revised: We adopt the same experimental setting as Section B.6.

Original: The result is shown in Figure 9.
Revised: The result is shown in Figure 9.

Original: When the zeroing trick is applied, the ratio of contrastive term being negative (shown as the red curve on the right subplot) is 0 . 2 , which is 1 N way where N way = 5 in our setting.
Revised: When the zeroing trick is applied, the ratio of contrastive term being negative (shown as the red curve on the right subplot) is 0 . 2 , which is 1 N way where N way = 5 in our setting.

Original: On the other hand, when the zeroing trick is not applied, the ratio of contrastive term being negative (shown as the orange color on the right subplot) is larger than 0 .
Revised: On the other hand, when the zeroing trick is not applied, the ratio of contrastive term being negative (shown as the orange color on the right subplot) is larger than 0 .

Original: This additional experiment necessitates the application of the zeroing trick.
Revised: This additional experiment necessitates the application of the zeroing trick.

Original: The original MAML, however, is a noisy SCL algorithm, and thus its ratio of negative contrastive is larger
Revised: [Figure]

Original: C A G ENERALIZATION OF OUR A NALYSIS
Revised: C A G ENERALIZATION OF OUR A NALYSIS

Original: In this section, we derive a more general case of the encoder update in the outer loop.
Revised: In this section, we derive a more general case of the encoder update in the outer loop.

Original: We consider drawing N batch tasks from the task distribution D and having N step update steps in the inner loop while keeping the EFIL assumption.
Revised: We consider drawing N batch tasks from the task distribution D and having N step update steps in the inner loop while keeping the EFIL assumption.

Original: To derive a more general case, we use w k i,n to denote the k th column of w i,n , where w i,n is updated from w 0 using support data S n for i inner-loop steps.
Revised: To derive a more general case, we use w k i,n to denote the k th column of w i,n , where w i,n is updated from w 0 using support data S n for i inner-loop steps.

Original: For simplicity, the k th channel softmax predictive output
Revised: For simplicity, the k th channel softmax predictive output

Original: [Equation]
Revised: [Equation]

Original: Nwayj
Revised: Nwayj

Original: [Equation]
Revised: [Equation]

Original: ( ϕ
Revised: ( ϕ

Original: [Equation]
Revised: [Equation]

Original: s )
Revised: s )

Original: [Equation]
Revised: [Equation]

Original: j i
Revised: j i

Original: [Equation]
Revised: [Equation]

Original: n ) of sample s (using w i − 1 ,n ) is denoted as s i,nk .
Revised: n ) of sample s (using w i − 1 ,n ) is denoted as s i,nk .

Original: Inner loop update for the linear layer We yield the inner loop update for the final linear layer in
Revised: Inner loop update for the linear layer We yield the inner loop update for the final linear layer in

Original: Eq.
Revised: Eq.

Original: (21) and Eq.
Revised: (21) and Eq.

Original: [Equation]
Revised: [Equation]

Original: Outer loop update for the linear layer We derive the outer loop update for the linear layer in SOMAML, with denoting I = { 1 , 2 , ..., N way } :
Revised: Outer loop update for the linear layer We derive the outer loop update for the linear layer in SOMAML, with denoting I = { 1 , 2 , ..., N way } :

Original: [Equation]
Revised: [Equation]

Original: When it comes to FOMAML, we have
Revised: When it comes to FOMAML, we have

Original: [Equation]
Revised: [Equation]

Original: Outer loop update for the encoder We derive the outer loop update of the encoder under FOMAML as below.
Revised: Outer loop update for the encoder We derive the outer loop update of the encoder under FOMAML as below.

Original: We consider the back-propagated error of the feature of one query data ( q, u ) ∼ Q n .
Revised: We consider the back-propagated error of the feature of one query data ( q, u ) ∼ Q n .

Original: Note that the third equality below holds by leveraging Eq.
Revised: Note that the third equality below holds by leveraging Eq.

Original: [Equation]
Revised: [Equation]

Original: Reformulating the Outer Loop Loss for the Encoder as Noisy SCL Loss.
Revised: Reformulating the Outer Loop Loss for the Encoder as Noisy SCL Loss.

Original: From Eq.
Revised: From Eq.

Original: (25), we can derive the generalized loss (of one query sample ( q, u ) ∼ Q n ) that the encoder uses under FOMAML scheme.
Revised: (25), we can derive the generalized loss (of one query sample ( q, u ) ∼ Q n ) that the encoder uses under FOMAML scheme.

Original: [Equation]
Revised: [Equation]

Original: stopgradient
Revised: stopgradient

Original: [Equation]
Revised: [Equation]

Original: D E XPERIMENTS ON MINI -I MAGE N ET D ATASET D. E XPERIMENTAL D ETAILS IN MINI -I MAGE N ET D ATASET
Revised: D E XPERIMENTS ON MINI -I MAGE N ET D ATASET D. E XPERIMENTAL D ETAILS IN MINI -I MAGE N ET D ATASET

Original: The model architecture contains four basic blocks and one fully connected linear layer, where each block comprises a convolution layer with a kernel size of 3 × 3 and filter size of 64 , batch normalization, ReLU nonlineartity and 2 × 2 max-poling.
Revised: The model architecture contains four basic blocks and one fully connected linear layer, where each block comprises a convolution layer with a kernel size of 3 × 3 and filter size of 64 , batch normalization, ReLU nonlineartity and 2 × 2 max-poling.

Original: The models are trained with the softmax cross entropy loss function using the Adam optimizer with an outer loop learning rate of 0.001 (Antoniou et al., 2019).
Revised: The models are trained with the softmax cross entropy loss function using the Adam optimizer with an outer loop learning rate of 0.001 (Antoniou et al., 2019).

Original: The inner loop step size η is set to 0.01.
Revised: The inner loop step size η is set to 0.01.

Original: The models are trained for 30000 iterations (Raghu et al., 2020).
Revised: The models are trained for 30000 iterations (Raghu et al., 2020).

Original: The results are averaged over four random seeds, and we use the shaded region to indicate the standard deviation.
Revised: The results are averaged over four random seeds, and we use the shaded region to indicate the standard deviation.

Original: Each experiment is run on either a single NVIDIA 1080-Ti or V100 GPU.
Revised: Each experiment is run on either a single NVIDIA 1080-Ti or V100 GPU.

Original: The detailed implementation is based on Long (2018) (MIT License).
Revised: The detailed implementation is based on Long (2018) (MIT License).

Original: D. T HE E XPERIMENTAL RESULT OF SOMAML
Revised: D. T HE E XPERIMENTAL RESULT OF SOMAML

Original: The results with SOMAML are shown in Figure 10.
Revised: The results with SOMAML are shown in Figure 10.

Original: Note that as it is possible that longer training can eventually overcome the noise factor and reach similar performance as the zeroing trick, the benefit of the zeroing trick is best seen at the observed faster convergence results when compared to vanilla MAML.
Revised: Note that as it is possible that longer training can eventually overcome the noise factor and reach similar performance as the zeroing trick, the benefit of the zeroing trick is best seen at the observed faster convergence results when compared to vanilla MAML.

Original: [Figure]
Revised: [Figure]

Original: D. C OSINE S IMILARITY A NALYSIS ON S EMANTICALLY S IMILAR CLASSES V ERIFIES THE I MPLICIT C ONTRASTIVENESS IN MAML
Revised: D. C OSINE S IMILARITY A NALYSIS ON S EMANTICALLY S IMILAR CLASSES V ERIFIES THE I MPLICIT C ONTRASTIVENESS IN MAML

Original: In Figure
Revised: In Figure

Original: [Figure]
Revised: [Figure]

Original: we randomly sample five classes of images under each random seed.
Revised: we randomly sample five classes of images under each random seed.

Original: Given the rich diversity of the classes in mini-ImageNet, we can consider that the five selected classes as semantically dissimilar or independent for each random seed.
Revised: Given the rich diversity of the classes in mini-ImageNet, we can consider that the five selected classes as semantically dissimilar or independent for each random seed.

Original: Here, we also provide the experimental outcomes using a dataset composed of five semantically similar classes selected from the miniImageNet dataset: French bulldog, Saluki, Walker hound, African hunting dog, and Golden retriever.
Revised: Here, we also provide the experimental outcomes using a dataset composed of five semantically similar classes selected from the miniImageNet dataset: French bulldog, Saluki, Walker hound, African hunting dog, and Golden retriever.

Original: Likewise to the original setting, we train the model using FOMAML and average the results over ten random seeds.
Revised: Likewise to the original setting, we train the model using FOMAML and average the results over ten random seeds.

Original: As shown in Figure 11, the result is consistent with Figure 2.
Revised: As shown in Figure 11, the result is consistent with Figure 2.

Original: In conclusion, we show that the supervised contrastiveness is manifested with the application of the zeroing trick even if a semantically similar dataset is considered.
Revised: In conclusion, we show that the supervised contrastiveness is manifested with the application of the zeroing trick even if a semantically similar dataset is considered.

Original: D. E XPERIMENTAL R ESULTS ON L ARGER N UMBER OF S HOTS
Revised: D. E XPERIMENTAL R ESULTS ON L ARGER N UMBER OF S HOTS

Original: To empirically verify if our theoretical derivation generalizes to the setting where the number of shots is large, we conduct experiment of a 5-way 25-shot classification task using FOMAML with four random seeds where we adopt mini-ImageNet as the example dataset.
Revised: To empirically verify if our theoretical derivation generalizes to the setting where the number of shots is large, we conduct experiment of a 5-way 25-shot classification task using FOMAML with four random seeds where we adopt mini-ImageNet as the example dataset.

Original: As shown in Figure 12, we observe that models trained with the zeroing trick again yield the best performance, consistent with our theoretical work that MAML with the zeroing trick is SCL without noises and interference.
Revised: As shown in Figure 12, we observe that models trained with the zeroing trick again yield the best performance, consistent with our theoretical work that MAML with the zeroing trick is SCL without noises and interference.

Original: [Figure]
Revised: [Figure]

Original: D.

Original: [Figure]
Revised: [Figure]

Original: The channel memorization problem (Jamal & Qi, 2019; Rajendran et al., 2020) is a known issue occurring in a non-mutually-exclusive task setting, e.g., the task-specific class-to-label is not randomly assigned, and thus the label can be inferred from the query data alone (Yin et al., 2020).
Revised: The channel memorization problem (Jamal & Qi, 2019; Rajendran et al., 2020) is a known issue occurring in a non-mutually-exclusive task setting, e.g., the task-specific class-to-label is not randomly assigned, and thus the label can be inferred from the query data alone (Yin et al., 2020).

Original: Consider a 5-way K-shot experiment where the total number of training classes is 5 × L.
Revised: Consider a 5-way K-shot experiment where the total number of training classes is 5 × L.

Original: Now we construct tasks by assigning the label t to a class sampled from class t L to ( t + 1)
Revised: Now we construct tasks by assigning the label t to a class sampled from class t L to ( t + 1)

Original: L. It is conceivable that the model will learn to directly map the query data to the label without using the information of the support data and thus fails to generalize to unseen tasks.
Revised: L. It is conceivable that the model will learn to directly map the query data to the label without using the information of the support data and thus fails to generalize to unseen tasks.

Original: This phenomenon can be explained from the perspective that the t th column of the final linear layer already accumulates the query features from t L th to ( t + 1) L th classes.
Revised: This phenomenon can be explained from the perspective that the t th column of the final linear layer already accumulates the query features from t L th to ( t + 1) L th classes.

Original: Zeroing the final linear layer implicitly forces the model to use the imprinted information from the support features for inferring the label and thus mitigates this problem.
Revised: Zeroing the final linear layer implicitly forces the model to use the imprinted information from the support features for inferring the label and thus mitigates this problem.

Original: We use the mini-ImageNet dataset and consider the case of L = 12 .
Revised: We use the mini-ImageNet dataset and consider the case of L = 12 .

Original: As shown in
Revised: As shown in

Original: Figure 13, the zeroing trick prevents the model from the channel memorization problem whereas zero-initialization of the linear layer only works out at the beginning.
Revised: Figure 13, the zeroing trick prevents the model from the channel memorization problem whereas zero-initialization of the linear layer only works out at the beginning.

Original: Besides, the performance of models trained with the zeroing trick under this non-mutually-exclusive task setting equals the ones under the conventional few-shot setting as shown in Figure 5.
Revised: Besides, the performance of models trained with the zeroing trick under this non-mutually-exclusive task setting equals the ones under the conventional few-shot setting as shown in Figure 5.

Original: As the zeroing trick clears out the final linear layer and equalizes the value of logits, our result essentially accords with Jamal & Qi (2019) that proposes a regularizer to maximize the entropy of prediction of the meta-initialized model.
Revised: As the zeroing trick clears out the final linear layer and equalizes the value of logits, our result essentially accords with Jamal & Qi (2019) that proposes a regularizer to maximize the entropy of prediction of the meta-initialized model.

Original: [Figure]
Revised: [Figure]

Original: E E XPERIMENTS ON O MNIGLOT D ATASET
Revised: E E XPERIMENTS ON O MNIGLOT D ATASET

Original: Omniglot is a hand-written character dataset containing 1623 character classes, each with 20 drawn samples from different people (Lake et al., 2015).
Revised: Omniglot is a hand-written character dataset containing 1623 character classes, each with 20 drawn samples from different people (Lake et al., 2015).

Original: The dataset set is splitted into training ( classes), validation (172 classes) and testing (423 classes) sets (Vinyals et al., 2016).
Revised: The dataset set is splitted into training ( classes), validation (172 classes) and testing (423 classes) sets (Vinyals et al., 2016).

Original: Since we follow
Revised: Since we follow

Original: Finn et al.
Revised: Finn et al.

Original: (2017) for setting hyperparamters, we do not use the the validation data.
Revised: (2017) for setting hyperparamters, we do not use the the validation data.

Original: The character images are resized to 28 × 28.
Revised: The character images are resized to 28 × 28.

Original: For all our experiments, we adopt two experimental settings: 
Revised: For all our experiments, we adopt two experimental settings: 5way 1-shot and 5-way shot where the batch size N batch is 32 and N query is 15 for both cases (Finn et al. The inner loop learning rate η is The models are trained for 3000 iterations using FOMAML or SOMAML. The few-shot classification accuracy is calculated by averaging the results over 1000 tasks in the test stage.

Original: The detailed implementation is based on Deleu (2020) (MIT License).
Revised: The model architecture follows the architecture used to train on mini-ImageNet, but we substitute the convolution with max-pooling with strided convolution operation as in Finn et al. The loss function, optimizer, and outer loop learning rate are the same as those used in the experiments on mini-ImageNet. Each experiment is run on either a single NVIDIA 1080-Ti.

Revised: The results are averaged over four random seeds, and the standard deviation is illustrated with the shaded region.

Revised: The models are trained using FOMAML unless stated otherwise.

Revised: The detailed implementation is based on Deleu (2020) (MIT License).

Original: We revisit the application of the zeroing trick at the testing stage on Omniglot in Figure 14 and observe the increasing testing accuracy, in which such results are compatible with the ones on miniImageNet (cf.
Revised: We revisit the application of the zeroing trick at the testing stage on Omniglot in Figure 14 and observe the increasing testing accuracy, in which such results are compatible with the ones on miniImageNet (cf.

Original: Figure 3 in the main manuscript).
Revised: Figure 3 in the main manuscript).

Original: In the following experiments, we evaluate the testing performance only after applying the zeroing trick.
Revised: In the following experiments, we evaluate the testing performance only after applying the zeroing trick.

Original: In Figure 15, the distinction between the performance of models trained with the zeroing trick and zero-initialized models is prominent, sharing remarkable similarity with the results in miniImageNet (cf.
Revised: In Figure 15, the distinction between the performance of models trained with the zeroing trick and zero-initialized models is prominent, sharing remarkable similarity with the results in miniImageNet (cf.

Original: Figure 5 in the main manuscript) in both 5-way 1-shot and 5-way 5-shot settings.
Revised: Figure 5 in the main manuscript) in both 5-way 1-shot and 5-way 5-shot settings.

Original: We also show the testing performance of models trained using SOMAML in Figure 16 under a 5-way 5-shot setting, where there is little distinction in performance (in comparison to the results on miniImageNet, cf.
Revised: We also show the testing performance of models trained using SOMAML in Figure 16 under a 5-way 5-shot setting, where there is little distinction in performance (in comparison to the results on miniImageNet, cf.

Original: Figure 10 in the main manuscript) between the models trained with the zeroing trick and the ones trained with random initialization.
Revised: Figure 10 in the main manuscript) between the models trained with the zeroing trick and the ones trained with random initialization.

Original: For channel memorization task, we construct non-mutually-exclusive training tasks by assigning the label t (where 1 ≤ t ≤ 5 in a few-shot 5-way setting) to a class sampled from class t L to ( t + 1) L where L is 205 on Omniglot.
Revised: For channel memorization task, we construct non-mutually-exclusive training tasks by assigning the label t (where 1 ≤ t ≤ 5 in a few-shot 5-way setting) to a class sampled from class t L to ( t + 1) L where L is 205 on Omniglot.

Original: The class-to-channel assignment is not applied to the testing tasks.
Revised: The class-to-channel assignment is not applied to the testing tasks.

Original: The result is shown in Figure 17.
Revised: The result is shown in Figure 17.

Original: For a detailed discussion, please refer to Section D.5.
Revised: For a detailed discussion, please refer to Section D.5.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: zero- is more prominent in

Original: [Figure] [Figure]
Revised: [Figure] [Figure]