Original: Unsupervised Meta-Learning via Few-shot Pseudo-supervised Contrastive Learning
Revised: Unsupervised Meta-learning via Few-shot Pseudo-supervised Contrastive Learning

Original: Abstract
Revised: Abstract

Original: Unsupervised meta-learning aims to learn generalizable knowledge across a distribution of tasks constructed from unlabeled data.
Revised: Unsupervised meta-learning aims to learn generalizable knowledge across a distribution of tasks constructed from unlabeled data.

Original: Here, the main challenge ishow to construct diverse tasks for meta-learning without label information; recentworks have proposed to create, e.g., pseudo-labeling via pretrained representationsor creating synthetic samples via generative models.
Revised: Here, the main challenge is how to construct diverse tasks for meta-learning without label information; recent works have proposed to create, e.g., pseudo-labeling via pretrained representations or creating synthetic samples via generative models.

Original: However, such a task construction strategy is fundamentally limited due to heavy reliance on the immutablepseudo-labels during meta-learning and the quality of the representations or thegenerated samples.
Revised: However, such a task construction strategy is fundamentally limited due to heavy reliance on the immutable pseudo-labels during meta-learning and the quality of the representations or the generated samples.

Original: To overcome the limitations, we propose a simple yet effective unsupervised meta-learning framework, coined P seudo- s upervised Co ntrast( PsCo ) , for few-shot classification.
Revised: To overcome the limitations, we propose a simple yet effective unsupervised meta-learning framework, coined P seudo- s upervised Co ntrast ( PsCo ) , for few-shot classification.

Original: We are inspired by the recent self-supervisedlearning literature; PsCo utilizes a momentum network and a queue of previousbatches to improve pseudo-labeling and construct diverse tasks in a progressivemanner.
Revised: We are inspired by the recent self-supervised learning literature; PsCo utilizes a momentum network and a queue of previous batches to improve pseudo-labeling and construct diverse tasks in a progressive manner.

Original: Our extensive experiments demonstrate that PsCo outperforms existingunsupervised meta-learning methods under various in-domain and cross-domainfew-shot classification benchmarks.
Revised: Our extensive experiments demonstrate that PsCo outperforms existing unsupervised meta-learning methods under various in-domain and cross-domain few-shot classification benchmarks.

Original: We also validate that PsCo is easily scalable toa large-scale benchmark, while recent prior-art meta-schemes are not.
Revised: We also validate that PsCo is easily scalable to a large-scale benchmark, while recent prior-art meta-schemes are not.

Original:  Introduction
Revised:  Introduction

Original: Meta-learning [1] aims to learn general knowledge about how to solve unseen, yet relevant tasks fromprior experiences solving diverse tasks.
Revised: Meta-learning [1] aims to learn general knowledge about how to solve unseen, yet relevant tasks from prior experiences solving diverse tasks.

Original: Few-shot classification [2, 3] is the most popular applicationof meta-learning, whose goal is to classify test samples of unseen classes after (meta-)training withfew labeled samples.
Revised: Few-shot classification [2, 3] is the most popular application of meta-learning, whose goal is to classify test samples of unseen classes after (meta-)training with few labeled samples.

Original: The common approach is to construct a distribution of tasks (i e., N -way
Revised: The common approach is to construct a distribution of tasks (i e., N -way

Original: K -shot) and optimize a model to generalize across tasks.
Revised: K -shot) and optimize a model to generalize across tasks.

Original: This approach has shown good performancebut suffers from limited scalability as constructing tasks requires a lot of human-annotated labels.
Revised: This approach has shown good performance but suffers from limited scalability as constructing tasks requires a lot of human-annotated labels.

Original: Tomitigate the issue, unsupervised meta-learning (UML) [4, 5, 6, 7, 8] attemps to apply meta-learning tounlabeled data.
Revised: To mitigate the issue, unsupervised meta-learning (UML) [4, 5, 6, 7, 8] attempts to apply meta-learning to unlabeled data.

Original: In particular, they have suggested various ways to construct synthetic tasks: assigningpseudo-labels [4, 5]; utilizing generative models [6, 7, 8].
Revised: In particular, they have suggested various ways to construct synthetic tasks: assigning pseudo-labels [4, 5]; utilizing generative models [6, 7, 8].

Original: They have achieved moderate performancein few-shot learning benchmarks, but are fundamentally limited as: (a) the pseudo-labels are fixedduring meta-learning and impossible to correct mislabeled samples; (b) the generative models heavilyrely on the quality of generated samples and are cumbersome to scale into large-scale setups.
Revised: They have achieved moderate performance in few-shot learning benchmarks, but are fundamentally limited as: (a) the pseudo-labels are fixed during meta-learning and impossible to correct mislabeled samples; (b) the generative models heavily rely on the quality of generated samples and are cumbersome to scale into large-scale setups.

Original: To overcome the limitations of the existing UML approaches, in this paper, we ask whether onecan (a) progressively improve a pseudo-labeling strategy during meta-learning, and (b) constructmore diverse tasks without generative models.
Revised: To overcome the limitations of the existing UML approaches, in this paper, we ask whether one can (a) progressively improve a pseudo-labeling strategy during meta-learning, and (b) construct more diverse tasks without generative models.

Original: We draw inspiration from recent advances in selfsupervised learning literature [9, 10], which has shown remarkable success in representation learningwithout labeled data.
Revised: We draw inspiration from recent advances in selfsupervised learning literature [9, 10], which has shown remarkable success in representation learning without labeled data.

Original: In particular, we utilize (a) a momentum network to improve pseudo-labelingprogressively via temporal ensemble; and (b) a momentum queue to construct diverse tasks usingprevious mini-batches in an online manner.
Revised: In particular, we utilize (a) a momentum network to improve pseudo-labeling progressively via temporal ensemble; and (b) a momentum queue to construct diverse tasks using previous mini-batches in an online manner.

Original: [Figure]
Revised: [Figure]

Original: Formally, we propose P seudo- s upervised Co ntrast (PsCo), a novel and effective unsupervised metalearning framework, for few-shot classification.
Revised: Formally, we propose P seudo- s upervised Co ntrast (PsCo), a novel and effective unsupervised metalearning framework, for few-shot classification.

Original: Our key idea is to construct few-shot classificationtasks using the current and previous mini-batches based on the momentum network and the momentumqueue.
Revised: Our key idea is to construct few-shot classification tasks using the current and previous mini-batches based on the momentum network and the momentum queue.

Original: Specifically, given a random mini-batch of N unlabeled samples, we treat them as N queries(i.e. , test samples) of different N labels, and then select K shots (i.e., training samples) from thequeue of previous mini-batches with the momentum network.
Revised: Specifically, given a random mini-batch of N unlabeled samples, we treat them as N queries (i.e., test samples) of different N labels, and then select K shots (i.e., training samples) from the queue of previous mini-batches with the momentum network.

Original: To further improve the selectionprocedure, we utilize top- K sampling after applying a matching algorithm, Sinkhorn-Knopp [11].
Revised: To further improve the selection procedure, we utilize top- K sampling after applying a matching algorithm, Sinkhorn-Knopp [11].

Original: Finally, we optimize our model via supervised contrastive learning [10] for solving the N -way K -shottask.
Revised: Finally, we optimize our model via supervised contrastive learning [10] for solving the N -way K -shot task.

Original: Remark that our task construction strategy (a) is progressively improved during meta-learningwith the momentum network, and (b) constructs diverse tasks since the shots can be selected from theapproximately entire dataset with the queue.
Revised: Remark that our task construction strategy (a) is progressively improved during meta-learning with the momentum network, and (b) constructs diverse tasks since the shots can be selected from the approximately entire dataset with the queue.

Original: Our framework is illustrated in Figure 1.
Revised: Our framework is illustrated in Figure 1.

Original: Throughout extensive experiments, we demonstrate the effectiveness of PsCo, under various few-shotclassification benchmarks.
Revised: Throughout extensive experiments, we demonstrate the effectiveness of PsCo, under various few-shot classification benchmarks.

Original: First, PsCo achieves state-of-the-art performance under both Omniglot[12] and miniImageNet [13] few-shot benchmarks (e.g., 58 .
Revised: First, PsCo achieves state-of-the-art performance under both Omniglot [12] and miniImageNet [13] few-shot benchmarks (e.g., 58 .

Original: 03% → 63 .
Revised: 03% → 63 .

Original: 26% for 5-way 5-shot tasksof miniImageNet); its performance is even competitive with supervised meta-learning methods (see
Revised: 26% for 5-way 5-shot tasks of miniImageNet); its performance is even competitive with supervised meta-learning methods (see

Original: Table 1a).
Revised: Table 1a).

Original: Next, PsCo also shows superiority under cross-domain few-shot learning scenarios (see
Revised: Next, PsCo also shows superiority under cross-domain few-shot learning scenarios (see

Original: Table 1b).
Revised: Table 1b).

Original: We also demonstrate the scalability of PsCo to a large-scale benchmark in Table 1c.
Revised: We also demonstrate the scalability of PsCo to a large-scale benchmark in Table 1c.

Original:  Method
Revised:  Method

Original: In this section, we introduce P seudo- s upervised Co ntrast ( PsCo ), a novel and effective framework forunsupervised few-shot learning.
Revised: In this section, we introduce P seudo- s upervised Co ntrast ( PsCo ), a novel and effective framework for unsupervised few-shot learning.

Original: The problem statement and contrastive learning are in Section 2.1.
Revised: The problem statement and contrastive learning are in Section 2.1.

Original: The details of our frameworks are in Section 2.2.
Revised: The details of our frameworks are in Section 2.2.

Original: We provide pseudocode of PsCo in Appendix A.
Revised: We provide pseudocode of PsCo in Appendix A.

Original: 2.1 Preliminaries
Revised: 2.1 Preliminaries

Original: Problem statement.
Revised: Problem statement.

Original: Our goal is to learn generalizable knowledge from unlabeled data D meta _ train :=
Revised: Our goal is to learn generalizable knowledge from unlabeled data D meta _ train :=

Original: { x i } for quickly adapting to unseen but relevant N -way K -shot few-shot tasks {T i } ∼ D meta _ test .
Revised: { x i } for quickly adapting to unseen but relevant N -way K -shot few-shot tasks {T i } ∼ D meta _ test .

Original: Each T i consists of query samples { x q } q and support samples S
Revised: Each T i consists of query samples { x q } q and support samples S

Original: = { ( x s , y s ) } NKs =1 where there are
Revised: = { ( x s , y s ) } NKs =1 where there are

Original: K support samples for each label y s ∈ { 1 , . .
Revised: K support samples for each label y s ∈ { 1 , . .

Original: . , N } .
Revised: . , N } .

Original: Contrastive learning [14, 15, 9, 10] aims to learn meaningful representations by maximizing thesimilarity between similar (i.e., positive) samples, and minimizing the similarity between dissimilar(i.e., negative) samples on the representation space; the general form of the objectives is as follows:
Revised: Contrastive learning [14, 15, 9, 10] aims to learn meaningful representations by maximizing the similarity between similar (i.e., positive) samples, and minimizing the similarity between dissimilar (i.e., negative) samples on the representation space; the general form of the objectives is as follows:

Original: [Equation]
Revised: [Equation]

Original: where q i and k
Revised: where q i and k

Original: i are query and key representations, respectively, and A ∈ { 0 , 1 } NM such that
Revised: i are query and key representations, respectively, and A ∈ { 0 , 1 } NM such that

Original: A i,j = 1 if and only if q i and k j are positive.
Revised: A i,j = 1 if and only if q i and k j are positive.

Original: Following the literature [15, 16, 17, 18], in this paper, weuse the following form: q i = Normalize ( h θ ◦ g θ ◦ f θ ( t i, 1 ( x i ))) and k i = Normalize ( g ϕ ◦ f ϕ
Revised: Following the literature [15, 16, 17, 18], in this paper, we use the following form: q i = Normalize ( h θ ◦ g θ ◦ f θ ( t i, 1 ( x i ))) and k i = Normalize ( g ϕ ◦ f ϕ

Original: ( t i, 2 ( x i )))
Revised: ( t i, 2 ( x i )))

Original: where t is a random data augmentation, f is a backbone feature extractor, g and h are projection andprediction MLPs, respectively, ϕ is an exponential moving average (EMA) of the model parameter θ .
Revised: where t is a random data augmentation, f is a backbone feature extractor, g and h are projection and prediction MLPs, respectively, ϕ is an exponential moving average (EMA) of the model parameter θ .

Original: 2 PsCo: pseudo-supervised contrastive meta-learning
Revised: 2.2 PsCo: pseudo-supervised contrastive meta-learning

Original: Online pseudo-task construction.
Revised: Online pseudo-task construction.

Original: Let B := { x i } Ni =1 ⊂ D
Revised: Let B := { x i } Ni =1 ⊂ D

Original: meta _ train be a (random) current minibatch and Q :=
Revised: meta _ train be a (random) current minibatch and Q :=

Original: { ˜ x j } Mj =1 be a queue of previous mini-batch samples.
Revised: { ˜ x j } Mj =1 be a queue of previous mini-batch samples.

Original: Our idea is to treat B as queriesof N different pseudo-labels and a subset of Q as K support samples for each pseudo-label.
Revised: Our idea is to treat B as queries of N different pseudo-labels and a subset of Q as K support samples for each pseudo-label.

Original: To utilizeall the samples efficiently and consistently, we follow MoCo [9]: we compute the momentum queryrepresentations
Revised: To utilize all the samples efficiently and consistently, we follow MoCo [9]: we compute the momentum query representations

Original: z i :=
Revised: z i :=

Original: Normalize ( g ϕ ◦ f ϕ
Revised: Normalize ( g ϕ ◦ f ϕ

Original: ( t i, 2 ( x i )))
Revised: ( t i, 2 ( x i )))

Original: and store them into the queue Q z :
Revised: and store them into the queue Q z :

Original: =
Revised: =

Original: { ˜ z j } Mj =1 .
Revised: { ˜ z j } Mj =1 .

Original: To construct semantically meaningful few-shot tasks, we need two requirements: (i) shots and queriesof the same label should be semantically similar, and (ii) all shots should be different.
Revised: To construct semantically meaningful few-shot tasks, we need two requirements: (i) shots and queries of the same label should be semantically similar, and (ii) all shots should be different.

Original: Based on theserequirements, we formulate our assignment problem as follows:
Revised: Based on these requirements, we formulate our assignment problem as follows:

Original: [Equation]
Revised: [Equation]

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: Due to the expensive cost of solving the above assignment problem [19], we first find a (soft) equalsized clustering assignment matrix ˜
Revised: Due to the expensive cost of solving the above assignment problem [19], we first find a (soft) equalsized clustering assignment matrix ˜

Original: A ∈ [0 , 1] N × M by Sinkhorn-Knopp [11] following [20, 21].
Revised: A ∈ [0 , 1] N × M by Sinkhorn-Knopp [11] following [20, 21].

Original: Wethen select top- K elements for each row of ˜ A and finally construct an N -way K -shot pseudo-taskconsisting of (a) query samples B = { x i } Ni =1 , (b) the support representations S z :=
Revised: We then select top- K elements for each row of ˜ A and finally construct an N -way K -shot pseudo-task consisting of (a) query samples B = { x i } Ni =1 , (b) the support representations S z :=

Original: { ˜ z s } NKs =1 , and(c) the pseudo-label assignment matrix A ∈ { 0 , 1 } N × NK .
Revised: { ˜ z s } NKs =1 , and (c) the pseudo-label assignment matrix A ∈ { 0 , 1 } N × NK .

Original: We empirically observe that our taskconstruction strategy satisfies the above requirements (i) and (ii) (see Appendix C).
Revised: We empirically observe that our task construction strategy satisfies the above requirements (i) and (ii) (see Appendix C).

Original: Meta-training.
Revised: Meta-training.

Original: Our objective L PsCo for learning our pseudo-tasks is defined as follows:
Revised: Our objective L PsCo for learning our pseudo-tasks is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where q i :=
Revised: where q i :=

Original: Normalize ( h θ ◦ g θ ◦ f θ ( t i, 1 ( x i ))) is query representations obtained by θ .
Revised: Normalize ( h θ ◦ g θ ◦ f θ ( t i, 1 ( x i ))) is query representations obtained by θ .

Original: S z :=
Revised: S z :=

Original: { ˜ z s } NKs = 1and A ∈ { 0 , 1 } N × NK are constructed by our task construction strategy.
Revised: { ˜ z s } NKs = and A ∈ { 0 , 1 } N × NK are constructed by our task construction strategy.

Original: Since PsCo and MoCo [9]use the same architectural components, the MoCo objective L MoCo :=
Revised: Since PsCo and MoCo [9] use the same architectural components, the MoCo objective L MoCo :

Original:  L Contrast ( { q i } Ni =1 , { z i } Ni =1 ∪
Revised: = L Contrast ( { q i } Ni =1 , { z i } Ni =1 ∪

Original: Q z , A MoCo ; τ MoCo ) can be incorporated into PsCo without additional costs.
Revised: Q z , A MoCo ; τ MoCo ) can be incorporated into PsCo without additional costs.

Original: Note that ( A MoCo ) i,j = 1if and only if i = j , and z i :=
Revised: Note that ( A MoCo ) i,j =if and only if i = j , and z i :=

Original: Normalize ( g ϕ ◦ f ϕ
Revised: Normalize ( g ϕ ◦ f ϕ

Original: ( t i, 2 ( x i )))
Revised: ( t i, 2 ( x i )))

Original: following [9].
Revised: following [9].

Original: We optimize our model θvia L total :=
Revised: We optimize our model θ via L total :=

Original: L PsCo + L MoCo , and we update ϕ by EMA, i.e., ϕ ← mϕ + (1 − m ) θ .
Revised: L PsCo + L MoCo , and we update ϕ by EMA, i.e., ϕ ← mϕ + (1 − m ) θ .

Original: To successfully find the pseudo-label assignment matrix A , we apply weak augmentations for themomentum representations as Zheng et al.
Revised: To successfully find the pseudo-label assignment matrix A , we apply weak augmentations for the momentum representations as Zheng et al.

Original: [22] did.
Revised: [22] did.

Original: This reduces the noise in the representations andconsequently enhances the performance of PsCo as A becomes more accurate (see Appendix C).
Revised: This reduces the noise in the representations and consequently enhances the performance of PsCo as A becomes more accurate (see Appendix C).

Original: Meta-test.
Revised: Meta-test.

Original: At the meta-test stage, we discard the momentum network ϕ and use only the onlinenetwork θ .
Revised: At the meta-test stage, we discard the momentum network ϕ and use only the online network θ .

Original: Given N -way K -shot task T consisting of query samples { x q } and support samples
Revised: Given N -way K -shot task T consisting of query samples { x q } and support samples

Original: S = { ( x s , y s ) } NKs =1 , we first compute the query representation q q :=
Revised: S = { ( x s , y s ) } NKs =1 , we first compute the query representation q q :=

Original: Normalize ( h θ ◦ g θ ◦ f θ ( x q ))and the support representations z s :=
Revised: Normalize ( h θ ◦ g θ ◦ f θ ( x q )) and the support representations z s :=

Original: Normalize ( g θ ◦ f θ ( x s ))) .
Revised: Normalize ( g θ ◦ f θ ( x s ))) .

Original: Then we predict a label by thefollowing rule:ˆ y :=
Revised: Then we predict a label by the following rule:

Original:  arg max y q ⊤ q c y where c
Revised: ˆ y := arg max y q ⊤ q c y where c

Original: y :=
Revised: y :=

Original: Normalize ( (cid:80) s 1 y s =
Revised: Normalize ( (cid:80) s 1 y s =

Original: y · z s ) is the prototype vector.
Revised: y · z s ) is the prototype vector.

Original: This is inspired by our L PsCo , which can be interpreted as minimizing distance from the mean (i.e.,prototype) of the shot representations.
Revised: This is inspired by our L PsCo , which can be interpreted as minimizing distance from the mean (i.e., prototype) of the shot representations.

Original: Under cross-domain few-shot classification scenarios, the model θ should further adapt to the metatest domain due to the dissimilarity from meta-training.
Revised: Under cross-domain few-shot classification scenarios, the model θ should further adapt to the metatest domain due to the dissimilarity from meta-training.

Original: We here suggest an efficient adaptationscheme using only a few labeled samples.
Revised: We here suggest an efficient adaptation scheme using only a few labeled samples.

Original: Our idea is to consider the support samples as queries:compute the query representation q s :=
Revised: Our idea is to consider the support samples as queries: compute the query representation q s :=

Original: Normalize ( h θ ◦ g θ ◦ f θ ( x s )) for each support sample x s ,and construct the label assignment matrix A ′ as A ′ s,s ′
Revised: Normalize ( h θ ◦ g θ ◦ f θ ( x s )) for each support sample x s , and construct the label assignment matrix A ′ as A ′ s,s ′

Original: = 1 if and only if y s =
Revised: = 1 if and only if y s =

Original: y s ′ .
Revised: y s ′ .

Original: Then we simplyoptimize only g θ and h θ via L Contrast ( { q s } , { z s } , A ′ ; τ PsCo ) , for few iterations.
Revised: Then we simply optimize only g θ and h θ via L Contrast ( { q s } , { z s } , A ′ ; τ PsCo ) , for few iterations.

Original: We empiricallyobserve that this adaptation scheme is effective under cross-domain settings (see Appendix C).[Equation] depending on A .
Revised: We empirically observe that this adaptation scheme is effective under cross-domain settings (see Appendix C).

Original: [Table]
Revised: [Table]

Original:  Experiments
Revised:  Experiments

Original: Setup.
Revised: Setup.

Original: Following Lee et al.
Revised: Following Lee et al.

Original: [7], we primarily focus on standard few-shot benchmarks, Omniglot [12]and miniImageNet [13] with Conv4 and Conv5 architectures, respectively.
Revised: [7], we primarily focus on standard few-shot benchmarks, Omniglot [12] and miniImageNet [13] with Conv4 and Conv5 architectures, respectively.

Original: All the implementation,standard and cross-domain benchmark details are described in Appendix D, E, and F, respectively.
Revised: All the implementation, standard and cross-domain benchmark details are described in Appendix D, E, and F, respectively.

Original: Standard few-shot benchmarks.
Revised: Standard few-shot benchmarks.

Original: Table 1a shows the results of the few-shot classification withvarious (way, shot) tasks of Omniglot and miniImageNet.
Revised: Table 1a shows the results of the few-shot classification with various (way, shot) tasks of Omniglot and miniImageNet.

Original: The proposed framework, PsCo, achievesstate-of-the-art performance on both Omniglot and miniImageNet benchmarks under the unsupervisedsetting.
Revised: The proposed framework, PsCo, achieves state-of-the-art performance on both Omniglot and miniImageNet benchmarks under the unsupervised setting.

Original: For example, we obtain 5% accuracy gain (67.07 → 72.22) compared to the prior art, MetaSVEBM [8], on miniImageNet 5-way 20-shot tasks.
Revised: For example, we obtain 5% accuracy gain (67.07 → 72.22) compared to the prior art, MetaSVEBM [8], on miniImageNet 5-way 20-shot tasks.

Original: Moreover, the performance is even competitivewith supervised meta-learning methods, ProtoNets [2], and MAML [3] as well.
Revised: Moreover, the performance is even competitive with supervised meta-learning methods, ProtoNets [2], and MAML [3] as well.

Original: Cross-domain few-shot benchmarks.
Revised: Cross-domain few-shot benchmarks.

Original: To further evaluate the generalization ability across morediverse tasks, we evaluate PsCo on cross-domain few-shot classification benchmarks following
Revised: To further evaluate the generalization ability across more diverse tasks, we evaluate PsCo on cross-domain few-shot classification benchmarks following

Original: Oh et al.
Revised: Oh et al.

Original: The benchmark details are described in Appendix F. We here use our adaptationscheme (Section 2.2) with 50 iterations.
Revised: The benchmark details are described in Appendix F. We here use our adaptation scheme (Section 2.2) with 50 iterations.

Original: We first evaluate various Conv5 models meta-trained onminiImageNet.
Revised: We first evaluate various Conv5 models meta-trained on miniImageNet.

Original: Table 1b shows that PsCo outperforms all the baselines across all the benchmarks,except ChestX, which is too different from the distribution of miniImageNet [23].
Revised: Table 1b shows that PsCo outperforms all the baselines across all the benchmarks, except ChestX, which is too different from the distribution of miniImageNet [23].

Original: Somewhatinterestingly, PsCo performs better than supervised learning under these benchmarks.
Revised: Somewhat interestingly, PsCo performs better than supervised learning under these benchmarks.

Original: We further validate that our meta-learning framework is applicable to the large-scale benchmark,
Revised: We further validate that our meta-learning framework is applicable to the large-scale benchmark,

Original: ImageNet [24], using ResNet-50 [25].
Revised: ImageNet [24], using ResNet-50 [25].

Original: Table 1c shows that (i) PsCo consistently outperforms MoCov2 [16] under this setup (e.g., 12% accuracy gain in CUB), and (ii) PsCo much adds benefits fromthe large-scale dataset as we obtain a huge amount of performance gain on the benchmarks.
Revised: Table 1c shows that (i) PsCo consistently outperforms MoCo v2 [16] under this setup (e.g., 12% accuracy gain in CUB), and (ii) PsCo much adds benefits from the large-scale dataset as we obtain a huge amount of performance gain on the benchmarks.

Original: Theseresults show that our PsCo is applicable to large-scale unlabeled datasets.
Revised: These results show that our PsCo is applicable to large-scale unlabeled datasets.

Original: Ablation study.
Revised: Ablation study.

Original: We provide ablation experiments to validate PsCo’s components in Appendix C.
Revised: We provide ablation experiments to validate PsCo’s components in Appendix C.

Original: B Related works
Revised: B Related works

Original: Unsupervised meta-learning.
Revised: Unsupervised meta-learning.

Original: Unsupervised meta-learning [4, 5, 7, 8, 6] links meta-learning andunsupervised learning by constructing synthetic tasks and extracting the meaningful informationfrom unlabeled data.
Revised: Unsupervised meta-learning [4, 5, 7, 8, 6] links meta-learning and unsupervised learning by constructing synthetic tasks and extracting the meaningful information from unlabeled data.

Original: For example, CACTUs [4] cluster the data on the pretrained representationsat the beginning of meta-learning to assign pseudo-labels.
Revised: For example, CACTUs [4] cluster the data on the pretrained representations at the beginning of meta-learning to assign pseudo-labels.

Original: Instead of pseudo-labeling, UMTRA[5] and LASIUM [6] generate synthetic samples using data augmentations or pretrained generativenetworks like BigBiGAN [26].
Revised: Instead of pseudo-labeling, UMTRA [5] and LASIUM [6] generate synthetic samples using data augmentations or pretrained generative networks like BigBiGAN [26].

Original: Meta-GMVAE [7] and Meta-SVEBM [8] represent unknown labelsvia categorical latent variables using variational autoencoders [27] and energy-based models [28],respectively.
Revised: Meta-GMVAE [7] and Meta-SVEBM [8] represent unknown labels via categorical latent variables using variational autoencoders [27] and energy-based models [28], respectively.

Original: In this paper, we suggest a novel online pseudo-labeling strategy to construct diversetasks without help from any pretrained network or generative model.
Revised: In this paper, we suggest a novel online pseudo-labeling strategy to construct diverse tasks without help from any pretrained network or generative model.

Original: As a result, our method is easilyapplicable to large-scale datasets.
Revised: As a result, our method is easily applicable to large-scale datasets.

Original: Self-supervised learning.
Revised: Self-supervised learning.

Original: Self-supervised learning (SSL) [29] has shown remarkable success forunsupervised representation learning across various domains, including vision [9, 15], speech [14],and reinforcement learning [30].
Revised: Self-supervised learning (SSL) [29] has shown remarkable success for unsupervised representation learning across various domains, including vision [9, 15], speech [14], and reinforcement learning [30].

Original: Among SSL objectives, contrastive learning [14, 15, 9] is arguablymost popular for learning meaningful representations.
Revised: Among SSL objectives, contrastive learning [14, 15, 9] is arguably most popular for learning meaningful representations.

Original: In addition, recent advances have been madewith the development of various architectural components: e.g., Siamese networks [29], momentumnetworks [9], and asymmetric architectures [18, 31].
Revised: In addition, recent advances have been made with the development of various architectural components: e.g., Siamese networks [29], momentum networks [9], and asymmetric architectures [18, 31].

Original: In this paper, we utilize the SSL components toconstruct diverse few-shot tasks in an unsupervised manner.
Revised: In this paper, we utilize the SSL components to construct diverse few-shot tasks in an unsupervised manner.

Original: C Ablation study
Revised: C Ablation study

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Component analysis.
Revised: Component analysis.

Original: In Table 2, we demonstrate the necessity of each component in PsCo byremoving the components one by one: momentum encoder ϕ , prediction head h , Sinkhorn-Knoppalgorithm, and top- K sampling for sampling support samples.
Revised: In Table 2, we demonstrate the necessity of each component in PsCo by removing the components one by one: momentum encoder ϕ , prediction head h , Sinkhorn-Knopp algorithm, and top- K sampling for sampling support samples.

Original: We found that the momentum networkϕ and the prediction head h are critical architectural components in our framework like recent selfsupervised learning frameworks [18, 17].
Revised: We found that the momentum network ϕ and the prediction head h are critical architectural components in our framework like recent selfsupervised learning frameworks [18, 17].

Original: To further validate that our task construction is progressivelyimproved during meta-learning, we evaluate whether a query and a corresponding support samplehave the same true label.
Revised: To further validate that our task construction is progressively improved during meta-learning, we evaluate whether a query and a corresponding support sample have the same true label.

Original: Figure 2a shows that our task construction is progressively improved, i.e.,the task requirement (i) described in Section 2.2 satisfies.
Revised: Figure 2a shows that our task construction is progressively improved, i.e., the task requirement (i) described in Section 2.2 satisfies.

Original: Table 2 also verifies the contribution of the Sinkhorn-Knopp algorithm and Top- K sampling for theperformance of PsCo.
Revised: Table 2 also verifies the contribution of the Sinkhorn-Knopp algorithm and Top- K sampling for the performance of PsCo.

Original: We further analyze the effect of the Sinkhorn-Knopp algorithm by measuringthe overlap ratio of selected supports between different pseudo-labels.
Revised: We further analyze the effect of the Sinkhorn-Knopp algorithm by measuring the overlap ratio of selected supports between different pseudo-labels.

Original: As shown in Figure 2b, thereare almost zero overlaps when using the Sinkhorn-Knopp algorithm, which means the constructedtask is a valid few-shot task, satisfying the task requirement (ii) described in Section 2.2.
Revised: As shown in Figure 2b, there are almost zero overlaps when using the Sinkhorn-Knopp algorithm, which means the constructed task is a valid few-shot task, satisfying the task requirement (ii) described in Section 2.2.

Original: Adaptation effect on cross-domain.
Revised: Adaptation effect on cross-domain.

Original: To validate the effect of our adaptation scheme (Section 2.2),we evaluate the few-shot classification accuracy during the adaptation process on miniImageNet (i.e.,in-domain) and CropDiseases (i.e., cross-domain) benchmarks.
Revised: To validate the effect of our adaptation scheme (Section 2.2), we evaluate the few-shot classification accuracy during the adaptation process on miniImageNet (i.e., in-domain) and CropDiseases (i.e., cross-domain) benchmarks.

Original: As shown in Figure 2d, we foundthat the adaptation scheme is more useful in cross-domain benchmarks than in-domain ones.
Revised: As shown in Figure 2d, we found that the adaptation scheme is more useful in cross-domain benchmarks than in-domain ones.

Original: Basedon these results, we apply the scheme to only the cross-domain scenarios.
Revised: Based on these results, we apply the scheme to only the cross-domain scenarios.

Original: We also found that ouradaptation does not cause over-fitting since we only optimize the projection and prediction heads g θand h θ .
Revised: We also found that our adaptation does not cause over-fitting since we only optimize the projection and prediction heads g θ and h θ .

Original: The results for the adaptation effect on the whole benchmarks are represented in Table 3.
Revised: The results for the adaptation effect on the whole benchmarks are represented in Table 3.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Augmentations.
Revised: Augmentations.

Original: We here confirm that weak augmentation for the momentum network (i.e., A 2 ) ismore effective than strong augmentation unlike other self-supervised learning literature [15, 9].
Revised: We here confirm that weak augmentation for the momentum network (i [Table] e., A 2 ) is more effective than strong augmentation unlike other self-supervised learning literature [15, 9].

Original: Wedenote the standard augmentation consisting of both geometric and color transformations by Strong ,and a weaker augmentation consisting of only geometric transformations as Weak (see details in
Revised: We denote the standard augmentation consisting of both geometric and color transformations by Strong , and a weaker augmentation consisting of only geometric transformations as Weak (see details in

Original: Appendix D).
Revised: Appendix D).

Original: As shown in Table 4, utilizing the weak augmentation for A 2 is much beneficial sinceit is helpful for finding an accurate pseudo-label assignment matrix A .
Revised: As shown in Table 4, utilizing the weak augmentation for A 2 is much beneficial since it is helpful for finding an accurate pseudo-label assignment matrix A .

Original: Training K .
Revised: Training K .

Original: We also look at the effect of the training K , i.e. number of shots sampled online.
Revised: We also look at the effect of the training K , i [Table] e. number of shots sampled online.

Original: We conduct the experiment with K ∈ { 1 , 4 , 16 , 64 } .
Revised: We conduct the experiment with K ∈ { 1 , 4 , 16 , 64 } .

Original: We observe that PsCo performs consistentlywell regardless of the choice of K as shown in Table 5.
Revised: We observe that PsCo performs consistently well regardless of the choice of K as shown in Table 5.

Original: The proper K is suggested to obtain thebest-performing models, e.g., K = 4 for miniImageNet and K = 1 for Omniglot are the best.
Revised: The proper K is suggested to obtain the best-performing models, e.g., K = 4 for miniImageNet and K = 1 for Omniglot are the best.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Momentum and temperature hyperparameters.
Revised: Momentum and temperature hyperparameters.

Original: For the small-scale experiments, we use amomentum of m = 0 .
Revised: For the small-scale experiments, we use a momentum of m = 0 .

Original: 99 and a temperature of τ PsCo = 1 .
Revised: 99 and a temperature of τ PsCo = 1 .

Original: We here provide more ablation experimentswith varying the hyperparameters m and τ PsCo .
Revised: We here provide more ablation experiments with varying the hyperparameters m and τ PsCo .

Original: Table 6 and 7 show the sensitivity of hyperparameterson miniImageNet dataset.
Revised: Table 6 and 7 show the sensitivity of hyperparameters on miniImageNet dataset.

Original: We observe that PsCo achieves good performance even for non-optimalhyperparameters.
Revised: We observe that PsCo achieves good performance even for non-optimal hyperparameters.

Original: D Implementation details
Revised: D Implementation details

Original: We train our models via stochastic gradient descent (SGD) with a batch size of N = 256 for 400epochs.
Revised: We train our models via stochastic gradient descent (SGD) with a batch size of N = 256 forepochs.

Original: Following Chen et al.
Revised: Following Chen et al.

Original: [16], Chen and He [31], we use an initial learning rate of 0.03 withthe cosine learning schedule, τ MoCo = 0 . 2 , and a weight deacy of 5 × 10 − 4 .
Revised: [16], Chen and He [31], we use an initial learning rate of 0.03 with the cosine learning schedule, τ MoCo = 0 . 2 , and a weight deacy of 5 × 10 − 4 .

Original: We use a queue size of
Revised: We use a queue size of

Original: M = 16384 since Omniglot [12] and miniImageNet [13] has roughly 100k meta-training samples.
Revised: M = 16384 since Omniglot [12] and miniImageNet [13] has roughly 100k meta-training samples.

Original: Following Lee et al.
Revised: Following Lee et al.

Original: [7], we use Conv4 and Conv5 for Omniglot and miniImageNet, respectively, forthe backbone feature extractor f θ .
Revised: [7], we use Conv4 and Conv5 for Omniglot and miniImageNet, respectively, for the backbone feature extractor f θ .

Original: We describe the detailed architectures in Table 8.
Revised: We describe the detailed architectures in Table 8.

Original: For projectionand prediction MLPs, g θ and h θ , we use 2-layer MLPs with a hidden size of 2048 and an outputdimenension of 128.
Revised: For projection and prediction MLPs, g θ and h θ , we use 2-layer MLPs with a hidden size of 2048 and an output dimenension of 128.

Original: For the hyperparameters of PsCo, we use τ PsCo = 1 and a momentum parameterof m = 0 .
Revised: For the hyperparameters of PsCo, we use τ PsCo = 1 and a momentum parameter of m = 0 .

Original: 99 (see Appendix C for the hyperparameter sensitivity).
Revised: 99 (see Appendix C for the hyperparameter sensitivity).

Original: For the number of shots duringmeta-learning, we use K = 1 for Omniglot and K = 4 for miniImageNet (see Table 5 for sensitivityof K ).
Revised: For the number of shots during meta-learning, we use K = 1 for Omniglot and K = 4 for miniImageNet (see Table 5 for sensitivity of K ).

Original: We use the last-epoch model for evaluation without any guidance from the meta-validationdataset.
Revised: We use the last-epoch model for evaluation without any guidance from the meta-validation dataset.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: Augmentations.
Revised: Augmentations.

Original: We describe the augmentations for Omniglot and miniImagenet in Table 9.
Revised: We describe the augmentations for Omniglot and miniImagenet in Table 9.

Original: For
Revised: For

Original: Omniglot, because it is difficult to apply many augmentations to gray-scale images, we use the samerule for weak and strong augmentations.
Revised: Omniglot, because it is difficult to apply many augmentations to gray-scale images, we use the same rule for weak and strong augmentations.

Original: For miniImageNet, we use only geometric transformationsfor the weak augmentation following Zheng et al.
Revised: For miniImageNet, we use only geometric transformations for the weak augmentation following Zheng et al.

Original: [Table]
Revised: [Table]

Original: Training procedures.
Revised: Training procedures.

Original: To ensure performance of PsCo and self-supervised learning models [Table] we usethree independently-trained models with random seeds and report the average performance of them
Revised: To ensure performance of PsCo and self-supervised learning models, we use three independently-trained models with random seeds and report the average performance of them

Original: [Table]
Revised: [Table]

Original: E Setup for standard few-shot benchmarks
Revised: E Setup for standard few-shot benchmarks

Original: We here describe details of benchmarks and baselines in Section E.1 and E.2, respectively, for thestandard few-shot classification experiments (Section 3).
Revised: We here describe details of benchmarks and baselines in Section E.1 and E.2, respectively, for the standard few-shot classification experiments (Section 3).

Original: E.1 Datasets
Revised: E.1 Datasets

Original: Omniglot [12] is a 28 × 28 gray-scale dataset of 1623 characters with 20 samples each.
Revised: Omniglot [12] is a 28 × 28 gray-scale dataset of 1623 characters with 20 samples each.

Original: We followthe setup of unsupervised meta-learning approaches [4].
Revised: We follow the setup of unsupervised meta-learning approaches [4].

Original: We split the dataset into 120, 100, and 323classes for meta-training, meta-validation, and meta-test respectively.
Revised: We split the dataset into 120, 100, andclasses for meta-training, meta-validation, and meta-test respectively.

Original: In addition, the 0, 90, 180, and270 degrees rotated views for each class become the different categories.
Revised: In addition, the 0, 90, 180, and 270 degrees rotated views for each class become the different categories.

Original: Thus, we have a total of6492, 400, and 1292 classes for meta-training, meta-validation, and meta-test respectively.
Revised: Thus, we have a total of 6492, 400, and 1292 classes for meta-training, meta-validation, and meta-test respectively.

Original: MiniImageNet [13] is an 84 × 84 resized subset of ILSVRC-2012 [24] with 600 samples each.
Revised: MiniImageNet [13] is an 84 × 84 resized subset of ILSVRC-2012 [24] with 600 samples each.

Original: We split the dataset into 64, 16, and 20 classes for meta-training, meta-validation, and meta-testrespectively as introduced in Ravi and Larochelle [13].
Revised: We split the dataset into 64, 16, and 20 classes for meta-training, meta-validation, and meta-test respectively as introduced in Ravi and Larochelle [13].

Original: E.2 Baselines
Revised: E.2 Baselines

Original: We compare our performance with unsupervised meta-learning, self-supervised learning, and supervised meta-learning methods.
Revised: We compare our performance with unsupervised meta-learning, self-supervised learning, and supervised meta-learning methods.

Original: To be specific, (a) for the unsupervised meta-learning, we use CACTUs[4] of the best options (ACAI clustering for Omniglot and DeepCluster for miniImageNet), UMTRA[5], LASIUM [30] of the best options (LASIUM-RO-GAN for Omniglot and LASIUM-N-GAN forminiImageNet), Meta-GMVAE [7], Meta-SVEBM [8]; (b) for the self-supervised learning methods,we use SimCLR [15], MoCo v2 [16], and SwAV [20]; (c) for the supervised meta-learning, we usethe results of MAML [3] and ProtoNets [2] reported in [4].
Revised: To be specific, (a) for the unsupervised meta-learning, we use CACTUs [4] of the best options (ACAI clustering for Omniglot and DeepCluster for miniImageNet), UMTRA [5], LASIUM [30] of the best options (LASIUM-RO-GAN for Omniglot and LASIUM-N-GAN for miniImageNet), Meta-GMVAE [7], Meta-SVEBM [8]; (b) for the self-supervised learning methods, we use SimCLR [15], MoCo v2 [16], and SwAV [20]; (c) for the supervised meta-learning, we use the results of MAML [3] and ProtoNets [2] reported in [4].

Original: For training self-supervised learning methods in our experimental setups, we use the same architectureand hyperparameters.
Revised: For training self-supervised learning methods in our experimental setups, we use the same architecture and hyperparameters.

Original: For the hyperparameter of temperature scaling, we use the value provided ineach paper:
Revised: For the hyperparameter of temperature scaling, we use the value provided in each paper:

Original: τ SimCLR = 0 .
Revised: τ SimCLR = 0 .

Original: 5 for SimCLR, τ MoCo = 0 .
Revised: 5 for SimCLR, τ MoCo = 0 .

Original: 2 for MoCo v2, and τ SwAV = 0 .
Revised: 2 for MoCo v2, and τ SwAV = 0 .

Original: 1 for SwAV.
Revised: 1 for SwAV.

Original: Forevaluation, we use K-Nearest Neightobrs (K-NN) for self-supervised learning methods since theirclassification rules are not specified.
Revised: For evaluation, we use K-Nearest Neightobrs (K-NN) for self-supervised learning methods since their classification rules are not specified.

Original: F Setup for cross-domain few-shot benchmarks
Revised: F Setup for cross-domain few-shot benchmarks

Original: We now describe the setup for cross-domain few-shot benchmarks, including detailed information ondatasets, baseline experiments, implementational details, and the setup for large-scale experiments.
Revised: We now describe the setup for cross-domain few-shot benchmarks, including detailed information on datasets, baseline experiments, implementational details, and the setup for large-scale experiments.

Original: F.1 Datasets
Revised: F.1 Datasets

Original: For the cross-domain few-shot benchmarks, we use eight different datasets.
Revised: For the cross-domain few-shot benchmarks, we use eight different datasets.

Original: We describe the datasetinformation in Table 10.
Revised: We describe the dataset information in Table 10.

Original: We use the dataset split described in Tseng et al.
Revised: We use the dataset split described in Tseng et al.

Original: [32] for the benchmark ofhigh-similarity
Revised: [32] for the benchmark of high-similarity

Original: and we use the dataset split described in Guo et al.
Revised: and we use the dataset split described in Guo et al.

Original: [33] for the benchmark of lowsimilarity.
Revised: [33] for the benchmark of lowsimilarity.

Original: Because we do not perform the meta-training procedure using the datasets of cross-domainbenchmarks, we only utilize the meta-test splits on these datasets.
Revised: Because we do not perform the meta-training procedure using the datasets of cross-domain benchmarks, we only utilize the meta-test splits on these datasets.

Original: We use the 84 × 84 resized samplesfor evaluation on small-scale experiments.
Revised: We use the 84 × 84 resized samples for evaluation on small-scale experiments.

Original: [Table]
Revised: [Table]

Original: F.2 Baselines
Revised: F.2 Baselines

Original: We compare our performance with (a) previous in-domain state-of-the-art methods of unsupervisedmeta-learning, self-supervised learning models, and supervised meta-learning models.
Revised: We compare our performance with (a) previous in-domain state-of-the-art methods of unsupervised meta-learning, self-supervised learning models, and supervised meta-learning models.

Original: Unsupervised meta-learning models.
Revised: Unsupervised meta-learning models.

Original: We use previous in-domain state-of-the-art methods of unsupervised meta-learning models, Meta-GMVAE[7] and Meta-SVEBM [8].
Revised: We use previous in-domain state-of-the-art methods of unsupervised meta-learning models, Meta-GMVAE[7] and Meta-SVEBM [8].

Original: We use the miniImageNetpretrained parameters that the paper provided, and follow the meta-test procedure of each model toevaluate the performance.
Revised: We use the miniImageNet pretrained parameters that the paper provided, and follow the meta-test procedure of each model to evaluate the performance.

Original: Self-supervised learning models.
Revised: Self-supervised learning models.

Original: We use SimCLR [15], MoCo v2 [16], and SwAV [20] of miniImageNet pretrained parameters as our baselines.
Revised: We use SimCLR [15], MoCo v2 [16], and SwAV [20] of miniImageNet pretrained parameters as our baselines.

Original: Because self-supervised learning models are pretrainedon miniImageNet, we additionally fine-tune the models with a linear classifier to let the models adaptto each domain.
Revised: Because self-supervised learning models are pretrained on miniImageNet, we additionally fine-tune the models with a linear classifier to let the models adapt to each domain.

Original: Following the setting provided in Guo et al.
Revised: Following the setting provided in Guo et al.

Original: [33], Oh et al.
Revised: [33], Oh et al.

Original: [23], we detach the headof the models g θ and attach the linear classifier c ψ to the model.
Revised: [23], we detach the head of the models g θ and attach the linear classifier c ψ to the model.

Original: We freeze the base network f θ whilefine-tuning and only c ψ is learned.
Revised: We freeze the base network f θ while fine-tuning and only c ψ is learned.

Original: We fine-tune the models via SGD with an initial learning rate of0.01, a momentum of 0.9, weight decay of 0.001, and a batch size of N = 4 for 100 epochs.
Revised: We fine-tune the models via SGD with an initial learning rate of 0.01, a momentum of 0.9, weight decay of 0.001, and a batch size of N = 4 for 100 epochs.

Original: Supervised meta-learning models.
Revised: Supervised meta-learning models.

Original: We use MAML [3] and ProtoNets [2] of Conv5 architectures ofminiImageNet pretrained.
Revised: We use MAML [3] and ProtoNets [2] of Conv5 architectures of miniImageNet pretrained.

Original: Following the procedure of Snell et al.
Revised: Following the procedure of Snell et al.

Original: [2], we train the models via Adam[42] with a learning rate of 0.001 and cut the learning rate in half for every training of 2000 episodes.
Revised: [2], we train the models via Adam [42] with a learning rate of 0.001 and cut the learning rate in half for every training of 2000 episodes.

Original: We train them for 60K episodes and use the model of the best validation accuracy.
Revised: We train them for 60K episodes and use the model of the best validation accuracy.

Original: We train themthrough a 5-way 5-shot, and the rest of the hyperparameters are referenced in their respective papers.
Revised: We train them through a 5-way 5-shot, and the rest of the hyperparameters are referenced in their respective papers.

Original: We observe that their performances are similar to the performance described in Table 1.
Revised: We observe that their performances are similar to the performance described in Table 1.

Original: F.3 Evaluation details
Revised: F.3 Evaluation details

Original: To evaluate our method, we apply our adaptation scheme.
Revised: To evaluate our method, we apply our adaptation scheme.

Original: Following Section 2.2, we freeze the basenetwork f θ .
Revised: Following Section 2.2, we freeze the base network f θ .

Original: We train only projection head g θ and prediction head h θ via SGD with an initial learningrate of 0.01, a momentum of 0.9, and weight decay of 0.001 as self-supervised learning models arefine-tuned.
Revised: We train only projection head g θ and prediction head h θ via SGD with an initial learning rate of 0.01, a momentum of 0.9, and weight decay of 0.001 as self-supervised learning models are fine-tuned.

Original: We only apply 50 iterations of our adaptation scheme when reporting performance.
Revised: We only apply 50 iterations of our adaptation scheme when reporting performance.

Original: F.4 Large-scale setup
Revised: F.4 Large-scale setup

Original: Here, we describe the setup for large-scale experiments.
Revised: Here, we describe the setup for large-scale experiments.

Original: For evaluating, we use the same protocolwith the small-scale experiments, except the scale of images is 224 × 224 .
Revised: For evaluating, we use the same protocol with the small-scale experiments, except the scale of images is 224 × 224 .

Original: Augmentations.
Revised: Augmentations.

Original: For large-scale experiments, we use 224 × 224 -scaled data.
Revised: For large-scale experiments, we use 224 × 224 -scaled data.

Original: Thus, we use similaryet slightly different augmentation schemes with small-scale experiments.
Revised: Thus, we use similar yet slightly different augmentation schemes with small-scale experiments.

Original: Following the strongaugmentation used in Chen et al.
Revised: Following the strong augmentation used in Chen et al.

Original: [16, 15], we additionally apply GaussianBlur as a randomaugmentation.
Revised: [16, 15], we additionally apply GaussianBlur as a random augmentation.

Original: We use the same configuration for the weak augmentation.
Revised: We use the same configuration for the weak augmentation.

Original: For evaluation, we resizethe images into 256 × 256 and then apply the CenterCrop to make 224 × 224 images by following
Revised: For evaluation, we resize the images into 256 × 256 and then apply the CenterCrop to make 224 × 224 images by following

Original: Guo et al.
Revised: Guo et al.

Original: ImageNet pretraining.
Revised: ImageNet pretraining.

Original: We pretrain MoCo v2 [16] and our PsCo of ResNet-18/50 [25] via SGDwith a batch size of N = 256 for 200 epochs.
Revised: We pretrain MoCo v2 [16] and our PsCo of ResNet-18/50 [25] via SGD with a batch size of N = 256 for 200 epochs.

Original: Following [16, 31], we use an initial learning rate of0.03 with the cosine learning schedule, τ MoCo = 0 .
Revised: Following [16, 31], we use an initial learning rate of 0.03 with the cosine learning schedule, τ MoCo = 0 .

Original: 2 and a weight decay of 0.0001.
Revised: 2 and a weight decay of 0.0001.

Original: We use a queuesize of M = 65536 and momentum of m = 0 . 999 .
Revised: We use a queue size of M = 65536 and momentum of m = 0 . 999 .

Original: For the parameters of PsCo, we use τ PsCo = 0 .
Revised: For the parameters of PsCo, we use τ PsCo = 0 .and

Original: 2and K = 16 as the queue is 4 times bigger.
Revised: K = 16 as the queue is 4 times bigger.

Original: G Experimental results with 95% confidence interval
Revised: G Experimental results with 95% confidence interval

Original: We here provide the experimental results of Table 1, 1b [Table] and 1c with 95% confidence intervals in
Revised: We here provide the experimental results of Table 1, 1b, and 1c with 95% confidence intervals in

Original: Table 11,
Revised: Table 11,

Original: [Table]
Revised: [Table]

Original: and 13, respectively.
Revised: and 13, respectively.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: to miniImageNet.

Original: [Table] [Table]
Revised: [Table] [Table]