Original: M O P RO : W EBLY S UPERVISED L EARNING WITH M OMENTUM P ROTOTYPES
Revised: M O P RO : W EBLY S UPERVISED L EARNING WITH M OMENTUM P ROTOTYPES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a webly-supervised representation learning method that does not suffer from the annotation unscalability of supervised learning, nor the computation unscalability of self-supervised learning.
Revised: We propose a webly-supervised representation learning method that does not suffer from the annotation unscalability of supervised learning, nor the computation unscalability of self-supervised learning.

Original: Most existing works on weblysupervised representation learning adopt a vanilla supervised learning method without accounting for the prevalent noise in the training data, whereas most prior methods in learning with label noise are less effective for real-world large-scale noisy data.
Revised: Most existing works on weblysupervised representation learning adopt a vanilla supervised learning method without accounting for the prevalent noise in the training data, whereas most prior methods in learning with label noise are less effective for real-world large-scale noisy data.

Original: We propose momentum prototypes (MoPro), a simple contrastive learning method that achieves online label noise correction, out-of-distribution sample removal, and representation learning.
Revised: We propose momentum prototypes (MoPro), a simple contrastive learning method that achieves online label noise correction, out-of-distribution sample removal, and representation learning.

Original: MoPro achieves state-of-the-art performance on WebVision, a weakly-labeled noisy dataset.
Revised: MoPro achieves state-of-the-art performance on WebVision, a weakly-labeled noisy dataset.

Original: MoPro also shows superior performance when the pretrained model is transferred to down-stream image classiﬁcation and detection tasks.
Revised: MoPro also shows superior performance when the pretrained model is transferred to down-stream image classiﬁcation and detection tasks.

Original: It outperforms the ImageNet supervised pretrained model by +10 .
Revised: It outperforms the ImageNet supervised pretrained model by +10 .

Original: 5 on 1-shot classiﬁcation on VOC, and outperforms the best self-supervised pretrained model by +17 . 3 when ﬁnetuned on 1% of ImageNet labeled samples.
Revised: 5 on 1-shot classiﬁcation on VOC, and outperforms the best self-supervised pretrained model by +17 . 3 when ﬁnetuned on 1% of ImageNet labeled samples.

Original: Furthermore, MoPro is more robust to distribution shifts 1 .
Revised: Furthermore, MoPro is more robust to distribution shifts.

Revised: Code and pretrained models are available at https://github.com/ salesforce/MoPro .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Large-scale datasets with human-annotated labels have revolutionized computer vision.
Revised: Large-scale datasets with human-annotated labels have revolutionized computer vision.

Original: Supervised pretraining on ImageNet (Deng et al., 2009) has been the de facto formula of success for almost all state-of-the-art visual perception models.
Revised: Supervised pretraining on ImageNet (Deng et al., 2009) has been the de facto formula of success for almost all state-of-the-art visual perception models.

Original: However, it is extremely labor intensive to manually annotate millions of images, which makes it a non-scalable solution.
Revised: However, it is extremely labor intensive to manually annotate millions of images, which makes it a non-scalable solution.

Original: One alternative to reduce annotation cost is self-supervised representation learning, which leverages unlabeled data.
Revised: One alternative to reduce annotation cost is self-supervised representation learning which leverages unlabeled data.

Original: However, self-supervised learning methods (Goyal et al., 2019;
Revised: However, self-supervised learning methods (Goyal et al., 2019;

Original: He et al., 2019; Chen et al., 2020a; Li et al., 2020b) have yet consistently shown superior performance compared to supervised learning, especially when transferred to downstream tasks with limited labels.
Revised: He et al., 2019; Chen et al., 2020a; Li et al., 2020b) have yet consistently shown superior performance compared to supervised learning, especially when transferred to downstream tasks with limited labels.

Original: With the help of commercial search engines, photo-sharing websites, and social media platforms, there is near-inﬁnite amount of weakly-labeled images available on the web.
Revised: With the help of commercial search engines, photo-sharing websites, and social media platforms, there is near-inﬁnite amount of weakly-labeled images available on the web.

Original: Several works have exploited the scalable source of web images and demonstrated promising results with weblysupervised representation learning (Mahajan et al., 2018;
Revised: Several works have exploited the scalable source of web images and demonstrated promising results with weblysupervised representation learning (Mahajan et al., 2018;

Original: Sun et al., 2017; Li et al., 2017; Kolesnikov et al., 2020).
Revised: Sun et al., 2017; Li et al., 2017; Kolesnikov et al., 2020).

Original: However, there exists two competing claims on whether weakly-labeled noisy datasets lead to worse generalization performance.
Revised: However, there exists two competing claims on whether weakly-labeled noisy datasets lead to worse generalization performance.

Original: One claim argues that the effect of noise can be overpowered by the scale of data, and simply applies standard supervised learning method on web datasets (Mahajan et al., 2018; Sun et al., 2017; Li et al., 2017; Kolesnikov et al., 2020).
Revised: One claim argues that the effect of noise can be overpowered by the scale of data, and simply applies standard supervised learning method on web datasets (Mahajan et al., 2018; Sun et al., 2017; Li et al., 2017; Kolesnikov et al., 2020).

Original: The other claim argues that deep models can easily memorize noisy labels, resulting in worse generalization (Zhang et al., 2017; Ma et al., 2018).
Revised: The other claim argues that deep models can easily memorize noisy labels, resulting in worse generalization (Zhang et al., 2017; Ma et al., 2018).

Original: In this paper, we show that both claims are partially true.
Revised: In this paper, we show that both claims are partially true.

Original: While increasing the size of data does improve the model’s robustness to noise, our method can substantially boost the representation learning performance by addressing noise.
Revised: While increasing the size of data does improve the model’s robustness to noise, our method can substantially boost the representation learning performance by addressing noise.

Original: There exists a large body of literature on learning with label noise (Jiang et al., 2018; Han et al., 2018;
Revised: There exists a large body of literature on learning with label noise (Jiang et al., 2018; Han et al., 2018;

Original: Guo et al., 2018; Tanaka et al., 2018;
Revised: Guo et al., 2018; Tanaka et al., 2018;

Original: Arazo et al., 2019; Li et al., 2020a).
Revised: Arazo et al., 2019; Li et al., 2020a).

Original: However, existing methods have several limitations that make them less effective for webly-supervised representation learning.
Revised: However, existing methods have several limitations that make them less effective for webly-supervised representation learning.

Original: First, most methods do not consider out-of-distribution (OOD) samples, which is a major source of noise in real-world web datasets.
Revised: First, most methods do not consider out-of-distribution (OOD) samples, which is a major source of noise in real-world web datasets.

Original: Second, many methods perform computation-heavy procedures for noise cleaning (Jiang et al., 2018; Li et al., 2019; 2020a), or require access to a set of samples with clean labels (Vahdat, 2017; Veit et al., 2017; Lee et al., 2018), which limit their scalability in practice.
Revised: Second, many methods perform computation-heavy procedures for noise cleaning (Jiang et al., 2018; Li et al., 2019; 2020a), or require access to a set of samples with clean labels (Vahdat, 2017; Veit et al., 2017; Lee et al., 2018), which limit their scalability in practice.

Original: We propose a new method for efﬁcient representation learning from weakly-labeled web images.
Revised: We propose a new method for efﬁcient representation learning from weakly-labeled web images.

Original: Our method is inspired by recent developments in contrastive learning for self-supervised learning (He et al., 2019;
Revised: Our method is inspired by recent developments in contrastive learning for self-supervised learning (He et al., 2019;

Original: Chen et al., 2020a; Li et al., 2020b)
Revised: Chen et al., 2020a; Li et al., 2020b)

Original: We introduce Momentum Prototypes (MoPro), a simple component which is effective in label noise correction , OOD sample removal , and representation learning .
Revised: We introduce Momentum Prototypes (MoPro), a simple component which is effective in label noise correction , OOD sample removal , and representation learning .

Original: A visual explanation of our method is shown in Figure 1.
Revised: A visual explanation of our method is shown in Figure 1.

Original: We use a deep network to project images into normalized low-dimensional embeddings, and calculate the prototype for a class as the moving-average embedding for clean samples in that class.
Revised: We use a deep network to project images into normalized low-dimensional embeddings, and calculate the prototype for a class as the moving-average embedding for clean samples in that class.

Original: We train the network such that embeddings are pulled closer to their corresponding prototypes, while pushed away from other prototypes.
Revised: We train the network such that embeddings are pulled closer to their corresponding prototypes, while pushed away from other prototypes.

Original: Images with corrupted labels are corrected either as another class or as an OOD sample based on their distance to the momentum prototypes.
Revised: Images with corrupted labels are corrected either as another class or as an OOD sample based on their distance to the momentum prototypes.

Original: We experimentally show that:
Revised: We experimentally show that:

Original: • MoPro achieves state-of-the-art performance on the upstream weakly-supervised learning task.
Revised: • MoPro achieves state-of-the-art performance on the upstream weakly-supervised learning task.

Original: • MoPro substantially improves representation learning performance when the pretrained model is transferred to downstream image classiﬁcation and object detection tasks.
Revised: • MoPro substantially improves representation learning performance when the pretrained model is transferred to downstream image classiﬁcation and object detection tasks.

Original: For the ﬁrst time, we show that weakly-supervised representation learning achieves similar performance as supervised representation learning, under the same data and computation budget.
Revised: For the ﬁrst time, we show that weakly-supervised representation learning achieves similar performance as supervised representation learning, under the same data and computation budget.

Original: With a larger web dataset, MoPro outperforms ImageNet supervised learning by a large margin.
Revised: With a larger web dataset, MoPro outperforms ImageNet supervised learning by a large margin.

Original: • MoPro learns a more robust and calibrated model that generalizes better to distribution variations.
Revised: • MoPro learns a more robust and calibrated model that generalizes better to distribution variations.

Original:  R ELATED WORK2. W EBLY - SUPERVISED REPRESENTATION LEARNING
Revised:  R ELATED WORK

Revised:  W EBLY - SUPERVISED REPRESENTATION LEARNING

Original: A number of prior works exploit large web datasets for visual representation learning (Divvala et al., 2014; Chen & Gupta, 2015; Joulin et al., 2016; Mahajan et al., 2018; Sun et al., 2017; Li et al., 2017;
Revised: A number of prior works exploit large web datasets for visual representation learning (Divvala et al., 2014; Chen & Gupta, 2015; Joulin et al., 2016; Mahajan et al., 2018; Sun et al., 2017; Li et al., 2017;

Original: Kolesnikov et al., 2020).
Revised: Kolesnikov et al., 2020).

Original: These datasets contain a considerable amount of noise.
Revised: These datasets contain a considerable amount of noise.

Original: Approximately 20% of the labels in the JMT-300M dataset (Sun et al., 2017) are noisy, whereas 34% of images in the WebVision dataset (Li et al., 2017) are considered outliers.
Revised: Approximately 20% of the labels in the JMT-300M dataset (Sun et al., 2017) are noisy, whereas 34% of images in the WebVision dataset (Li et al., 2017) are considered outliers.

Original: Surprisingly, most prior works have chosen to ignore the noise and applied vanilla supervised method, with the claim that the scale of data can overpower the noise (Mahajan et al., 2018;
Revised: Surprisingly, most prior works have chosen to ignore the noise and applied vanilla supervised method, with the claim that the scale of data can overpower the noise (Mahajan et al., 2018;

Original: Sun et al., 2017; Li et al., 2017).
Revised: Sun et al., 2017; Li et al., 2017).

Original: However, we show that supervised method cannot fully harvest the power of large-scale weakly-labeled datasets.
Revised: However, we show that supervised method cannot fully harvest the power of large-scale weakly-labeled datasets.

Original: Our method achieves substantial improvement by addressing noise, and advances the potential of webly-supervised representation learning.
Revised: Our method achieves substantial improvement by addressing noise, and advances the potential of webly-supervised representation learning.

Original: [Figure]

Original: 2. L EARNING WITH LABEL NOISE
Revised: 2. L EARNING WITH LABEL NOISE

Original: Learning with label noise has been widely studied.
Revised: Learning with label noise has been widely studied.

Original: Some methods require access to a small set of clean samples (Xiao et al., 2015; Vahdat, 2017; Veit et al., 2017; Lee et al., 2018; Zhang et al., 2020), and other methods assume that no clean labels are available.
Revised: Some methods require access to a small set of clean samples (Xiao et al., 2015; Vahdat, 2017; Veit et al., 2017; Lee et al., 2018; Zhang et al., 2020), and other methods assume that no clean labels are available.

Original: There exist two major types of approaches.
Revised: There exist two major types of approaches.

Original: The ﬁrst type performs label correction using predictions from the network (Reed et al., 2015; Ma et al., 2018; Tanaka et al., 2018; Yi & Wu, 2019;
Revised: The ﬁrst type performs label correction using predictions from the network (Reed et al., 2015; Ma et al., 2018; Tanaka et al., 2018; Yi & Wu, 2019;

Original: Yang et al., 2020).
Revised: Yang et al., 2020).

Original: The second type separates clean samples from corrupted samples, and trains the model on clean samples (Han et al., 2018;
Revised: The second type separates clean samples from corrupted samples, and trains the model on clean samples (Han et al., 2018;

Original: Arazo et al., 2019; Jiang et al., 2018; Wang et al., 2018; Chen et al., 2019; Li et al., 2020a).
Revised: Arazo et al., 2019; Jiang et al., 2018; Wang et al., 2018; Chen et al., 2019; Li et al., 2020a).

Original: However, existing methods have yet shown promising results for large-scale weakly-supervised representation learning.
Revised: However, existing methods have yet shown promising results for large-scale weakly-supervised representation learning.

Original: The main reasons include: (1) most methods do not consider OOD samples, which commonly occur in real-world web datasets; (2) most methods are computational-heavy due to co-training (Han et al., 2018;
Revised: The main reasons include: (1) most methods do not consider OOD samples, which commonly occur in real-world web datasets; (2) most methods are computational-heavy due to co-training (Han et al., 2018;

Original: Li et al., 2020a; Jiang et al., 2018; 2020), iterative training (Tanaka et al., 2018; Yi & Wu, 2019;
Revised: Li et al., 2020a; Jiang et al., 2018; 2020), iterative training (Tanaka et al., 2018; Yi & Wu, 2019;

Original: Wang et al., 2018; Chen et al., 2019), or meta-learning (Li et al., 2019;
Revised: Wang et al., 2018; Chen et al., 2019), or meta-learning (Li et al., 2019;

Original: Zhang et al., 2019).
Revised: Zhang et al., 2019).

Original: Different from existing methods, MoPro achieves both label correction and OOD sample removal on-the-ﬂy with a single step, based on the similarity between an image embedding and the momentum prototypes.
Revised: Different from existing methods, MoPro achieves both label correction and OOD sample removal on-the-ﬂy with a single step, based on the similarity between an image embedding and the momentum prototypes.

Original: MoPro also leverages contrastive learning to learn a robust embedding space.
Revised: MoPro also leverages contrastive learning to learn a robust embedding space.

Original: 2. S ELF - SUPERVISED REPRESENTATION LEARNING
Revised: 2. S ELF - SUPERVISED REPRESENTATION LEARNING

Original: Self-supervised methods have been proposed for representation learning using unlabeled data.
Revised: Self-supervised methods have been proposed for representation learning using unlabeled data.

Original: The recent developments in self-supervised representation learning can be attributed to contrastive learning.
Revised: The recent developments in self-supervised representation learning can be attributed to contrastive learning.

Original: Most methods (He et al., 2019; Chen et al., 2020a; Oord et al., 2018; Wu et al., 2018) leverage the task of instance discrimination, where augmented crops from the same source image are enforced to have similar embeddings.
Revised: Most methods (He et al., 2019; Chen et al., 2020a; Oord et al., 2018; Wu et al., 2018) leverage the task of instance discrimination, where augmented crops from the same source image are enforced to have similar embeddings.

Original: Prototypical contrastive learning (PCL) (Li et al., 2020b) performs clustering to ﬁnd prototypical embeddings, and enforces an image embedding to be similar to its assigned prototypes.
Revised: Prototypical contrastive learning (PCL) (Li et al., 2020b) performs clustering to ﬁnd prototypical embeddings, and enforces an image embedding to be similar to its assigned prototypes.

Original: Different from PCL, we update prototypes on-the-ﬂy in a weakly-supervised setting, where the momentum prototype of a class is the moving average of clean samples’ embeddings.
Revised: Different from PCL, we update prototypes on-the-ﬂy in a weakly-supervised setting, where the momentum prototype of a class is the moving average of clean samples’ embeddings.

Original: Furthermore, we jointly optimize two contrastive losses and a cross-entropy loss.
Revised: Furthermore, we jointly optimize two contrastive losses and a cross-entropy loss.

Original: Current self-supervised representation learning methods are limited in (1) inferior performance in low-shot task adaptation, (2) huge computation cost, and (3) inadequate to harvest larger datasets.
Revised: Current self-supervised representation learning methods are limited in (1) inferior performance in low-shot task adaptation, (2) huge computation cost, and (3) inadequate to harvest larger datasets.

Original: We show that weakly-supervised learning with MoPro addresses these limitations.
Revised: We show that weakly-supervised learning with MoPro addresses these limitations.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we delineate the details of our method.
Revised: In this section, we delineate the details of our method.

Original: First, we introduce the components in our representation learning framework.
Revised: First, we introduce the components in our representation learning framework.

Original: Then, we describe the loss functions.
Revised: Then, we describe the loss functions.

Original: Finally, we explain the noise correction procedure for label correction and OOD sample removal.
Revised: Finally, we explain the noise correction procedure for label correction and OOD sample removal.

Original: A pseudo-code of MoPro is provided in appendix B.
Revised: A pseudo-code of MoPro is provided in appendix B.

Original: 3. R EPRESENTATION L EARNING F RAMEWORK
Revised: 3. R EPRESENTATION L EARNING F RAMEWORK

Original: Our proposed framework consists of the following components.
Revised: Our proposed framework consists of the following components.

Original: Figure 2 gives an illustration.
Revised: Figure 2 gives an illustration.

Original: • A noisy training dataset { ( x i , y i ) } ni =1 , where x i is an image and y i 2 { 1 , ..., K } is its class label.
Revised: • A noisy training dataset { ( x i , y i ) } ni =1 , where x i is an image and y i ∈ { 1 , ..., K } is its class label.

Original: • A pseudo-label
Revised: • A pseudo-label

Original: ˆ
Revised: ˆ

Original: y i for each image x i , which is its corrected label.
Revised: y i for each image x i , which is its corrected label.

Original: Details for generating the pseudo-label is explained in Sec 3.3.
Revised: Details for generating the pseudo-label is explained in Sec 3.3.

Original: • An encoder network, which maps an augmented image ˜ x
Revised: • An encoder network, which maps an augmented image ˜ x

Original: i to a representation vector v i 2 R d e .
Revised: i to a representation vector v i ∈ R d e .

Original: We experiment with ResNet-50 (He et al., 2016) as the encoder, where the activations of the ﬁnal global pooling layer ( d e = 2048 ) are used as the representation vector.
Revised: We experiment with ResNet-50 (He et al., 2016) as the encoder, where the activations of the ﬁnal global pooling layer ( d e = 2048 ) are used as the representation vector.

Original: • A classiﬁer (a fully-connected layer followed by softmax) which receives the representation v i as input and outputs class predictions p i . 
Revised: • A classiﬁer (a fully-connected layer followed by softmax) which receives the representation v i as input and outputs class predictions p i . CNN backbone softmax “Chihuahua” low-dim

Revised: L2 norm 2048-D 128-D ... momentum embeddings ... contrastive momentum prototypes “Chihuahua” “orange” ...

Revised: “baseball” contrastive loss function negatives positives cross-entropy

Original: • A projection network, which maps the representation v i into a low-dimensional embedding z iR d p ( d p = 128 ).
Revised: • A projection network, which maps the representation v i into a low-dimensional embedding z i ∈ R d p ( d p = 128 ).

Original: z i is always normalized to the unit sphere.
Revised: z i is always normalized to the unit sphere.

Original: Following SimCLR (Chen et al., 2020a), we use a MLP with one hidden layer as the projection network.
Revised: Following SimCLR (Chen et al., 2020a), we use a MLP with one hidden layer as the projection network.

Original: • Momentum embeddings z [-
-]
Revised: • Momentum embeddings z [-
-](cid:48)

Original: i generated by a momentum encoder.
Revised: i generated by a momentum encoder.

Original: The momentum encoder has the same architecture as the encoder followed by the projection network, and its parameters are the moving-average of the encoder’s and the projection network’s parameters.
Revised: The momentum encoder has the same architecture as the encoder followed by the projection network, and its parameters are the moving-average of the encoder’s and the projection network’s parameters.

Original: Same as in MoCo (He et al., 2019), we maintain a queue of momentum embeddings of past samples.
Revised: Same as in MoCo (He et al., 2019), we maintain a queue of momentum embeddings of past samples.

Original: • Momentum prototypes C 2 R d p ⇥ K .
Revised: • Momentum prototypes C ∈ R d p × K .

Original: The momentum prototype of the k -th class, c k , is the normalized moving-average embedding for samples with pseudo-label ˆ
Revised: The momentum prototype of the k -th class, c k , is the normalized moving-average embedding for samples with pseudo-label ˆ

Original: y i
Revised: y i

Original: =
Revised: =

Original: k .
Revised: k .

Original: 3. C ONTRASTIVE L OSS
Revised: 3. C ONTRASTIVE L OSS

Original: As illustrated in Figure  we aim to learn an embedding space where samples from the same class gather around its class prototype, while samples from different classes are seperated.
Revised: As illustrated in Figure 1, we aim to learn an embedding space where samples from the same class gather around its class prototype, while samples from different classes are seperated.

Original: We achieve it with two contrastive losses: (1) a prototypical contrastive loss L pro which increases the similarity between an embedding and its corresponding class prototype, ( z i , c ˆ y i ) , in contrast to other prototypes; (2) an instance contrastive loss L ins which increases the similarity between two embeddings of the same source image, ( z i , z 0 i ) , in contrast to embeddings of other images.
Revised: We achieve it with two contrastive losses: (1) a prototypical contrastive loss L pro which increases the similarity between an embedding and its corresponding class prototype, ( z i , c ˆ y i ) , in contrast to other prototypes; (2) an instance contrastive loss L ins which increases the similarity between two embeddings of the same source image, ( z i , z (cid:48) i ) , in contrast to embeddings of other images.

Original: Speciﬁcally, the contrastive losses are deﬁned as:
Revised: Speciﬁcally, the contrastive losses are deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where ⌧ is a temperature parameter, and ˆ y i is the pseudo-label.
Revised: where τ is a temperature parameter, and ˆ y i is the pseudo-label.

Original: We use R negative momentum embeddings to construct the denominator of the instance contrastive loss.
Revised: We use R negative momentum embeddings to construct the denominator of the instance contrastive loss.

Original: We train the classiﬁer with cross-entropy loss, using pseudo-labels as targets.
Revised: We train the classiﬁer with cross-entropy loss, using pseudo-labels as targets.

Original: [Equation]
Revised: [Equation]

Original: We jointly optimize the contrastive losses and the classiﬁcation loss.
Revised: We jointly optimize the contrastive losses and the classiﬁcation loss.

Original: The training objective is:
Revised: The training objective is:

Original: [Equation]
Revised: [Equation]

Original: For simplicity, we set  pro =  ins = 1 for all experiments.
Revised: For simplicity, we set λ pro = λ ins = 1 for all experiments.

Original: 3. N OISE C ORRECTION
Revised: 3. N OISE C ORRECTION

Original: We propose a simple yet effective method for online noise correction during training, which cleans label noise and removes OOD samples.
Revised: We propose a simple yet effective method for online noise correction during training, which cleans label noise and removes OOD samples.

Original: For each sample, we generate a soft pseudo-label q i by combining the classiﬁer’s output probability p i with s i , a class probability distribution calculated using the sample’s similarity w.r.t the momentum prototypes:[Equation] c k / ⌧ ) .
Revised: For each sample, we generate a soft pseudo-label q i by combining the classiﬁer’s output probability p i with s i , a class probability distribution calculated using the sample’s similarity w.r.t the momentum prototypes:

Original: [Equation]
Revised: [Equation]

Original: where the combination weight is simply set as ↵ = 0 .
Revised: where the combination weight is simply set as α = 0 .

Original: 5 in all experiments.
Revised: 5 in all experiments.

Original: We convert q i into a hard pseudo-label ˆ y i based on the following rules: (1) if the highest score of q i is above certain threshold T , use the class with the highest score as the pseudo-label; (2) otherwise, if the score for the original label y i is higher than uniform probability, use y i as the pseudo-label; (3) otherwise, label it as an OOD sample.
Revised: We convert q i into a hard pseudo-label ˆ y i based on the following rules: (1) if the highest score of q i is above certain threshold T , use the class with the highest score as the pseudo-label; (2) otherwise, if the score for the original label y i is higher than uniform probability, use y i as the pseudo-label; (3) otherwise, label it as an OOD sample.

Original: [Equation]
Revised: [Equation]

Original: We remove OOD samples from both the cross-entropy loss and the prototypical contrastive loss so that they do not affect class-speciﬁc learning, but include them in the instance contrastive loss to further separate them from in-distribution samples.
Revised: We remove OOD samples from both the cross-entropy loss and the prototypical contrastive loss so that they do not affect class-speciﬁc learning, but include them in the instance contrastive loss to further separate them from in-distribution samples.

Original: Examples of OOD images and corrected pseudo-labels are shown in the appendices.
Revised: Examples of OOD images and corrected pseudo-labels are shown in the appendices.

Original: 3. M OMENTUM P ROTOTYPES
Revised: 3. M OMENTUM P ROTOTYPES

Original: For each class k , we calculate its momentum prototype as a moving-average of the normalized embeddings for samples with pseudo-label k .
Revised: For each class k , we calculate its momentum prototype as a moving-average of the normalized embeddings for samples with pseudo-label k .

Original: Speciﬁcally, we update c k by:
Revised: Speciﬁcally, we update c k by:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where Normalize( c )
Revised: where Normalize( c )

Original: =
Revised: =

Original: c / k c k 2 .
Revised: c / (cid:107) c (cid:107) 2 .

Original: The momentum coefﬁcient m is set 0.999 in our experiments.
Revised: The momentum coefﬁcient m is set 0.999 in our experiments.

Original:  E XPERIMENTS 4. D ATASET FOR UPSTREAM TRAINING
Revised:  E XPERIMENTS 4. D ATASET FOR UPSTREAM TRAINING

Original: We use the WebVision (Li et al., 2017) dataset as the noisy training data.
Revised: We use the WebVision (Li et al., 2017) dataset as the noisy training data.

Original: It consists of images automatically crawled from Google and Flickr, using visual concepts from ImageNet as queries.
Revised: It consists of images automatically crawled from Google and Flickr, using visual concepts from ImageNet as queries.

Original: We experiment with three versions of WebVision with different sizes: (1) WebVision-V1.0 contains 2.44m images with the same classes as the ImageNet-1k (ILSVRC 2012) dataset; (2) WebVisionV0.5 is a randomly sampled subset of WebVision-V1.0, which contains the same number of images (1.28m) as ImageNet-1k; (3) WebVision-V2.0 contains 16m images with 5k classes.
Revised: We experiment with three versions of WebVision with different sizes: (1) WebVision-V1.0 contains 2.44m images with the same classes as the ImageNet-1k (ILSVRC 2012) dataset; (2) WebVisionV0.5 is a randomly sampled subset of WebVision-V1.0, which contains the same number of images (1.28m) as ImageNet-1k; (3) WebVision-V2.0 contains 16m images with 5k classes.

Original: 4. I MPLEMENTATION DETAILS
Revised: 4. I MPLEMENTATION DETAILS

Original: We follow standard settings for ImageNet training: batch size is 256; total number of epochs is 90; optimizer is SGD with a momentum of 0.9; initial learning rate is 0.1, decayed at 40 andepochs; weight decay is 0.0001.
Revised: We follow standard settings for ImageNet training: batch size is 256; total number of epochs is 90; optimizer is SGD with a momentum of 0.9; initial learning rate is 0.1, decayed at 40 andepochs; weight decay is 0.0001.

Original: We use ResNet-50 (He et al., 2016) as the encoder.
Revised: We use ResNet-50 (He et al., 2016) as the encoder.

Original: For MoProspeciﬁc hyperparameters, we set ⌧ = 0 .
Revised: For MoProspeciﬁc hyperparameters, we set τ = 0 .

Original: 1 , ↵ = 0 .
Revised: 1 , α = 0 .

Original: 5 , T = 0 .
Revised: 5 , T = 0 .

Original: 8 ( T = 0 .
Revised: 8 ( T = 0 .

Original: 6 for WebVision-V2.0).
Revised: 6 for WebVision-V2.0).

Original: The momentum for both the momentum encoder and momentum prototypes is set as 0.999.
Revised: The momentum for both the momentum encoder and momentum prototypes is set as 0.999.

Original: The queue to store momentum embeddings has a size of 8192.
Revised: The queue to store momentum embeddings has a size of 8192.

Original: We apply standard data augmentation (crop and horizontal ﬂip) to the encoder’s input, and stronger data augmentation (color changes in MoCo (He et al., 2019)) to the momentum encoder’s input.
Revised: We apply standard data augmentation (crop and horizontal ﬂip) to the encoder’s input, and stronger data augmentation (color changes in MoCo (He et al., 2019)) to the momentum encoder’s input.

Original: We warm-up the model for 10 epochs by training on all samples with original labels, before applying noise correction.
Revised: We warm-up the model for 10 epochs by training on all samples with original labels, before applying noise correction.

Original: 4. U PSTREAM TASK PERFORMANCE
Revised: 4. U PSTREAM TASK PERFORMANCE

Original: In Table 1, we compare MoPro with existing weakly-supervised learning methods trained on
Revised: In Table 1, we compare MoPro with existing weakly-supervised learning methods trained on

Original: WebVision-V1.0, where MoPro achieves state-of-the-art performance.
Revised: WebVision-V1.0, where MoPro achieves state-of-the-art performance.

Original: Since the training dataset  has imbalanced number of samples per-class, inspired by Kang et al.
Revised: Since the training dataset [Table] has imbalanced number of samples per-class, inspired by Kang et al.

Original: (2020), we perform the following decoupled training steps to re-balance the classiﬁer: (1) pretrain the model with MoPro; (2) perform noise correction on the training data using the pretrained model, following the method in
Revised: (2020), we perform the following decoupled training steps to re-balance the classiﬁer: (1) pretrain the model with MoPro; (2) perform noise correction on the training data using the pretrained model, following the method in

Original: Section 3.3; (3) keep the pretrained encoder ﬁxed and ﬁnetune the classiﬁer on the cleaned dataset, using square-root data sampling (Mahajan et al., 2018) which balances the classes.
Revised: Section 3.3; (3) keep the pretrained encoder ﬁxed and ﬁnetune the classiﬁer on the cleaned dataset, using square-root data sampling (Mahajan et al., 2018) which balances the classes.

Original: We retrain the classiﬁer for 15 epochs, using a learning rate of 0.01 which is decayed at 5 and 10 epochs.
Revised: We retrain the classiﬁer for 15 epochs, using a learning rate of 0.01 which is decayed at 5 and 10 epochs.

Original: Surprisingly, we also ﬁnd that a vanilla cross-entropy method with decoupled classiﬁer re-balancing can also achieve competitive performance, outperforming most existing baselines.[Table]
Revised: Surprisingly, we also ﬁnd that a vanilla cross-entropy method with decoupled classiﬁer re-balancing can also achieve competitive performance, outperforming most existing baselines.

Original:  T RANSFER LEARNING
Revised:  T RANSFER LEARNING

Original: In this section [Table] we transfer weakly-supervised learned models to a variety of downstream tasks.
Revised: In this section, we transfer weakly-supervised learned models to a variety of downstream tasks.

Original: We show that MoPro yields superior performance in image classiﬁcation, object detection, instance segmentation, and obtains better robustness to domain shifts.
Revised: We show that MoPro yields superior performance in image classiﬁcation, object detection, instance segmentation, and obtains better robustness to domain shifts.

Original: Implementation details for the transfer learning experiments are described in appendix C.
Revised: Implementation details for the transfer learning experiments are described in appendix C.

Original: 5. L OW - SHOT IMAGE CLASSIFICATION ON FIXED REPRESENTATION
Revised: 5. L OW - SHOT IMAGE CLASSIFICATION ON FIXED REPRESENTATION

Original: First, we transfer the learned representation to downstream tasks with few training samples.
Revised: First, we transfer the learned representation to downstream tasks with few training samples.

Original: We perform low-shot classiﬁcation on two datasets: PASCAL VOC2007 (Everingham et al., 2010) for object classiﬁcation and Places205 (Zhou et al., 2014) for scene recognition.
Revised: We perform low-shot classiﬁcation on two datasets: PASCAL VOC2007 (Everingham et al., 2010) for object classiﬁcation and Places205 (Zhou et al., 2014) for scene recognition.

Original: Following the setup by Goyal et al.
Revised: Following the setup by Goyal et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: (2020b), we train linear SVMs using ﬁxed representations from pretrained models.
Revised: (2020b), we train linear SVMs using ﬁxed representations from pretrained models.

Original: We vary the number k of samples per-class and report the average result  across 5 independent runs.
Revised: We vary the number k of samples per-class and report the average result [Table] across 5 independent runs.

Original: Table 2 shows the results.
Revised: Table 2 shows the results.

Original: When pretrained on weakly-labeled datasets,
Revised: When pretrained on weakly-labeled datasets,

Original: MoPro consistently outperforms the vanilla CE method.
Revised: MoPro consistently outperforms the vanilla CE method.

Original: The improvement of MoPro becomes less signiﬁcant when the number of web images increases from 2.4m to 16m, suggesting that increasing dataset size is a viable solution to combat noise.
Revised: The improvement of MoPro becomes less signiﬁcant when the number of web images increases from 2.4m to 16m, suggesting that increasing dataset size is a viable solution to combat noise.

Original: When compared with ImageNet pretrained models, MoPro substantially outperforms self-supervised learning (MoCo v2 (Chen et al., 2020b) and PCL v2 (Li et al., 2020b)), and achieves comparable performance with supervised learning when the same amount of web images ( i .
Revised: When compared with ImageNet pretrained models, MoPro substantially outperforms self-supervised learning (MoCo v2 (Chen et al., 2020b) and PCL v2 (Li et al., 2020b)), and achieves comparable performance with supervised learning when the same amount of web images ( i .

Original: e .
Revised: e .

Original: WebVision-V0.5) is used.
Revised: WebVision-V0.5) is used.

Original: Our results for the ﬁrst time show that weakly-supervised representation learning can be as powerful as supervised representation learning under the same data and computation budget.[Table] = [Table]
Revised: Our results for the ﬁrst time show that weakly-supervised representation learning can be as powerful as supervised representation learning under the same data and computation budget.

Original: =

Original: [Table]

Original: k =

Original: [Table]

Original: k =

Original: [Table]

Original: k =

Original: [Table]

Original: k =

Original: [Table]

Original: 5. L OW - RESOURCE TRANSFER WITH FINETUNING
Revised: 5. L OW - RESOURCE TRANSFER WITH FINETUNING

Original: Next [Table] we perform experiment to evaluate whether the pretrained model provides a good basis for ﬁnetuning when the downstream task has limited training data.
Revised: Next, we perform experiment to evaluate whether the pretrained model provides a good basis for ﬁnetuning when the downstream task has limited training data.

Original: Following the setup by Chen et al.
Revised: Following the setup by Chen et al.

Original: (2020a) [Table] we ﬁnetune the pretrained model on 1% or 10% of ImageNet training samples.
Revised: (2020a), we ﬁnetune the pretrained model on 1% or 10% of ImageNet training samples.

Original: Tableshows the results.
Revised: Tableshows the results.

Original: MoPro consistently outperforms CE when pretrained on Web datasets.
Revised: MoPro consistently outperforms CE when pretrained on Web datasets.

Original: Compared to self-supervised learning methods pretrained on ImageNet, weakly-supervised learning achieves signiﬁcantly better performance with fewer number of epochs.
Revised: Compared to self-supervised learning methods pretrained on ImageNet, weakly-supervised learning achieves signiﬁcantly better performance with fewer number of epochs.

Original: Surprisingly, pretraining on the larger WebVision-V2 leads to worse performance compared to V0.
Revised: Surprisingly, pretraining on the larger WebVision-V2 leads to worse performance compared to V0.

Original: and V1.0.
Revised: and V1.0.

Original: This is because WebVision-V0.5 and V1.0 contain the same 1k class as ImageNet, whereas V2 also contains 4k extra classes.
Revised: This is because WebVision-V0.5 and V1.0 contain the same 1k class as ImageNet, whereas V2 also contains 4k extra classes.

Original: Hence, the representations learned from V2 are less task-speciﬁc and more difﬁcult to adapt to ImageNet, especially with only 1% of samples for ﬁnetuning.
Revised: Hence, the representations learned from V2 are less task-speciﬁc and more difﬁcult to adapt to ImageNet, especially with only 1% of samples for ﬁnetuning.

Original: This suggests that if the classes for a downstream task are known a priori [Table] it is more effective to curate a task-speciﬁc weakly-labeled dataset with the same classes.
Revised: This suggests that if the classes for a downstream task are known a priori, it is more effective to curate a task-speciﬁc weakly-labeled dataset with the same classes.

Original: [Table]
Revised: [Table]

Original: 5. O BJECT DETECTION AND INSTANCE SEGMENTATION
Revised: 5. O BJECT DETECTION AND INSTANCE SEGMENTATION

Original: We further transfer the pretrained model to object detection and instance segmentation tasks on
Revised: We further transfer the pretrained model to object detection and instance segmentation tasks on

Original: COCO (Lin et al., 2014).
Revised: COCO (Lin et al., 2014).

Original: Following the setup by He et al.
Revised: Following the setup by He et al.

Original: (2019), we use the pretrained ResNetas the backbone for a Mask-RCNN (He et al., 2017) with FPN (Lin et al., 2017).
Revised: (2019), we use the pretrained ResNetas the backbone for a Mask-RCNN (He et al., 2017) with FPN (Lin et al., 2017).

Original: We ﬁnetune all layers end-to-end, including BN.
Revised: We ﬁnetune all layers end-to-end, including BN.

Original: The schedule is the default 1 ⇥ or 2 ⇥ in Girshick et al.
Revised: The schedule is the default 1 × or 2 × in Girshick et al.

Original: Table 4 shows the results.
Revised: Table 4 shows the results.

Original: Weakly-supervised learning with MoPro outperforms both supervised learning on ImageNet and self-supervised learning on one billion Instagram images.
Revised: Weakly-supervised learning with MoPro outperforms both supervised learning on ImageNet and self-supervised learning on one billion Instagram images.

Original: 5. R OBUSTNESS
Revised: 5. R OBUSTNESS

Original: It has been shown that deep models trained on ImageNet lack robustness to out-of-distribution samples, often falsely producing over-conﬁdent predictions.
Revised: It has been shown that deep models trained on ImageNet lack robustness to out-of-distribution samples, often falsely producing over-conﬁdent predictions.

Original: Hendricks et al .
Revised: Hendricks et al .

Original: have curated two benchmark datasets to evaluate models’ robustness to real-world distribution variation: (1) ImageNetR (Hendrycks et al., 2020) which contains various artistic renditions of object classes from the original ImageNet dataset, and
Revised: have curated two benchmark datasets to evaluate models’ robustness to real-world distribution variation: (1) ImageNetR (Hendrycks et al., 2020) which contains various artistic renditions of object classes from the [Table]

Original: (2) ImageNet-A
Revised: [Table]

Original:  (Hendrycks et al., 2019) which contains natural images where ImageNet-pretrained models consistently fail due to variations in background elements, color, or texture.
Revised: original ImageNet dataset, and (2) ImageNet-A (Hendrycks et al., 2019) which contains natural images where ImageNet-pretrained models consistently fail due to variations in background elements, color [Table]

Original:  Both datasets contain 200 classes, a subset of ImageNet’s 1,000 classes.
Revised: or texture. Both datasets contain 200 classes, a subset of ImageNet’s 1,000 classes.

Original: We evaluate weakly-supervised trained models on these two robustness benchmarks.
Revised: We evaluate weakly-supervised trained models on these two robustness benchmarks.

Original: We report both accuracy and the ` 2 calibration error (Kumar et al., 2019).
Revised: We report both accuracy and the (cid:96) 2 calibration error (Kumar et al., 2019).

Original: The calibration error measures the misalignment between a model’s conﬁdence and its accuracy.
Revised: The calibration error measures the misalignment between a model’s conﬁdence and its accuracy.

Original: Concretely, a well-calibrated classiﬁer which give examples 80% conﬁdence should be correct 80% of the time.
Revised: Concretely, a well-calibrated classiﬁer which give examples 80% conﬁdence should be correct 80% of the time.

Original: Results are shown in
Revised: Results are shown in

Original: Table 5.
Revised: Table 5.

Original: Webly-supervised learning show signiﬁcantly higher accuracy and lower calibration error.
Revised: Webly-supervised learning show signiﬁcantly higher accuracy and lower calibration error.

Original: The robustness to distribution shift could come from the higher diversity of samples in Web images.
Revised: The robustness to distribution shift could come from the higher diversity of samples in Web images.

Original: Compared to vanilla CE, MoPro further improves the model’s robustness on both datasets.
Revised: Compared to vanilla CE, MoPro further improves the model’s robustness on both datasets.

Original: Note that we made sure that the training data of WebVision does not overlap with the test data.[Table] [Table] [Table]
Revised: Note that we made sure that the training data of WebVision does not overlap with the test data.

Original:  A BLATION S TUDY
Revised:  A BLATION S TUDY

Original: We perform ablation study to verify the effectiveness of three important components in MoPro: (1) prototypical contrastive loss L pro , (2) instance contrastive loss L ins , (3) prototypical similarity s i used for noise correction (equation 4).
Revised: We perform ablation study to verify the effectiveness of three important components in MoPro: (1) prototypical contrastive loss L pro , (2) instance contrastive loss L ins , (3) prototypical similarity s i used for noise correction (equation 4).

Original: We choose low-resource ﬁnetuning on 1% of ImageNet training data as the benchmark, and report the top-1 accuracy for models pretrained on WebVisionV0.5.
Revised: We choose low-resource ﬁnetuning on 1% of ImageNet training data as the benchmark, and report the top-1 accuracy for models pretrained on WebVisionV0.5.

Original: As shown in Table 6, all of the three components contribute to the efﬁcacy of MoPro.
Revised: As shown in Table 6, all of the three components contribute to the efﬁcacy of MoPro.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper introduces a new contrastive learning framework for webly-supervised representation learning.
Revised: This paper introduces a new contrastive learning framework for webly-supervised representation learning.

Original: We propose momentum prototypes, a simple component that is effective in label noise  correction, OOD sample removal, and representation learning.
Revised: We propose momentum prototypes, a simple component that is effective in label noise [Table] correction, OOD sample removal, and representation learning.

Original: MoPro achieves state-of-the-art performance on the upstream task of learning from real-world noisy data, and superior representation learning performance on multiple down-stream tasks.
Revised: MoPro achieves state-of-the-art performance on the upstream task of learning from real-world noisy data, and superior representation learning performance on multiple down-stream tasks.

Original: Webly-supervised learning with MoPro does not require the expensive annotation cost in supervised learning, nor the huge computation budget in self-supervised learning.
Revised: Webly-supervised learning with MoPro does not require the expensive annotation cost in supervised learning, nor the huge computation budget in self-supervised learning.

Original: For future work, MoPro could be extended to utilize other sources of free Web data, such as weakly-labeled videos, for representation learning in other domains.
Revised: For future work, MoPro could be extended to utilize other sources of free Web data, such as weakly-labeled videos, for representation learning in other domains A PPENDIX A N OISY SAMPLE VISUALIZATION

Revised: In Figure 3, we show example images randomly chosen from the out-of-distribution samples ﬁltered out by our method.

Revised: In Figure 4, we show random examples where their pseudo-labels are different from the original training labels.

Revised: By visual examination, we observe that our method can remove

Revised: OOD samples and correct noisy labels at a high success rate.

Revised: A PPENDIX B P SEUDO - CODE OF M O P RO

Revised: Algorithm 1 summarizes the proposed method.

Revised: Algorithm

Revised: 1 Input: number of classes K , temperature τ , threshold T , momentum m , encoder network f ( · ) , projection network g ( · ) , classiﬁer h ( · ) , momentum encoder g (cid:48) ( f (cid:48) ( · )) .

Revised: 2 for { ( x i , y i ) } bi =1 in loader do // load a minibatch of noisy training data for i ∈ { 1 , ..., b } do ˜ x

Revised: i = weak aug( x i ) //

Revised: weak augmentation ˜ x (cid:48)

Revised: i =

Revised: strong aug( x i ) //

Revised: strong augmentation v i =

Revised: f (˜ x i ) //

Revised: representation z

Revised: i = g ( v i ) // normalized low-dimensional embedding z

Revised: i

Revised: =

Revised: g (cid:48)

Revised: ( f (cid:48) (˜ x (cid:48) i ))

Revised: momentum embedding p i

Revised: = h ( v i ) //

Revised: class prediction s i

Revised: = { s ki } Kk =1 , s ki = exp

Revised: ( z i · c k /τ )

Revised: (cid:80)

Revised: Kk =1 exp( z i · c k /τ ) //

Revised: prototypical score // noise correction q i

Revised: = ( p i + s i ) ///

Revised: soft pseudo-label

Revised: if max k q ki > T then ˆ y i = arg max k q ki else if q y i i > 1 /K

Revised: then ˆ y i

Revised: =

Revised: y i else

Revised: ˆ y i

Revised: =

Revised: OOD end // calculate losses

Revised: L i ins =

Revised: − log exp( z i · z (cid:48)

Revised: i /τ )

Revised: (cid:80)

Revised: Rr =0 exp( z i · z (cid:48) r /τ ) //

Revised: instance contrastive loss if ˆ y i is not OOD then

Revised: L i pro = − log exp( z i · c ˆ yi /τ ) (cid:80) Kk

Revised: =1 exp( z i · c k /τ ) //

Revised: prototypical contrastive loss

Revised: L i ce

Revised: = − log( p ˆ y i i ) //

Revised: cross entropy loss else

Revised: L i pro =

Revised: L i ce

Revised: = end // update momentum prototypes

Revised: c ˆ y i ← Normalize( m c ˆ

Revised: y i + (1 − m )

Revised: z i ) end

Original: {+L = (cid:80) b i =
+}
Revised: {+L = (cid:80) b i =
+}

Revised: ( L i ce + L i pro + L i ins ) //

Revised: total loss update networks f, g, h to minimize L .

Revised: 30 end

Revised: A PPENDIX C T RANSFER LEARNING IMPLEMENTATION DETAILS

Revised: For low-shot image classiﬁcation on Places and VOC, we follow the procedure in Li et al.

Revised: (2020b)

Revised: and train linear SVMs on the global average pooling features of ResNet-50.

Revised: We preprocess all images by resizing to 256 pixels along the shorter side and taking a 224 × 224 center crop.

Revised: The

Revised: SVMs are implemented in the LIBLINEAR (Fan et al., 2008) package.

Revised: For low-resource ﬁnetuning on ImageNet, we adopt different ﬁnetuning strategy for different versions of WebVision pretrained models.

Revised: For WebVision V0.5 and V1.0, since they contain the same 1000 classes as ImageNet, we ﬁnetune the entire model including the classiﬁcation layer.

Revised: We train with SGD, using a batch size of 256, a momentum of 0.9, a weight decay of 0, and a learning rate of 0.005.

Revised: We train for 40 epochs, and drop the learning rate by 0.2 at 15 and 30 epochs.

Revised: For WebVision 2.0, since it contains 5000 classes, we randomly initialize a new classiﬁcation layer with 1000 output dimension, and ﬁnetune the model end-to-end.

Revised: We train for 50 epochs, using a learning rate of 0.01, which is dropped by 0.1 at 20 and 40 epochs

Revised: [Table]

Revised: For object detection and instance segmentation on COCO, we adopt the same setup in MoCo (He et al., 2019), using Detectron2 (Girshick et al., 2018) codebase.

Revised: The image scale is in [640, 800] pixels during training and is 800 at inference.

Revised: We ﬁne-tune all layers end-to-end.

Revised: We ﬁnetune on the train2017 set ( ∼ 118k images) and evaluate on val2017 .

Revised: A PPENDIX D S TANDARD DEVIATION FOR LOW - SHOT CLASSIFICATION

Revised: Table 7 reports the standard deviation for the low-shot image classiﬁcation experiment in Section 5.

Revised: [Table]