Original: N OISE -R OBUST C ONTRASTIVE L EARNING 
Revised: L EARNING FROM NOISY DATA WITH ROBUST REPRESENTATION LEARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning from noisy data has attracted much attention, where most methods focus on label noise.
Revised: Learning from noisy data has attracted much attention, where most methods focus on label noise.

Original: In this work, we propose a new framework which simultaneously addresses three types of noise commonly seen in real-world data: label noise, outof-distribution input, and input corruption.
Revised: In this work, we propose a new framework which simultaneously addresses three types of noise commonly seen in real-world data: label noise, outof-distribution input, and input corruption.

Original: In contrast to most existing methods, we combat noise by learning robust representation.
Revised: In contrast to most existing methods, we combat noise by learning robust representation.

Original: Speciﬁcally, we embed images into a low-dimensional subspace by training an autoencoder on the deep features.
Revised: Speciﬁcally, we embed images into a low-dimensional subspace by training an autoencoder on the deep features.

Original: We regularize the geometric structure of the subspace with robust contrastive learning, which includes an unsupervised consistency loss and a supervised mixup prototypical loss.
Revised: We regularize the geometric structure of the subspace with robust contrastive learning, which includes an unsupervised consistency loss and a supervised mixup prototypical loss.

Original: Furthermore, we leverage the structure of the learned subspace for noise cleaning, by aggregating information from neighboring samples.
Revised: Furthermore, we leverage the structure of the learned subspace for noise cleaning, by aggregating information from neighboring samples.

Original: Experiments on multiple benchmarks demonstrate state-of-the-art performance of our method and robustness of the learned representation.
Revised: Experiments on multiple benchmarks demonstrate state-of-the-art performance of our method and robustness of the learned representation.

Original: Our code will be released 1 .
Revised: Our code will be released 1 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Data in real life is noisy .
Revised: Data in real life is noisy .

Original: However, deep models with remarkable performance are mostly trained on clean datasets with high-quality human annotations.
Revised: However, deep models with remarkable performance are mostly trained on clean datasets with high-quality human annotations.

Original: Manual data cleaning and labeling is an expensive process that is difﬁcult to scale.
Revised: Manual data cleaning and labeling is an expensive process that is difﬁcult to scale.

Original: On the other hand, there exists almost inﬁnite amount of noisy data online.
Revised: On the other hand, there exists almost inﬁnite amount of noisy data online.

Original: It is crucial that deep neural networks (DNNs) could harvest noisy training data.
Revised: It is crucial that deep neural networks (DNNs) could harvest noisy training data.

Original: However, it has been shown that DNNs are susceptible to overﬁtting to noise (Zhang et al., 2017).
Revised: However, it has been shown that DNNs are susceptible to overﬁtting to noise (Zhang et al., 2017).

Original: As shown in Figure 1, a real-world noisy image dataset often consists of multiple types of noise.
Revised: As shown in Figure 1, a real-world noisy image dataset often consists of multiple types of noise.

Original: Label noise refers to samples that are wrongly labeled as another class ( e .
Revised: Label noise refers to samples that are wrongly labeled as another class ( e .

Original: g .
Revised: g .

Original: ﬂower labeled as orange).
Revised: ﬂower labeled as orange).

Original: Out-of-distribution input refers to samples that do not belong to any known classes.
Revised: Out-of-distribution input refers to samples that do not belong to any known classes.

Original: Input corruption refers to image-level distortion ( e .
Revised: Input corruption refers to image-level distortion ( e .

Original: g .
Revised: g .

Original: low brightness) that causes data shift between training and test.
Revised: low brightness) that causes data shift between training and test.

Original: Most of the methods in literature focus on addressing the more detrimental label noise.
Revised: Most of the methods in literature focus on addressing the more detrimental label noise.

Original: Two dominant approaches include: (1) ﬁnd clean samples as those with smaller loss and assign larger weights to them (Han et al., 2018;
Revised: Two dominant approaches include: (1) ﬁnd clean samples as those with smaller loss and assign larger weights to them (Han et al., 2018;

Original: Yu et al., 2019; Shen & Sanghavi, 2019; Arazo et al., 2019); (2) relabel noisy samples using model’s predictions (Reed et al., 2015; Ma et al., 2018; Tanaka et al., 2018; Yi & Wu, 2019).
Revised: Yu et al., 2019; Shen & Sanghavi, 2019; Arazo et al., 2019); (2) relabel noisy samples using model’s predictions (Reed et al., 2015; Ma et al., 2018; Tanaka et al., 2018; Yi & Wu, 2019).

Original: The recently proposed DivideMix (Li et al., 2020a) integrates both approaches in a co-training framework, but it also increases computation cost.
Revised: The recently proposed DivideMix (Li et al., 2020a) integrates both approaches in a co-training framework, but it also increases computation cost.

Original: Previous methods that focus on addressing label noise do not consider out-of-distribution input or input corruption, which limits their performance in real-world scenarios.
Revised: Previous methods that focus on addressing label noise do not consider out-of-distribution input or input corruption, which limits their performance in real-world scenarios.

Original: Furthermore, using a model’s own prediction to relabel samples could cause conﬁrmation bias, where the prediction error accumulates and harms performance.
Revised: Furthermore, using a model’s own prediction to relabel samples could cause conﬁrmation bias, where the prediction error accumulates and harms performance.

Original: We propose a new direction for effective learning from noisy data.
Revised: We propose a new direction for effective learning from noisy data.

Original: Our method embeds images into noise-robust low-dimensional representations, and regularizes the geometric structure of the representations with contrastive learning.
Revised: Our method embeds images into noise-robust low-dimensional representations, and regularizes the geometric structure of the representations with contrastive learning.

Original: Speciﬁcally, our algorithmic contributions include:
Revised: Speciﬁcally, our algorithmic contributions include:

Original: • We propose noise-robust contrastive learning, which introduces two contrastive losses.
Revised: • We propose noise-robust contrastive learning, which introduces two contrastive losses.

Original: The ﬁrst is an unsupervised consistency contrastive loss.
Revised: The ﬁrst is an unsupervised consistency contrastive loss.

Original: It enforces inputs with perturbations to have similar normalized embeddings, which helps learn robust and discriminative representation.
Revised: It enforces inputs with perturbations to have similar normalized embeddings, which helps learn robust and discriminative representation.

Original: • Our second contrastive loss is a supervised mixup prototypical loss.
Revised: • Our second contrastive loss is a weakly-supervised mixup prototypical loss.

Original: We compute class prototypes as normalized mean embeddings, and enforces each sample’s embedding to be closer to  its class prototype.
Revised: We compute class prototypes as normalized mean embeddings, and enforces each sample’s embedding to be closer to [Figure] its class prototype.

Original: Inspired by Mixup (Zhang et al., 2018), we construct virtual training samples as linear interpolation of inputs, and encourage the same linear relationship w.r.t the class prototypes.[Figure]
Revised: Inspired by Mixup (Zhang et al., 2018), we construct virtual training samples as linear interpolation of inputs, and encourage the same linear relationship w.r.t the class prototypes.

Original: • We train a linear autoencoder to reconstruct the high-dimensional features using low-dimensional embeddings.
Revised: • We train a linear autoencoder to reconstruct the high-dimensional features using low-dimensional embeddings.

Original: The autoendoer enables the high-dimensional features to maximally preserve the robustness of the low-dimensional embeddings, thus regularizing the classiﬁer.
Revised: The autoendoer enables the high-dimensional features to maximally preserve the robustness of the low-dimensional embeddings, thus regularizing the classiﬁer.

Original: • We propose a new noise cleaning method which exploits the structure of the learned representations.
Revised: • We propose a new noise cleaning method which exploits the structure of the learned representations.

Original: For each sample, we aggregate information from its top- k neighbors to create a pseudo-label.
Revised: For each sample, we aggregate information from its top- k neighbors to create a pseudo-label.

Original: A subset of training samples with conﬁdent pseudo-labels are selected to compute the supervised losses.
Revised: A subset of training samples with conﬁdent pseudo-labels are selected to compute the weaklysupervised losses.

Original: This process can effectively clean both label noise and out-of-distribution (OOD) noise.
Revised: This process can effectively clean both label noise and out-of-distribution (OOD) noise.

Original: Our experimental contributions include
Revised: Our experimental contributions include:

Original: : • We experimentally show that our method is robust to label noise, OOD input, and input corruption.
Revised: • We experimentally show that our method is robust to label noise, OOD input, and input corruption.

Original: Experiments are performed on multiple datasets with controlled noise and real-world noise, where our method achieves state-of-the-art performance.
Revised: Experiments are performed on multiple datasets with controlled noise and real-world noise, where our method achieves state-of-the-art performance.

Original: • We demonstrate that the proposed noise cleaning method can effectively clean a majority of label noise.
Revised: • We demonstrate that the proposed noise cleaning method can effectively clean a majority of label noise.

Original: It also learns a curriculum that gradually leverages more samples to compute the supervised losses as the pseudo-labels become more accurate.
Revised: It also learns a curriculum that gradually leverages more samples to compute the weakly-supervised losses as the pseudo-labels become more accurate.

Original: • We validate the robustness of the learned low-dimensional representation by showing (1) k -nearest neighbor classiﬁcation outperforms the softmax classiﬁer.
Revised: • We validate the robustness of the learned low-dimensional representation by showing (1) k -nearest neighbor classiﬁcation outperforms the softmax classiﬁer.

Original: (2) OOD samples can be separated from in-distribution samples.
Revised: (2) OOD samples can be separated from in-distribution samples.

Original: The efﬁcacy of the proposed autoencoder is also veriﬁed.
Revised: The efﬁcacy of the proposed autoencoder is also veriﬁed.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Label noise learning.
Revised: Label noise learning.

Original: Learning from noisy labels have been extensively studied in the literature.
Revised: Learning from noisy labels have been extensively studied in the literature.

Original: While some methods require access to a small set of clean samples (Xiao et al., 2015; Vahdat, 2017; Veit et al., 2017; Lee et al., 2018; Hendrycks et al., 2018), most methods focus on the more challenging scenario where no clean labels are available.
Revised: While some methods require access to a small set of clean samples (Xiao et al., 2015; Vahdat, 2017; Veit et al., 2017; Lee et al., 2018; Hendrycks et al., 2018), most methods focus on the more challenging scenario where no clean labels are available.

Original: These methods can be categorized into two major types.
Revised: These methods can be categorized into two major types.

Original: The ﬁrst type performs label correction using predictions from the network (Reed et al., 2015; Ma et al., 2018; Tanaka et al., 2018; Yi & Wu, 2019).
Revised: The ﬁrst type performs label correction using predictions from the network (Reed et al., 2015; Ma et al., 2018; Tanaka et al., 2018; Yi & Wu, 2019).

Original: The second type tries to separate clean samples from corrupted samples, and trains the model on clean samples (Han et al., 2018; Arazo et al., 2019; Jiang et al., 2018; 2020;
Revised: The second type tries to separate clean samples from corrupted samples, and trains the model on clean samples (Han et al., 2018; Arazo et al., 2019; Jiang et al., 2018; 2020;

Original: Wang et al., 2018; Chen et al., 2019; Lyu & Tsang, 2020).
Revised: Wang et al., 2018; Chen et al., 2019; Lyu & Tsang, 2020).

Original: The recently proposed DivideMix (Li et al., 2020a) effectively combines label correction and sample selection with the Mixup (Zhang et al., 2018) data augmentation under a co-training framework.
Revised: The recently proposed DivideMix (Li et al., 2020a) effectively combines label correction and sample selection with the Mixup (Zhang et al., 2018) data augmentation under a co-training framework.

Original: However, it cost 2 ⇥ the computational resource of our method.
Revised: However, it cost 2 ⇥ the computational resource of our method.

Original: Different from existing methods, our method combats noise by learning noise-robust low-dimensional representations.
Revised: Different from existing methods, our method combats noise by learning noise-robust low-dimensional representations.

Original: We propose a more effective noise cleaning method by leveraging the structure of the learned representations.
Revised: We propose a more effective noise cleaning method by leveraging the structure of the learned representations.

Original: Furthermore, our model is robust not only to label noise, but also to out-of-distribution and corrupted input.
Revised: Furthermore, our model is robust not only to label noise, but also to out-of-distribution and corrupted input.

Original: A previous work has studied open-set noisy labels (Wang et al., 2018), but their method does not enjoy the same level of robustness as ours.
Revised: A previous work has studied open-set noisy labels (Wang et al., 2018), but their method does not enjoy the same level of robustness as ours.

Original: Contrastive learning.
Revised: Contrastive learning.

Original: Contrastive learning is at the core of recent self-supervised representation learning methods (Chen et al., 2020;
Revised: Contrastive learning is at the core of recent self-supervised representation learning methods (Chen et al., 2020;

Original: He et al., 2019; Oord et al., 2018; Wu et al., 2018).
Revised: He et al., 2019; Oord et al., 2018; Wu et al., 2018).

Original: In selfsupervised contrastive learning, two randomly augmented images are generated for each input image.
Revised: In selfsupervised contrastive learning, two randomly augmented images are generated for each input image.

Original: Then a contrastive loss is applied to pull embeddings from the same source image closer, while pushing embeddings from different source images apart.
Revised: Then a contrastive loss is applied to pull embeddings from the same source image closer, while pushing embeddings from different source images apart.

Original: Recently, prototypical contrastive learning (PCL) (Li et al., 2020b) has been proposed, which uses cluster centroids as prototypes, and trains the network by pulling an image embedding closer to its assigned prototypes.
Revised: Recently, prototypical contrastive learning (PCL) (Li et al., 2020b) has been proposed, which uses cluster centroids as prototypes, and trains the network by pulling an image embedding closer to its assigned prototypes.

Original: Different from previous methods, our method performs contrastive learning in the principal subspace of the high-dimensional feature space, by training a linear autoencoder.

Original: Furthermore, our supervised contrastive loss improves PCL (Li et al., 2020b) with Mixup (Zhang et al., 2018).

Original: Different from the

Original: [Figure]

Original: ℒ

Original: [Figure]
Revised: [Figure]

Original: InterpolatedInput
Revised: InterpolatedInput

Original: [Figure]
Revised: [Figure]

Revised: Different from previous methods, our method performs contrastive learning in the principal subspace of the high-dimensional feature space, by training a linear autoencoder.

Revised: Furthermore, our supervised contrastive loss improves PCL (Li et al., 2020b) with Mixup (Zhang et al., 2018).

Original:  original Mixup where learning happens at the classiﬁcation layer, our learning takes places in the low-dimensional subspace.
Revised: Different from the original Mixup where learning happens at the classiﬁcation layer, our learning takes places in the low-dimensional subspace.

Original:  M ETHOD
Revised:  M ETHOD

Original: Given a noisy training dataset D = { ( x i , y i ) } ni =1 , where x i is an image and y i 2 { 1 , ..., C } is its class label.
Revised: Given a noisy training dataset D = { ( x i , y i ) } ni =1 , where x i is an image and y i 2 { 1 , ..., C } is its class label.

Original: We aim to train a network that is robust to the noise in training data ( i .
Revised: We aim to train a network that is robust to the noise in training data ( i .

Original: e . label noise, OOD input, input corruption) and achieves high accuracy on a clean test set.
Revised: e . label noise, OOD input, input corruption) and achieves high accuracy on a clean test set.

Original: The proposed network consists of three components: (1) a deep encoder (a convolutional neural network) that encodes an image x
Revised: The proposed network consists of three components: (1) a deep encoder (a convolutional neural network) that encodes an image x

Original: i to a high-dimensional feature v i ; (2) a classiﬁer (a fully-connected layer followed by softmax) that receives v i as input and outputs class predictions; (3) a linear autoencoder that projects v i into a low-dimensional embedding z i 2 R d .
Revised: i to a high-dimensional feature v i ; (2) a classiﬁer (a fully-connected layer followed by softmax) that receives v i as input and outputs class predictions; (3) a linear autoencoder that projects v i into a low-dimensional embedding z i 2 R d .

Original: We show an illustration of our method in Figure 2, and a pseudo-code in appendix B. Next, we delineate its details.
Revised: We show an illustration of our method in Figure 2, and a pseudo-code in appendix B. Next, we delineate its details.

Original: 3. C ONTRASTIVE LEARNING IN ROBUST LOW - DIMENSIONAL SUBSPACE
Revised: 3. C ONTRASTIVE LEARNING IN ROBUST LOW - DIMENSIONAL SUBSPACE

Original: Let z i
Revised: Let z i

Original: = W e v i be the linear projection from high-dimensional features to low-dimensional embeddings, and ˆ z i
Revised: = W e v i be the linear projection from high-dimensional features to low-dimensional embeddings, and ˆ z i

Original: =
Revised: =

Original: z i / k
Revised: z i / k

Original: z i k 2 be the normalized embeddings.
Revised: z i k 2 be the normalized embeddings.

Original: We aim to learn robust embeddings with two contrastive losses: unsupervised consistency loss and supervised mixup prototypical loss.
Revised: We aim to learn robust embeddings with two contrastive losses: unsupervised consistency loss and weakly-supervised mixup prototypical loss.

Original: Unsupervised consistency contrastive loss .
Revised: Unsupervised consistency contrastive loss .

Original: Following the NT-Xent (Chen et al., 2020) loss for selfsupervised representation learning, our consistency contrastive loss enforces images with semanticpreserving perturbations to have similar embeddings.
Revised: Following the NT-Xent (Chen et al., 2020) loss for selfsupervised representation learning, our consistency contrastive loss enforces images with semanticpreserving perturbations to have similar embeddings.

Original: Speciﬁcally, given a minibatch of b images, we apply weak-augmentation and strong-augmentation to each image, and obtain 2 b inputs { x i } 2 bi =1 .
Revised: Speciﬁcally, given a minibatch of b images, we apply weak-augmentation and strong-augmentation to each image, and obtain 2 b inputs { x i } 2 bi =1 .

Original: Weak augmentation is a standard ﬂip-and-shift augmentation strategy, while strong augmentation consists of color and brightness changes with details given in Section 4.1.
Revised: Weak augmentation is a standard ﬂip-and-shift augmentation strategy, while strong augmentation consists of color and brightness changes with details given in Section 4.1.

Original: We project the inputs into the low-dimensional space to obtain their normalized embeddings { ˆ z i } 2 bi =1 .
Revised: We project the inputs into the low-dimensional space to obtain their normalized embeddings { ˆ z i } 2 bi =1 .

Original: Let i 2 { 1 , ..., b } be the index of a weakly-augmented input, and j ( i ) be the index of the strongaugmented input from the same source image, the consistency contrastive loss is deﬁned as:
Revised: Let i 2 { 1 , ..., b } be the index of a weakly-augmented input, and j ( i ) be the index of the strong- augmented input from the same source image, the consistency contrastive loss is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where ⌧ is a scalar temperature parameter.
Revised: where ⌧ is a scalar temperature parameter.

Original: The consistency contrastive loss maximizes the inner product between the pair of positive embeddings
Revised: The consistency contrastive loss maximizes the inner product between the pair of positive embeddings

Original: ˆ z i and ˆ z j ( i ) , while minimizing the inner product between 2( b  1) pairs of negative embeddings.
Revised: ˆ z i and ˆ z j ( i ) , while minimizing the inner product between 2( b  1) pairs of negative embeddings.

Original: By mapping different views (augmentations) of the same image to neighboring embeddings, the consistency contrastive loss encourages the network to learn discriminative representation that is robust to low-level image corruption.
Revised: By mapping different views (augmentations) of the same image to neighboring embeddings, the consistency contrastive loss encourages the network to learn discriminative representation that is robust to low-level image corruption.

Original: Supervised mixup prototypical contrastive loss .
Revised: Weakly-supervised mixup prototypical contrastive loss .

Original: Our second contrastive loss injects structural knowledge of classes into the embedding space.
Revised: Our second contrastive loss injects structural knowledge of classes into the embedding space.

Original: Let I c denote indices for the subset of images in D labeled with class c , we calculate the class prototype as the normalized mean embedding:
Revised: Let I c denote indices for the subset of images in D labeled with class c , we calculate the class prototype as the normalized mean embedding:

Original: [Equation]
Revised: [Equation]

Original: where ˆ z i is the embedding of a center-cropped image, and the class prototypes are calculated at the beginning of each epoch.
Revised: where ˆ z i is the embedding of a center-cropped image, and the class prototypes are calculated at the beginning of each epoch.

Original: The prototypical contrastive loss enforces an image embedding ˆ z i to be more similar to its corresponding class prototype
Revised: The prototypical contrastive loss enforces an image embedding ˆ z i to be more similar to its corresponding class prototype

Original: ˆ z
Revised: ˆ z

Original: y i , in contrast to other class prototypes:
Revised: y i , in contrast to other class prototypes:

Original: [Equation]
Revised: [Equation]

Original: Since the label y i is noisy, we would like to regularize the encoder from memorizing training labels.
Revised: Since the label y i is noisy, we would like to regularize the encoder from memorizing training labels.

Original: Mixup (Zhang et al., 2018) has been shown to be an effective method against label noise (Arazo et al., 2019;
Revised: Mixup (Zhang et al., 2018) has been shown to be an effective method against label noise (Arazo et al., 2019;

Original: Li et al., 2020a).
Revised: Li et al., 2020a).

Original: Inspired by it, we create virtual training samples by linearly interpolating a sample (indexed by i ) with another sample (indexed by m ( i ) ) randomly chosen from the same minibatch:
Revised: Inspired by it, we create virtual training samples by linearly interpolating a sample (indexed by i ) with another sample (indexed by m ( i ) ) randomly chosen from the same minibatch:

Original: [Equation]
Revised: [Equation]

Original: where  ⇠ Beta( ↵ , ↵ ) .
Revised: where  ⇠ Beta( ↵ , ↵ ) .

Original: Let ˆ z mi be the normalized embedding for x mi , the mixup version of the prototypical contrastive loss is deﬁned as a weighted combination of the two L pc w.r.t class y i and y m ( i ) .
Revised: Let ˆ z mi be the normalized embedding for x mi , the mixup version of the prototypical contrastive loss is deﬁned as a weighted combination of the two L pc w.r.t class y i and y m ( i ) .

Original: It enforces the embedding for the interpolated input to have the same linear relationship w.r.t.
Revised: It enforces the embedding for the interpolated input to have the same linear relationship w.r.t.

Original: the class prototypes.
Revised: the class prototypes.

Original: [Equation]
Revised: [Equation]

Original: Reconstruction loss .
Revised: Reconstruction loss .

Original: We also train a linear decoder W d to reconstruct the high-dimensional feature v i based on z i .
Revised: We also train a linear decoder W d to reconstruct the high-dimensional feature v i based on z i .

Original: The reconstruction loss is deﬁned as:
Revised: The reconstruction loss is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: There are several beneﬁts for training the autoencoder.
Revised: There are several beneﬁts for training the autoencoder.

Original: First, with an optimal linear autoencoder,
Revised: First, with an optimal linear autoencoder,

Original: W e will project v i into its low-dimensional principal subspace and can be understood as applying
Revised: W e will project v i into its low-dimensional principal subspace and can be understood as applying

Original: PCA (Baldi & Hornik, 1989).
Revised: PCA (Baldi & Hornik, 1989).

Original: Thus the low-dimensional representation z i is intrinsically robust to input noise.
Revised: Thus the low-dimensional representation z i is intrinsically robust to input noise.

Original: Second, minimizing the reconstruction error is maximizing a lower bound of the mutual information between v i and z i (Vincent et al., 2010).
Revised: Second, minimizing the reconstruction error is maximizing a lower bound of the mutual information between v i and z i (Vincent et al., 2010).

Original: Therefore, knowledge learned from the proposed contrastive losses can be maximally preserved in the high-dimensional representation, which helps regularize the classiﬁer.
Revised: Therefore, knowledge learned from the proposed contrastive losses can be maximally preserved in the high-dimensional representation, which helps regularize the classiﬁer.

Original: Classiﬁcation loss .
Revised: Classiﬁcation loss .

Original: Given the softmax output from the classiﬁer, p ( y ; x i ) , we deﬁne the classiﬁcation loss as the cross-entropy loss.
Revised: Given the softmax output from the classiﬁer, p ( y ; x i ) , we deﬁne the classiﬁcation loss as the cross-entropy loss.

Original: Note that it is only applied to the weakly-augmented inputs.
Revised: Note that it is only applied to the weakly-augmented inputs.

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original: The overall training objective is to minimize a weighted sum of all losses:
Revised: The overall training objective is to minimize a weighted sum of all losses:

Original: [Equation]
Revised: [Equation]

Original: For all experiments, we ﬁx !
Revised: For all experiments, we ﬁx !

Original: cc = 1 , !
Revised: cc = 1 , !

Original: recon = 1 , and change !
Revised: recon = 1 , and change !

Original: pc only across datasets.
Revised: pc only across datasets.

Original: [Figure]

Original: 3. N OISE CLEANING WITH SMOOTH NEIGHBORS
Revised: 3. N OISE CLEANING WITH SMOOTH NEIGHBORS

Original: After warming-up the model by training with the noisy labels { y i } n i =1 for t 0 epochs, we aim to clean the noise by generating a soft pseudo-label q i for each training sample.
Revised: After warming-up the model by training with the noisy labels { y i } ni =1 for t 0 epochs, we aim to clean the noise by generating a soft pseudo-label q i for each training sample.

Original: Different from previous methods that perform label correction purely using the model’s softmax prediction, our method exploits the structure of the low-dimensional subspace by aggregating information from top- k neighboring samples, which helps alleviate the conﬁrmation bias problem.
Revised: Different from previous methods that perform label correction purely using the model’s softmax prediction, our method exploits the structure of the low-dimensional subspace by aggregating information from top- k neighboring samples, which helps alleviate the conﬁrmation bias problem.

Original: At the t -th epoch, for each sample x i , let p ti be the classiﬁer’s softmax prediction, let q t  1 i be its soft label from the previous epoch, we calculate the soft label for the current epoch as:
Revised: At the t -th epoch, for each sample x i , let p ti be the classiﬁer’s softmax prediction, let q t  1 i be its soft label from the previous epoch, we calculate the soft label for the current epoch as:

Original: [Equation]
Revised: [Equation]

Original: where w tij represents the normalized afﬁnity between a sample and its neighbor and is deﬁned as w tij
Revised: where w tij represents the normalized afﬁnity between a sample and its neighbor and is deﬁned as w tij

Original: = exp
Revised: = exp

Original: [Equation]
Revised: [Equation]

Original: ˆ z
Revised: ˆ z

Original: [Equation]
Revised: [Equation]

Original:  ⌧ 
Revised: / ⌧ )

Original: [Equation] P 
Revised: P kj =

Original: [Equation]
Revised: [Equation]

Revised: exp( ˆ z ti · ˆ z tj / ⌧ ) .

Original: We set k = 200 in all experiments.
Revised: We set k = 200 in all experiments.

Original: The soft label deﬁned by eqn.(9) is the minimizer of the following quadratic loss function:
Revised: The soft label deﬁned by eqn.(9) is the minimizer of the following quadratic loss function:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term is a smoothness constraint which encourages the soft label to take a similar value as its neighbors’ labels, whereas the second term attempts to maintain the model’s class prediction.
Revised: The ﬁrst term is a smoothness constraint which encourages the soft label to take a similar value as its neighbors’ labels, whereas the second term attempts to maintain the model’s class prediction.

Original: We construct a supervised subset which contains (1) clean sample whose soft label score for the original class y i is higher than a threshold ⌘ 0 , (2) pseudo-labeled sample whose maximum soft label score exceeds a threshold ⌘ 1 .
Revised: We construct a weakly-supervised subset which contains (1) clean sample whose soft label score for the original class y i is higher than a threshold ⌘ 0 , (2) pseudo-labeled sample whose maximum soft label score exceeds a threshold ⌘ 1 .

Original: For pseudo-labeled samples, we convert their soft labels into hard labels by taking the class with the maximum score.
Revised: For pseudo-labeled samples, we convert their soft labels into hard labels by taking the class with the maximum score.

Original: [Equation]
Revised: [Equation]

Original: Given the supervised subset, we modify the classiﬁcation loss L ce , the mixup prototypical contrastive loss L pc mix , and the calculation of prototypes ˆ z c , such that they only use samples from D t sup .
Revised: Given the weakly-supervised subset, we modify the classiﬁcation loss L ce , the mixup prototypical contrastive loss L pc mix , and the calculation of prototypes ˆ z c , such that they only use samples from D t sup .

Original: The unsupervised losses ( i .
Revised: The unsupervised losses ( i .

Original: e .
Revised: e .

Original: L cc and L recon ) still operate on all training samples.
Revised: L cc and L recon ) still operate on all training samples.

Original: Learning curriculum.
Revised: Learning curriculum.

Original: Our iterative noise cleaning method learns an effective training curriculum, which gradually increases the size of D t sup as the pseudo-labels become more accurate.
Revised: Our iterative noise cleaning method learns an effective training curriculum, which gradually increases the size of D t sup as the pseudo-labels become more accurate.

Original: To demonstrate  such curriculum, we analyse the noise cleaning statistics for training our model on CIFAR-10 and
Revised: To demonstrate [Table] such curriculum, we analyse the noise cleaning statistics for training our model on CIFAR-10 and

Original: CIFAR-100 datasets with 50% label noise (experimental details explained in the next section).
Revised: CIFAR-100 datasets with 50% label noise (experimental details explained in the next section).

Original: In
Revised: In

Original: Figure 3 (a), we show the accuracy of the soft pseudo-labels w.r.t to clean training labels (only used for analysis purpose).
Revised: Figure 3 (a), we show the accuracy of the soft pseudo-labels w.r.t to clean training labels (only used for analysis purpose).

Original: Our method can signiﬁcantly reduce the ratio of label noise from 50% to 5% (for CIFAR-10) and 17% (for CIFAR-100).
Revised: Our method can signiﬁcantly reduce the ratio of label noise from 50% to 5% (for CIFAR-10) and 17% (for CIFAR-100).

Original: Figure 3 (b) shows the size of D t sup as a percentage of the total number of training samples, and Figure 3 (c) shows the effective label noise ratio within the supervised subset D t sup .
Revised: Figure 3 (b) shows the size of D t sup as a percentage of the total number of training samples, and Figure 3 (c) shows the effective label noise ratio within the weakly-supervised subset D t sup .

Original: Our method maintains a low noise ratio in the supervised subset, while gradually increasing its size to utilize more samples for the supervised losses.[Table]
Revised: Our method maintains a low noise ratio in the weakly-supervised subset, while gradually increasing its size to utilize more samples for the weakly-supervised losses.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: In this section, we validate the proposed method on multiple benchmarks with controlled noise and real-world noise.
Revised: In this section, we validate the proposed method on multiple benchmarks with controlled noise and real-world noise.

Original: Our method achieves state-of-the-art performance across all benchmarks.
Revised: Our method achieves state-of-the-art performance across all benchmarks.

Original: For fair comparison, we compare with DivideMix (Li et al., 2020a) without ensemble.
Revised: For fair comparison, we compare with DivideMix (Li et al., 2020a) without ensemble.

Original: In appendix A, we report the result of our method with co-training and ensemble, which further improves performance.
Revised: In appendix A, we report the result of our method with co-training and ensemble, which further improves performance.

Original: 4. E XPERIMENTS ON CONTROLLED NOISY LABELS
Revised: 4. E XPERIMENTS ON CONTROLLED NOISY LABELS

Original: Dataset.
Revised: Dataset.

Original: Following Tanaka et al.
Revised: Following Tanaka et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: (2020a), we corrupt the training data of CIFAR10 and CIFAR-100 (Krizhevsky & Hinton, 2009) with two types of label noise: symmetric and asymmetric .
Revised: (2020a), we corrupt the training data of CIFAR10 and CIFAR-100 (Krizhevsky & Hinton, 2009) with two types of label noise: symmetric and asymmetric .

Original: Symmetric noise is injected by randomly selecting a percentage of samples and changing their labels to random labels.
Revised: Symmetric noise is injected by randomly selecting a percentage of samples and changing their labels to random labels.

Original: Asymmetric noise is class-dependant, where labels are only changed to similar classes ( e .
Revised: Asymmetric noise is class-dependant, where labels are only changed to similar classes ( e .

Original: g .
Revised: g .

Original: dog
Revised: dog

Original: $ cat, deer ! horse).
Revised: $ cat, deer ! horse).

Original: We experiment with multiple noise ratios: sym 20%, sym 50%, and asym 40% (see results for sym 80% and 90% in appendix A).
Revised: We experiment with multiple noise ratios: sym 20%, sym 50%, and asym 40% (see results for sym 80% and 90% in appendix A).

Original: Note that asymmetric noise ratio cannot exceed 50% because certain classes would become theoretically indistinguishable.
Revised: Note that asymmetric noise ratio cannot exceed 50% because certain classes would become theoretically indistinguishable.

Original: Implementation details.
Revised: Implementation details.

Original: Same as previous works (Arazo et al., 2019; Li et al., 2020a), we use
Revised: Same as previous works (Arazo et al., 2019; Li et al., 2020a), we use

Original: PreAct ResNet-18 (He et al., 2016) as our encoder model.
Revised: PreAct ResNet-18 (He et al., 2016) as our encoder model.

Original: We set the dimensionality of the bottleneck layer as d = 50 .
Revised: We set the dimensionality of the bottleneck layer as d = 50 .

Original: Our model is trained using SGD with a momentum of 0.9, a weight decay of 0.0005, and a batch size of 128.
Revised: Our model is trained using SGD with a momentum of 0.9, a weight decay of 0.0005, and a batch size of 128.

Original: The network is trained for 200 epochs.
Revised: The network is trained for 200 epochs.

Original: We set the initial learning rate as 0.02 and use a cosine decay schedule.
Revised: We set the initial learning rate as 0.02 and use a cosine decay schedule.

Original: We apply standard crop and horizontal ﬂip as the weak augmentation.
Revised: We apply standard crop and horizontal ﬂip as the weak augmentation.

Original: For strong augmentation, we use AugMix (Hendrycks et al., 2020), though other methods ( e . g .
Revised: For strong augmentation, we use AugMix (Hendrycks et al., 2020), though other methods ( e . g .

Original: SimAug (Chen et al., 2020)) work equally well.
Revised: SimAug (Chen et al., 2020)) work equally well.

Original: For all CIFAR experiments, we ﬁx the hyper-parameters as !
Revised: For all CIFAR experiments, we ﬁx the hyper-parameters as !

Original: cc = 1 , !
Revised: cc = 1 , !

Original: pc = 5 , !
Revised: pc = 5 , !

Original: recon = 1 , ⌧ = 0 . 3 , ↵ = 8 , ⌘ 1 = 0 .
Revised: recon = 1 , ⌧ = 0 . 3 , ↵ = 8 , ⌘ 1 = 0 .

Original: For CIFAR-10, we activate noise cleaning at epoch t 0 = 5 , and set ⌘ 0 = 0 .
Revised: For CIFAR-10, we activate noise cleaning at epoch t 0 = 5 , and set ⌘ 0 = 0 .

Original: 1 (sym.) or 0.4 (asym.).
Revised: 1 (sym.) or 0.4 (asym.).

Original: For CIFAR-100, we activate noise cleaning at epoch t 0 = 15 , and set ⌘ 0 = 0 .
Revised: For CIFAR-100, we activate noise cleaning at epoch t 0 = 15 , and set ⌘ 0 = 0 .

Original: We use faiss-gpu (Johnson et al., 2017) for efﬁcient knn search in the low-dimensional subspace, which ﬁnishes within 1 second.
Revised: We use faiss-gpu (Johnson et al., 2017) for efﬁcient knn search in the low-dimensional subspace, which ﬁnishes within 1 second.

Original: Results.
Revised: Results.

Original: Table 1 shows the comparison with existing methods.
Revised: Table 1 shows the comparison with existing methods.

Original: Our method outperforms previous methods across all label noise settings.
Revised: Our method outperforms previous methods across all label noise settings.

Original: On the more challenging CIFAR-100, we achieve 3-4% accuracy improvement compared to the second-best method DivideMix.
Revised: On the more challenging CIFAR-100, we achieve 3-4% accuracy improvement compared to the second-best method DivideMix.

Original: Moreover, our method is more computational efﬁcient than DivideMix, which needs co-training for noise ﬁltering.
Revised: Moreover, our method is more computational efﬁcient than DivideMix, which needs co-training for noise ﬁltering.

Original:  In order to demonstrate the advantage of the proposed low-dimensional embeddings, we perform knearest neighbor (knn) classiﬁcation ( k = 200 ), by projecting test images into normalized embeddings.
Revised: [Table] In order to demonstrate the advantage of the proposed low-dimensional embeddings, we perform knearest neighbor (knn) classiﬁcation ( k = 200 ), by projecting test images into normalized embeddings.

Original: Compared to the trained classiﬁer, knn achieves higher accuracy, which veriﬁes the robustness of the learned low-dimensional representations.
Revised: Compared to the trained classiﬁer, knn achieves higher accuracy, which veriﬁes the robustness of the learned low-dimensional representations.

Original: 4. E XPERIMENTS ON CONTROLLED NOISY LABELS WITH NOISY IMAGES
Revised: 4. E XPERIMENTS ON CONTROLLED NOISY LABELS WITH NOISY IMAGES

Original: Dataset.
Revised: Dataset.

Original: We further corrupt a noisy CIFAR-10 dataset (sym.
Revised: We further corrupt a noisy CIFAR-10 dataset (sym.

Original: 50%) by injecting two types of input noise: out-of-distribution (OOD) images and input corruption.
Revised: 50%) by injecting two types of input noise: out-of-distribution (OOD) images and input corruption.

Original: For OOD noise, we follow Wang et al.
Revised: For OOD noise, we follow Wang et al.

Original: [Table] (2018) and add 20 k images from either one of the two other datasets: CIFAR-100 and SVHN (Netzer et al., 2011), enlarging the training set to 70 k .
Revised: (2018) and add 20 k images from either one of the two other datasets: CIFAR-100 and SVHN (Netzer et al., 2011), enlarging the training set to 70 k .

Original: A random CIFAR-10 label is assigned to each OOD image.
Revised: A random CIFAR-10 label is assigned to each OOD image.

Original: For input corruption, we follow Hendrycks & Dietterich (2019) and corrupt each image in
Revised: For input corruption, we follow Hendrycks & Dietterich (2019) and corrupt each image in

Original: CIFAR-10 with a noise randomly chosen from the following four types:
Revised: CIFAR-10 with a noise randomly chosen from the following four types:

Original: Fog , Snow , Motion blur and
Revised: Fog , Snow , Motion blur and

Original: Gaussian noise .
Revised: Gaussian noise .

Original: Examples of both types of input noise are shown in Figure 4.
Revised: Examples of both types of input noise are shown in Figure 4.

Original: We follow the same implementation details as the CIFAR-10 experiments described in Section 4.1.
Revised: We follow the same implementation details as the CIFAR-10 experiments described in Section 4.1.

Original: [Figure]
Revised: [Figure]

Original: Results.
Revised: Results.

Original: Table 2 shows the results, where our method consistently outperforms existing methods by a substantial margin.
Revised: Table 2 shows the results, where our method consistently outperforms existing methods by a substantial margin.

Original: We observe that OOD images from a similar domain (CIFAR-100) are more harmful than OOD images from a more different domain (SVHN).
Revised: We observe that OOD images from a similar domain (CIFAR-100) are more harmful than OOD images from a more different domain (SVHN).

Original: This is because noisy images that are closer to the test data distribution are more likely to distort the decision boundary in a way that negatively affects test performance.
Revised: This is because noisy images that are closer to the test data distribution are more likely to distort the decision boundary in a way that negatively affects test performance.

Original: Nevertheless, performing knn classiﬁcation using the learned embeddings demonstrates high robustness to input noise.
Revised: Nevertheless, performing knn classiﬁcation using the learned embeddings demonstrates high robustness to input noise.

Original: In Figure 5, we show the t-SNE (Maaten & Hinton, 2008) visualization of the low-dimensional embeddings for all training samples.
Revised: In Figure 5, we show the t-SNE (Maaten & Hinton, 2008) visualization of the low-dimensional embeddings for all training samples.

Original: As training progresses, our model learns to separate OOD samples (represented as gray points) from in-distribution samples, and cluster samples of the same class together despite their noisy labels. 
Revised: As training progresses, our model learns to separate OOD samples (represented as gray points) from in-distribution samples, and cluster samples of the same class together despite their noisy labels. [Table] [Table]

Original: 4. E XPERIMENTS ON REAL - WORLD NOISY DATA
Revised: 4. E XPERIMENTS ON REAL - WORLD NOISY DATA

Original: Dataset and implementation details.
Revised: Dataset and implementation details.

Original: We verify our method on two real-word noisy datasets:
Revised: We verify our method on two real-word noisy datasets:

Original: WebVision (Li et al., 2017) and Clothing1M (Xiao et al., 2015).
Revised: WebVision (Li et al., 2017) and Clothing1M (Xiao et al., 2015).

Original: Webvision contains images crawled from the web using the same concepts from ImageNet ILSVRC

Revised: Webvision contains images crawled from the web using the same concepts from ImageNet ILSVRC

Original: (Deng et al., 2009).
Revised: (Deng et al., 2009).

Original: Following previous works (Chen et al., 2019; Li et al., 2020a), we perform experiments on the ﬁrst 50 classes of the Google image subset.
Revised: Following previous works (Chen et al., 2019; Li et al., 2020a), we perform experiments on the ﬁrst 50 classes of the Google image subset.

Original: Clothing1M consists of images collected from online shopping websites where labels were generated from surrounding texts.
Revised: Clothing1M consists of images collected from online shopping websites where labels were generated from surrounding texts.

Original: Note that we do not use the additional clean set [Table] [Table] [Table]
Revised: Note that we do not use the additional clean set for training.

Original: [Table]

Original: for training.

Original: For both experiments, we use the same model architecture as previous methods.
Revised: For both experiments, we use the same model architecture as previous methods.

Original: More implementation details are given in the appendix.
Revised: More implementation details are given in the appendix.

Original: Results.
Revised: Results.

Original: We report the results for WebVision in Table 3 and Clothing1M in Table 4, where we achieve state-of-the-art performance on both datasets.
Revised: We report the results for WebVision in Table 3 and Clothing1M in Table 4, where we achieve state-of-the-art performance on both datasets.

Original: Our method achieves competitive performance on WebVision even without performing noise cleaning, which demonstrates the robustness of the learned representation.
Revised: Our method achieves competitive performance on WebVision even without performing noise cleaning, which demonstrates the robustness of the learned representation.

Original: Appendix E shows examples of noisy images that are cleaned by our method.
Revised: Appendix D shows examples of noisy images that are cleaned by our method.

Original: 4. A BLATION STUDY
Revised: 4. A BLATION STUDY

Original: Effect of auto-encoder.
Revised: Effect of the proposed components.

Original: We remove the reconstruction loss L recon to study the effect of the autoencoder.
Revised: In order to study the effect of the proposed components, we remove each of them and report accuracy of the classiﬁer (knn) across four benchmarks.

Original: As shown in Table 5, test performance decrease across multiple benchmarks.
Revised: As shown in Table 5, the mixup prototypical contrastive loss ( L pc mix ) is most crucial to the model’s performance.

Original: This veriﬁes the effect  of the auto-encoder for mapping the high-dimensional representation to its principal subspace, and grounding the structure learned in the low-dimensional subspace back to the high-dimensional feature space. In the appendix, we also perform experiments to verify the importance of the two contrastive losses ( i .
Revised: The consistency contrastive loss ( L cc ) has a stronger effect with corrupted input or larger number of classes. We also experiment with removing mixup and using the standard prototypical contrastive loss, and using standard data augmentation (crop and horizontal ﬂip) instead of AugMix. The proposed method still achieves state-of-the-art result with standard data augmentation.

Original: e . L cc and L pc mix ).
Revised: [Table]

Original: Effect of bottleneck dimension.
Revised: Effect of bottleneck dimension.

Original: We vary the dimensionality of the bottleneck layer, d , and examine the performance change in Table 6.
Revised: We vary the dimensionality of the bottleneck layer, d , and examine the performance change in Table 6.

Original: Our model is in general not very sensitive to the change of d .
Revised: Our model is in general not very sensitive to the change of d .

Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper proposes noise-robust contrastive learning, a new method to combat noise in training data by learning robust representation.
Revised: This paper proposes noise-robust contrastive learning, a new method to combat noise in training data by learning robust representation.

Original: We demonstrate our model’s state-of-the-art performance with extensive experiments on multiple noisy datasets.
Revised: We demonstrate our model’s state-of-the-art performance with extensive experiments on multiple noisy datasets.

Original: For future work, we are interested in adapting our method to other domains such as NLP or speech.
Revised: For future work, we are interested in adapting our method to other domains such as NLP or speech.

Original: We would also like to explore the potential of our method for learning transferable representations that could be useful for down-stream tasks.
Revised: We would also like to explore the potential of our method for learning transferable representations that could be useful for down-stream tasks