Original: S3: S UPERVISED S ELF - SUPERVISED L EARNING UNDER L ABEL N OISE
Revised: S3: S UPERVISED S ELF - SUPERVISED L EARNING UNDER L ABEL N OISE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite the large progress in supervised learning with Neural Networks, there are signiﬁcant challenges in obtaining high-quality, large-scale and accurately labeled datasets.
Revised: Despite the large progress in supervised learning with Neural Networks, there are signiﬁcant challenges in obtaining high-quality, large-scale and accurately labeled datasets.

Original: In this context, in this paper we address the problem of classiﬁcation in the presence of label noise and more speciﬁcally, both close-set and open-set label noise, that is when the true label of a sample may, or may not belong to the set of the given labels.
Revised: In this context, in this paper we address the problem of classiﬁcation in the presence of label noise and more speciﬁcally, both close-set and open-set label noise, that is when the true label of a sample may, or may not belong to the set of the given labels.

Original: In the heart of our method is a sample selection mechanism that relies on the consistency between the annotated label of a sample and the distribution of the labels in its neighborhood in the feature space; a relabeling mechanism that relies on the conﬁdence of the classiﬁer across subsequent iterations; and a training strategy that trains the encoder both with a self-consistency loss and the classiﬁer-encoder with the cross-entropy loss on the selected samples alone.
Revised: In the heart of our method is a sample selection mechanism that relies on the consistency between the annotated label of a sample and the distribution of the labels in its neighborhood in the feature space; a relabeling mechanism that relies on the conﬁdence of the classiﬁer across subsequent iterations; and a training strategy that trains the encoder both with a self-consistency loss and the classiﬁer-encoder with the cross-entropy loss on the selected samples alone.

Original: Without bells and whistles, such as co-training so as to reduce the self-conﬁrmation bias, and with robustness with respect to settings of its few hyper-parameters, our method signiﬁcantly surpasses previous methods on both
Revised: Without bells and whistles, such as co-training so as to reduce the self-conﬁrmation bias, and with robustness with respect to settings of its few hyper-parameters, our method signiﬁcantly surpasses previous methods on both

Original: CIFAR10/CIFAR100 with artiﬁcial noise and real-world noisy datasets such as
Revised: CIFAR10/CIFAR100 with artiﬁcial noise and real-world noisy datasets such as

Original: WebVision and ANIMAL-10N.
Revised: WebVision and ANIMAL-10N.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: It is now commonly accepted that supervised learning with deep neural networks can provide excellent solutions for a wide range of problems, so long as there is sufﬁcient availability of labeled training data and computational resources.
Revised: It is now commonly accepted that supervised learning with deep neural networks can provide excellent solutions for a wide range of problems, so long as there is sufﬁcient availability of labeled training data and computational resources.

Original: However, these results have been mostly obtained using well-curated datasets in which the classes are balanced and the labels are of high quality.
Revised: However, these results have been mostly obtained using well-curated datasets in which the classes are balanced and the labels are of high quality.

Original: In the realworld, it is often costly to obtain high quality labels especially for large-scale datasets.
Revised: In the realworld, it is often costly to obtain high quality labels especially for large-scale datasets.

Original: A common approach is to use semi-automatic methods to obtain the labels (e.g. “webly-labeled” images where the images and labels are obtained by web-crawling).
Revised: A common approach is to use semi-automatic methods to obtain the labels (e.g. “webly-labeled” images where the images and labels are obtained by web-crawling).

Original: While such methods can greatly reduce the time and cost of manual labeling, they also lead to low quality noisy labels.
Revised: While such methods can greatly reduce the time and cost of manual labeling, they also lead to low quality noisy labels.

Original: To deal with noisy labels, earlier approaches tried to improve the robustness of the model using robust loss functions(Ghosh et al., 2017; Zhang & Sabuncu, 2018;
Revised: To deal with noisy labels, earlier approaches tried to improve the robustness of the model using robust loss functions (Ghosh et al., 2017; Zhang & Sabuncu, 2018;

Original: Wang et al., 2019) or robust regularizations (Srivastava et al., 2014; Zhang et al., 2017; Pereyra et al., 2017).
Revised: Wang et al., 2019) or robust regularizations (Srivastava et al., 2014; Zhang et al., 2017; Pereyra et al., 2017).

Original: Goldberger & BenReuven (2016) tried to model the noise transition matrix between classes while Han et al.
Revised: Goldberger & BenReuven (2016) tried to model the noise transition matrix between classes while Han et al.

Original: Patrini et al.
Revised: Patrini et al.

Original: ; Hendrycks et al.
Revised: ; Hendrycks et al.

Original: (2018) proposed to correct the losses of noisy samples.
Revised: (2018) proposed to correct the losses of noisy samples.

Original: More recently, sample selection methods became perhaps the dominant paradigm for learning with noisy labels.
Revised: More recently, sample selection methods became perhaps the dominant paradigm for learning with noisy labels.

Original: Most of the recent sample selection methods do so, by relying on the predictions of the model classiﬁer, for example on the per-sample loss (Arazo et al., 2019; Li et al., 2020a) or model prediction (Song et al., 2019; Malach & Shalev-Shwartz, 2017).
Revised: Most of the recent sample selection methods do so, by relying on the predictions of the model classiﬁer, for example on the per-sample loss (Arazo et al., 2019; Li et al., 2020a) or model prediction (Song et al., 2019; Malach & Shalev-Shwartz, 2017).

Original: By separating clean samples and noisy samples and subsequently performing supervised training on the clean set, or semi-supervised training on both, sample selection methods achieved the state-of-the-art results in synthetic and realworld noisy datasets.
Revised: By separating clean samples and noisy samples and subsequently performing supervised training on the clean set, or semi-supervised training on both, sample selection methods achieved the state-of-the-art results in synthetic and realworld noisy datasets.

Original: However, there are three main issues with current sample selection methods.
Revised: However, there are three main issues with current sample selection methods.

Original: Firstly, themodel classiﬁer based sample selection will be inevitably biased if  trained with noisy labels – this is immediately apparent in the case that the sample selection is based on the loss of the model itself (Arazo et al., 2019;
Revised: Firstly, the sample selection will be inevitably biased if the models (classiﬁer and feature extractor) are trained with noisy labels – this is immediately apparent in the case that the sample selection is based on the loss of the classiﬁer itself (Arazo et al., 2019;

Original: Li et al., 2020a; Yu et al., 2019; Han et al., 2018).
Revised: Li et al., 2020a; Yu et al., 2019; Han et al., 2018).

Original: This phenomenon is aggravated by the accumulation of errors induced by the iterative nature of self-train methods where labels are corrected based on the model’s predictions – the so-called self-conﬁrmation bias.

Original: Secondly, in su- pervised classiﬁcation problems , noisy samples usually come from two main categories: closed-set noise where the true labels belong to one of the given classes  and open-set noise where the true labels do not belong to the set of labels of the classiﬁcation problem.
Revised: Second, in supervised classiﬁcation problems, noisy samples usually come from two main categories: closedset noise where the true labels belong to one of the given classes (Set B in Fig. 1) and open-set noise where the true labels do not belong to the set of labels of the classiﬁcation problem (Set C in Fig. 1).

Original: Most of the works in the literature, including works that estimate the probabilities of label-exchange between pairs of classes (Goldberger & Ben-Reuven, 2016; Patrini et al., 2017), that do relabeling based on the model’s predictions (Song et al., 2019; Han et al., 2019) or works that adopt semi-supervised approacheswhere samples identiﬁed as noisy are considered unlabeled (Li et al., 2020a; Ortego et al., 2021) deal with the former and not directly address the latter.
Revised: Most of the works in the literature, including works that estimate the probabilities of label-exchange between pairs of classes (Goldberger & Ben-Reuven, 2016; Patrini et al., 2017), that do relabeling based on the model’s predictions (Song et al., 2019; Han et al., 2019) or works that adopt semisupervised approaches (Li et al., 2020a; Ortego et al., 2021) deal with the former and not directly address the latter.

Original: However, this is a considerable source of noise in real world scenarios, e.g., when training from web-crawled data, where there is less control over the collection of the dataset.
Revised: However, this is a considerable source of noise in real world scenarios, e.g., when training from web-crawled data, where there is less control over the collection of the dataset.

Revised: Crucially, those works (implicitly or explicitly) relabel all samples and do training based on the new labels of all samples.

Revised: Those are bound to be wrong for all samples in set C, but also are bound to be wrong for several samples of A and B that are (implicitly or explicitly) relabeled in the early stages of training.

Revised: For the latter reason, those works do not work well even under heavy close-set noise.

Original: Finally, current approaches usually require extensive hyperparameters tuning, often even on a per-dataset basi – this is unrealistic in scenarios where there is little knowledge about the types of noise.
Revised: Finally, current approaches usually require extensive hyperparameters tuning, often even on a perdataset basis – this is unrealistic in scenarios where there is little knowledge about the types of noise.

Original: This is partly due to the complexity of the applied semi-supervised learning method, and partly because of the complicated methods that are employed, such as model pretraining (Zheltonozhskii et al., 2021) and model cotraining (Han et al., 2018; Yu et al., 2019; Li et al., 2020a), so as to deal with self-conﬁrmation bias.
Revised: This is partly due to the complexity of the applied semi-supervised learning method, and partly because of the complicated methods that are employed, such as model pretraining (Zheltonozhskii et al., 2021) and model cotraining (Han et al., 2018; Yu et al., 2019; Li et al., 2020a), so as to deal with self-conﬁrmation bias.

Original: In this paper, we address the problem of training under different types of noise with a simple method– S3 ,  that aims at learning a robust feature space, relying on both noisy and clean samples, and identifying in the feature space reliably, clean samples based on the distribution of the labels of the samples in their neighborhood.
Revised: In this paper, we address the problem of training under different types of noise with a simple method– namely Supervised, Self-Supervised learning (S3) , with two major components that are clearly separated: a selection/relabelling mechanism that selects/relabels samples so as to construct a clean and a noisy set (Section 3.3), and a training framework that aims at learning a strong feature extractor f and a classiﬁcation head g

Original:  In the proposed scheme, learning of the feature space is based two factors:
Revised: [Fig. 2] from both noisy and clean samples (Section 3.4). In the training stage we use off-the-shelf learners and a) train the feature extractor and the classiﬁcation head using a classical Supervised cross-entropy loss applied only on the clean samples – this avoids treating samples identiﬁed as noisy as if they belonged to one of the given classes as most methods (Arazo et al., 2019;

Original: First, the consistency loss between the representations of augmented version of the sample (as in (Chen & He, 2021))  – this avoids false-negatives that are inherent in contrastive-learning with instance discrimination; second a classical cross-entropy loss applied only on clean samples – this avoids treating samples identiﬁed as noisy as if they belonged to one of the given classes as most methods (Arazo et al., 2019; Li et al., 2020a; Ortego et al., 2021; Wu et al., 2021) do.
Revised: Li et al., 2020a; Ortego et al., 2021; Wu et al., 2021) implictly or explicitly do; and b) train the feature extractor using a Self-Supervised loss , namely the consistency loss between the representations of augmented version of the sample (as in (Chen & He, 2021)) applied on all samples – this avoids false-negatives that are inherent in contrastive-learning with instance discrimination and different to works that apply the consistency on the label predictions that are unreliable at the ﬁrst iterations or in the presence of open-set noise.

Original: The noisy sample selection mechanism relies on a measure of conﬁdence what we deﬁne using the ground truth label of the sample in question and an estimate of distribution of the labels of its neighbours – in order to deal with noisy samples, we adopt a scheme in which the distribution is calculated based both on the ground truth labels and on consistently (over subsequent iterations) conﬁdent estimates of the labels.
Revised: The noisy sample selection mechanism relies on a measure of conﬁdence what we deﬁne using the ground truth label of the sample in question and an estimate of distribution of the labels of its neighbours – in order to deal with noisy samples, we adopt a scheme in which the distribution is calculated based both on the ground truth labels and on consistently (over subsequent iterations) conﬁdent estimates of the labels.

Original: Our method is embedded into a standard MixUp and data augmentation framework, and without bells and whistles, such as co-training of multiple models, it achieves state-of-the-art results in both synthetic and realistic noise patterns in CIFAR10, CIFAR100, ANIMAL-10N and WebVision datasets.
Revised: Our method is embedded into a standard MixUp and data augmentation framework, and without bells and whistles, such as co-training of multiple models, it achieves state-of-the-art results in both synthetic and realistic noise patterns in CIFAR10, CIFAR100, ANIMAL-10N, Clothing1M and WebVision datasets.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Leaning with noisy data by sample selection Some works focused on sample selection to ﬁlter out noisy samples.
Revised: Leaning with noisy data by sample selection Some works focused on sample selection to ﬁlter out noisy samples.

Original: Jiang et al.
Revised: Jiang et al.

Original: (2018) introduced a pretrained mentor network to guide the selection of a student network.
Revised: (2018) introduced a pretrained mentor network to guide the selection of a student network.

Original: Song et al.
Revised: Song et al.

Original: (2019) evaluates the per-sample losses and identify as clean the top r % of the samples – the precise ratio r % depends is either predeﬁned, or is an estimate of the noise level in the speciﬁc dataset.
Revised: (2019) evaluates the per-sample losses and identify as clean the top r % of the samples – the precise ratio r % depends is either predeﬁned, or is an estimate of the noise level in the speciﬁc dataset.

Original: Arazo et al.
Revised: Arazo et al.

Original: (2019) proposed to model per-sample losses with a Beta
Revised: (2019) proposed to model per-sample losses with a Beta

Original: Mixture Model (BMM) and split the dataset according to which of the components of the mixture each sample belongs.
Revised: Mixture Model (BMM) and split the dataset according to which of the components of the mixture each sample belongs.

Original: In a very similar approach, Li et al.
Revised: In a very similar approach, Li et al.

Original: (2020a) extended upon Arazo et al.
Revised: (2020a) extended upon Arazo et al.

Original: by introducing semi-supervised learning to fully utilize the dataset.
Revised: by introducing semi-supervised learning to fully utilize the dataset.

Original: Related to our work, Bahri et al.
Revised: Related to our work, Bahri et al.

Original: (2020); Ortego et al.
Revised: (2020); Ortego et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2018) also utilized the feature space for sample selection.
Revised: (2018) also utilized the feature space for sample selection.

Original: Bahri et al.
Revised: Bahri et al.

Original: (2020) applied KNN for sample selection for closed-set noisy dataset while Ortego et al.
Revised: (2020) applied KNN for sample selection for closed-set noisy dataset while Ortego et al.

Original: (2021) further proposed to relabel samples based on the KNN voting.
Revised: (2021) further proposed to relabel samples based on the KNN voting.

Original: Wang et al.
Revised: Wang et al.

Original: (2018) proposed to reweight samples based on its probability of being outliers in open-set noisy dataset.
Revised: (2018) proposed to reweight samples based on its probability of being outliers in open-set noisy dataset.

Original: Self-supervised learning Self-supervised methods attempt to learn good representation without human annotations.
Revised: Self-supervised learning Self-supervised methods attempt to learn good representation without human annotations.

Original: In the recent years, the dominant method is contrastive learning with instance discrimination task.
Revised: In the recent years, the dominant method is contrastive learning with instance discrimination task.

Original: MoCo (He et al., 2020) is an important baseline for current contrastive learning methods, which reuses the memory bank since samples in a single mini-batch may lead to insufﬁcient negative pairs, and proposes a momentum encoder to update the memory bank in real-time to avoid outdated data representation.
Revised: MoCo (He et al., 2020) is an important baseline for current contrastive learning methods, which reuses the memory bank since samples in a single mini-batch may lead to insufﬁcient negative pairs, and proposes a momentum encoder to update the memory bank in real-time to avoid outdated data representation.

Original: SimCLR (Chen et al., 2020) is another important baseline which found that setting mini-batch size to be large enough can eliminate the need for memory bank.
Revised: SimCLR (Chen et al., 2020) is another important baseline which found that setting mini-batch size to be large enough can eliminate the need for memory bank.

Original: More recently, SimSiam (Chen & He, 2021) and BYOL (Grill et al., 2020) proposed a non-contrastive learning framework which enforce the perturbation consitency between different views, and avoid mode collapse by applying stop-gradient and an extra predictor between two representation vectors.
Revised: More recently, SimSiam (Chen & He, 2021) and BYOL (Grill et al., 2020) proposed a non-contrastive learning framework which enforce the perturbation consitency between different views, and avoid mode collapse by applying stop-gradient and an extra predictor between two representation vectors.

Original:  M ETHOD 3. P ROBLEM FORMULATION
Revised:  M ETHOD 3. P ROBLEM FORMULATION

Original: Let us denote with X
Revised: Let us denote with X

Original: = { x i } Ni =1 , x i ∈ R d , a training set with the corresponding one-hot vector labels Y =
Revised: = { x i } Ni =1 , x i ∈ R d , a training set with the corresponding one-hot vector labels Y =

Original: { y i } Ni =1 , y i ∈ { 0 , 1 } K , where K is the number of classes and N is the number of samples.
Revised: { y i } N i =1 , y i ∈ { 0 , 1 } K , where K is the number of classes and N is the number of samples.

Original: For convenience, let us also denote the index where the one-hot vector
Revised: For convenience, let us also denote the index where the one-hot vector

Original: y i is one as the label l i ∈ { 0 , ..., K } .
Revised: y i is one as the label l i ∈ { 0 , ..., K } .

Original: Finally, let us denote the true labels with Y (cid:48)
Revised: Finally, let us denote the true labels with Y (cid:48)

Original: = { y (cid:48)
Revised: = { y (cid:48)

Original: i } Ni =1 .
Revised: i } Ni =1 .

Original: Clearly, for an open-set noisy label it is the case that y (cid:48)
Revised: Clearly, for an open-set noisy label it is the case that y (cid:48)

Original: i (cid:54) =
Revised: i (cid:54) =

Original: y i , y (cid:48)
Revised: y i , y (cid:48)

Original: i / ∈ { 0 , 1 } K , while for closed-set noisy samples y (cid:48)
Revised: i / ∈ { 0 , 1 } K , while for closed-set noisy samples y (cid:48)

Original: i (cid:54) =
Revised: i (cid:54) =

Original: y i , y (cid:48)
Revised: y i , y (cid:48)

Original: i ∈ { 0 , 1 } K .
Revised: i ∈ { 0 , 1 } K .

Original: Our goal is to train an encoder f and a classiﬁer g such that f i (cid:44) f

Original: ( x i ) ∈

Original: R D and g i (cid:44) g ( f ( x i ))

Original: ∈ R K are, respectively, a feature representation and a prediction logit of the class of the sample x .

Original: Typically, the loss for a sample x

Original: i is deﬁned using the cross-entropy loss:

Original: [Equation]

Original: where z i (cid:44) Softmax ( g i ) ∈ R K is the prediction probability vector.

Original: We can easily decompose the back-propagating gradient as:

Original: [Equation]

Original: From Eq.

Original: 2 it becomes apparent that the prediction logit will be optimized to correct class for a correctly labeled sample ( y i =

Original: y (cid:48) i ) while for a noisy sample ( y i (cid:54) =

Original: y (cid:48) i ), the prediction logit will be optimized towards a wrong class.

Original: This results to corrupted models when learning under noise.

Original: 3. O VERVIEW OF PROPOSED METHOD
Revised: 3. O VERVIEW OF PROPOSED METHOD

Original: Aiming to deal with potential concurrence of both open-set and close-set noise, we view the classiﬁcation network as an encoder f
Revised: Aiming to deal with potential concurrence of both open-set and close-set noise, we view the classiﬁcation network as an encoder f

Original: that extracts a feature representation and a classiﬁcation head g that deals with the classiﬁcation problem in question.
Revised: that extracts a feature representation and a classiﬁcation head g that deals with the classiﬁcation problem in question.

Original: The proposed method, named Supervised
Revised: The proposed method, named Supervised

Original: Semi-Supervised(S3) learning, attempts to decouple their training so as to deal with possible noise in the labels, by adopting a two stage, iterative scheme, as outlined in Fig.
Revised: Self-Supervised(S3) learning, attempts to decouple their training so as to deal with possible noise in the labels, by adopting a two stage, iterative scheme, as outlined in Fig.

Original: In the ﬁrst stage, we utilize a novel sample selection and a novel relabeling mechanism (top block in
Revised: In the ﬁrst stage, we utilize a novel sample selection and a novel relabeling mechanism (top block in

Original: Fig. 1) that prepares the set based on which the classiﬁer g should be trained in Stage 2.
Revised: Fig. 2) that prepares the set based on which the classiﬁer g should be trained in Stage 2.

Original: The selection mechanism is based on the assumption of smoothness of labels in the feature space, and more speciﬁcally, on a consistency measure that we deﬁned based on the annotated label of the sample in question and the distribution of the labels in its neighborhood in the feature space.
Revised: The selection mechanism is based on the assumption of smoothness of labels in the feature space, and more speciﬁcally, on a consistency measure that we deﬁned based on the annotated label of the sample in question and the distribution of the labels in its neighborhood in the feature space.

Original: Relabeling is performed on samples for which the classiﬁer gives conﬁdent predictions consistently across subsequent iterations.
Revised: Relabeling is performed on samples for which the classiﬁer gives conﬁdent predictions consistently across subsequent iterations.

Original: Clearly, the mechanism relies on the quality of the features extracted by the encoder f and should reject samples whose true labels are not in the class set (open-set noise).
Revised: Clearly, the mechanism relies on the quality of the features extracted by the encoder f and should reject samples whose true labels are not in the class set (open-set noise).

Original: This stage is explained in Section 3.3.
Revised: This stage is explained in Section 3.3.

Original: In the second stage (bottom block in Fig. 1), training is performed with two objectives/losses.
Revised: In the second stage (bottom block in Fig. 2), training is performed with two objectives/losses.

Original: First, a cross entropy loss on the output of the classiﬁer g , (i.e., on g ( f ( . )) ) on the samples selected in Stage 1, that updates both the encoder f
Revised: First, a cross entropy loss on the output of the classiﬁer g , (i.e., on g ( f ( . )) ) on the samples selected in Stage 1, that updates both the encoder f

Original: and the classiﬁer head g .
Revised: and the classiﬁer head g .

Original: Second, a self-supervision loss that enforces consistency between the representations of different augmentations of the same sample, and which utilizes all samples, that is both noisy and clean – this updates the encoder f and helps learning a strong feature space on which the selection mechanism of Stage 1 can rely.
Revised: Second, a self-supervision loss that enforces consistency between the representations of different augmentations of the same sample, and which utilizes all samples, that is both noisy and clean – this updates the encoder f and helps learning a strong feature space on which the selection mechanism of Stagecan rely.

Original: This is in contrast to other methods, which base the selection mechanism on the classiﬁer itself, and are therefore more susceptible to self-conﬁrmation bias, or need to rely on complex schemes (e.g. co-training) to reduce it.
Revised: By contrast to other methods that, either by using a noise transition matrix or in their semisupervised scheme, implicitly relabel all samples (e.g. DivideMix (Li et al., 2020a), MOIT (Ortego et al., 2021)) and use the new labels to learn, in our method the labels in the noisy set are not used at all.

Original: This stage is explained in Section 3.4.
Revised: This stage is explained in Section 3.4.

Original: 3. S AMPLE SELECTION & RELABELLING
Revised: 3. S AMPLE SELECTION & RELABELLING

Revised: The sample selection and relabeling mechanism are designed so as to construct a clean subset that has as many correctly labelled samples as possible.

Original: Clean sample selection by normalized neighboring voting Let us denote the similarity between the representations f i and f j of any two samples x i and x j by s ij , i, j = 1 , ..., N .
Revised: Clean sample selection by balanced neighboring voting Let us denote the similarity between the representations f i and f j of any two samples x i and x j by s ij , i, j = 1 , ..., N .

Original: In our implementation we used the cosine similarity, that is, s ij (cid:44)
Revised: In our implementation we used the cosine similarity, that is, s ij (cid:44)

Original: [Equation]
Revised: [Equation]

Original: Let us also denote by N i the index set of the k nearest neighbors of sample
Revised: Let us also denote by N i the index set of the k nearest neighbors of sample

Original: x i in X based on the calculated similarity.
Revised: x i in X based on the calculated similarity.

Original: Then, for each sample x i , we calculate the normalized label distribution p i ∈
Revised: Then, for each sample x i , we calculate the balanced label distribution p i ∈

Original: R K in its neighborhood in the feature space, as the normalized sum of its neighbors’ labels.
Revised: R K in its neighborhood in the feature space, as the normalized sum of its neighbors’ labels.

Original: Note, that we used at each epoch t , we use the labels y tn
Revised: Note, that we used at each epoch t , we use the labels y tn

Original: (Eq. 6) that a relabeling mechanism provides.
Revised: (Eq. 2) that a relabeling mechanism provides.

Original: More speciﬁcally,
Revised: More speciﬁcally,

Original: [Equation]
Revised: [Equation]

Original: where p (cid:48) i = 1 k (cid:80) n ∈ N i y tn and π = (cid:80) Ni =1 p (cid:48){+=
 y+} i .
Revised: where p (cid:48) i = 1 k (cid:80) n ∈ N i y tn and π = (cid:80) Ni {+=
 y+} i .

Original: With a slight abuse of notation, we denote by π −the vector whose entries are the inverses of the entries of the vector π of the class probabilities in the whole dataset – in this way we compensate for global class imbalances.
Revised: With a slight abuse of notation, we denote by π −the vector whose entries are the inverses of the entries of the vector π of the class probabilities in the whole dataset – in this way we compensate for global class imbalances.

Original: Once the normalized distribution p i of the labels in the neighborhood of x i is estimated, we deﬁne a consistency measure
Revised: Once the balanced distribution p i of the labels in the neighborhood of x i is estimated, we deﬁne a consistency measure

Original: c i as
Revised: c i as

Original: [Equation]
Revised: [Equation]

Original: that is the ratio of the value of the distribution p
Revised: that is the ratio of the value of the distribution p

Original: i at the label l i divided by the value of its highest peak max l p i ( l ) .
Revised: i at the label l i divided by the value of its highest peak max l p i ( l ) .

Original: Roughly speaking, a high consistency measure c i at a sample i means that its neighbors agree with the given annotation
Revised: Roughly speaking, a high consistency measure c i at a sample i means that its neighbors agree with the given annotation

Original: l i – this indicates that l i is likely to be correct.
Revised: l i – this indicates that l i is likely to be correct.

Original: By setting a threshold θ s to c i , a clean subset ( X c , Y c ) can be extracted from the noisy dataset.
Revised: By setting a threshold θ s to c i , a clean subset ( X c , Y c ) can be extracted from the noisy dataset.

Original: Noisy sample relabelling by classiﬁer thresholding The normalized distribution p i of the labels of the neighbors of a sample i constructed by Eq.
Revised: Noisy sample relabelling by classiﬁer thresholding The balanced distribution p i of the labels of the neighbors of a sample i constructed by Eq.

Original: 3 is to some degree affected by the noisy labels of its neighbors.
Revised: 1 is to some degree affected by the noisy labels of its neighbors.

Original: One option is to use the computed normalized nearest neighbor distribution p i to relabel the noisy samples – for example, relabel the sample x i as arg max p
Revised: One option is to use the computed normalized nearest neighbor distribution p i to relabel the noisy samples – for example, relabel the sample x i as arg max p

Original: i (Ortego et al., 2021).
Revised: i (Ortego et al., 2021).

Original: However, this process will introduce self-conﬁrmation bias as we relabel and select samples by relying in both cases on the feature space.
Revised: However, this process will introduce self-conﬁrmation bias as we relabel and select samples by relying in both cases on the feature space.

Original: In this paper, we propose to decouple sample selection and sample relabeling.
Revised: In this paper, we propose to decouple sample selection and sample relabeling.

Original: More speciﬁcally, given that the classiﬁer is trained in subsequent iterations with relatively clean samples (selected by the mechanism described above), we propose to used its predictions to identify samples for which it has high conﬁdence in its prediction, and relabel them if the conﬁdent prediction does not agree with the annotated label.
Revised: More speciﬁcally, given that the classiﬁer is trained in subsequent iterations with relatively clean samples (selected by the mechanism described above), we propose to used its predictions to identify samples for which it has high conﬁdence in its prediction, and relabel them if the conﬁdent prediction does not agree with the annotated label.

Original: More speciﬁcally, let us denote by z ti (cid:44) g
Revised: More speciﬁcally, let us denote by z ti (cid:44) g

Original: ( f ( x i ))
Revised: ( f ( x i ))

Original: the prediction at epoch t for sample x i .
Revised: the prediction at epoch t for sample x i .

Original: By keeping track of the most recent L predictions, we calculate
Revised: By keeping track of the most recent L predictions, we calculate

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where q i is the average prediction for the sample x i .
Revised: where q i is the average prediction for the sample x i .

Original: We then modify all the labels l i , i = 1 , ..., N by thresholding q i at t -th epoch as:
Revised: We then modify all the labels l i , i = 1 , ..., N by thresholding q i at t -th epoch as:

Original: [Equation]
Revised: [Equation]

Original: Please note, that similarly to Section 3.1, we denote the one-hot label corresponding to l ti as y ti – this will be used in Eq.3.
Revised: Please note, that similarly to Section 3.1, we denote the one-hot label corresponding to l ti as y ti – this will be used in Eq.

Original: By setting a high θ r , a highly conﬁdent sample x i will be relabeled – this can in turn further enhance the quality of sample selection.
Revised: By setting a high θ r , a highly conﬁdent sample x i will be relabeled – this can in turn further enhance the quality of sample selection.

Original: Note, that we avoid mis-relabelling open-set noise samples as those tend not to have highly conﬁdent average predictions.
Revised: Note, that we avoid mis-relabelling open-set noise samples as those tend not to have highly conﬁdent average predictions.

Original: 3. S UPERVISED S ELF - SUPERVISED T RAINING
Revised: 3. S UPERVISED S ELF - SUPERVISED T RAINING

Revised: For training, we have both a supervised learning loss on the clean subset (cross entropy) and a selfsupervised learning loss (consistency).

Revised: The latter makes no assumption at all (implicit or explicit) about the labels and therefore can be applied on all samples.

Original: Mixup training of the encoder-classiﬁer using the clean subset With two random samples x 1 , y 1 and x 2 , y 2 in the clean subset ( X c , Y c ) , a mixed new sample x m , y m will be generated by Mixup method (Zhang et al., 2017) as:
Revised: Supervised mixup training of the encoder-classiﬁer using the clean subset With two random samples x 1 , y 1 and x 2 , y 2 in the clean subset ( X c , Y c ) , a mixed new sample x m , y m will be generated by Mixup method (Zhang et al., 2017) as:

Original: [Equation]
Revised: [Equation]

Original: We then apply the normal cross-entropy loss for the new virtual mixed sample: 
Revised: We then apply the normal cross-entropy loss for the new virtual mixed sample: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where z m =
Revised: where z m =

Original: Softmax ( g ( f ( x m ))) .
Revised: Softmax ( g ( f ( x m ))) .

Original: Instead of direct training with samples from the clean subset, we expect that virtual samples generated by Mixup are further away from the dataset samples thus can alleviate the noise memorization effect (Zhang et al., 2017).
Revised: Instead of direct training with samples from the clean subset, we expect that virtual samples generated by Mixup are further away from the dataset samples thus can alleviate the noise memorization effect (Zhang et al., 2017).

Original: This loss is back-propagated so as to update both the encoder f and the classiﬁcation head g .
Revised: This loss is back-propagated so as to update both the encoder f and the classiﬁcation head g .

Original: Self-consistency regularization for training  the encoder To fully utilize allthe samples
Revised: Self-supervised consistency training of the encoder using all samples

Original: , we applied a self-consistency loss motivated by recent non-contrastive self-supervised learning methods (Chen & He, 2021; Grill et al., 2020).
Revised: To fully utilize all the samples, we applied a self-consistency loss motivated by recent non-contrastive self-supervised learning methods (Chen & He, 2021; Grill et al., 2020).

Original: With a projector head p and prediction head q , we minimize the negative cosine similarity between two different augmented views from the same sample x i .
Revised: With a projector head p and prediction head q , we minimize the negative cosine similarity between two different augmented views from the same sample x i .

Original: Denoting the two output vectors as q i (cid:44) q ( p ( f ( x i 1 ))) and p i (cid:44) p ( f ( x i 2 )) :
Revised: Denoting the two output vectors as q i (cid:44) q ( p ( f ( x i 1 ))) and p i (cid:44) p ( f ( x i 2 )) :

Original: [Equation]
Revised: [Equation]

Original: where x i 1 , x i 2 denotes two different augmented views.
Revised: where x i 1 , x i 2 denotes two different augmented views.

Original: L sc bears similarity to the commonly used consistency regularization in semi-supervised learning methods, however, the consistency is enforced between the projected features rather than the predictions.
Revised: L sc bears similarity to the commonly used consistency regularization in semi-supervised learning methods, however, the consistency is enforced between the projected features rather than the predictions.

Original: This allows us to utilize open-set noise also for training whose true labels are not in the label set.
Revised: This allows us to utilize open-set noise also for training whose true labels are not in the label set.

Original: Also, we applied gradient stopping and an extra predictor so as to avoid mode collapse.
Revised: Also, we applied gradient stopping and an extra predictor so as to avoid mode collapse.

Original: This loss is back-propagated so as to update the encoder f and contributes to learning a strong feature space from both clean and noisy samples.
Revised: This loss is back-propagated so as to update the encoder f and contributes to learning a strong feature space from both clean and noisy samples.

Revised: We also investigated on the use of the L2 loss as distance metric – details can be found in Appendix D.

Original: Data augmentations Strong augmentations, such as (Cubuk et al., 2020; 2019), have shown to be effective in both supervised and semi-supervised learning (Berthelot et al., 2019;
Revised: Data augmentations Strong augmentations, such as Cubuk et al. (2020; 2019), have shown to be effective in both supervised and semi-supervised learning (Berthelot et al., 2019;

Original: Sohn et al., 2020) and recently, Nishi et al.
Revised: Sohn et al., 2020) and recently, Nishi et al.

Original: (2021) validated the beneﬁts of strong augmentations within the
Revised: (2021) validated the beneﬁts of strong augmentations within the

Original: DivideMix (Li et al., 2020a) framework.
Revised: DivideMix (Li et al., 2020a) framework.

Original: In this work, we deﬁne and use three types of augmentations: the original image itself (augmentation type ’none’) is used for testing, random cropping and horizontal ﬂipping (augmentation type ’weak’), and the augmentation policy proposed in Cubuk et al.
Revised: In this work, we deﬁne and use three types of augmentations: the original image itself (augmentation type ’none’) is used for testing, random cropping and horizontal ﬂipping (augmentation type ’weak’), and the augmentation policy proposed in Cubuk et al.

Original: (augmentation type ’strong’).
Revised: (augmentation type ’strong’).

Original: In the model training phase, by default, we apply ’strong’ augmentation for x i 1 , ’weak’ augmentation for x i 2 in L sc , and ’strong’ augmentation for x 1 , x 2 in
Revised: In the model training phase, by default, we apply ’strong’ augmentation for x i 1 , ’weak’ augmentation for x i 2 in L sc , and ’strong’ augmentation for x 1 , x 2 in

Original: L ce .
Revised: L ce .

Original: In the sample selection and relabelling phase, we apply ’weak’ augmentation so as to introduce more variance and alleviate the accumulation of error – this is in contrast to most works that do not apply augmentations at similar stages.
Revised: In the sample selection and relabelling phase, we apply ’weak’ augmentation so as to introduce more variance and alleviate the accumulation of error – this is in contrast to most works that do not apply augmentations at similar stages.

Original: Balanced sampler In order to address possible class imbalances in the noise-agnostic dataset, we normalized the local distribution of class labels, with the inverse of the global class probabilities vector π during sample selection (Eq. 3).
Revised: Balanced sampler In order to address possible class imbalances in the noise-agnostic dataset, we balanced the local distribution of class labels, with the inverse of the global class probabilities vector π during sample selection (Eq. 1).

Original: Similarly, the extracted clean subset ( X c , Y c ) might also potentially suffer from class imbalances.
Revised: Similarly, the extracted clean subset ( X c , Y c ) might also potentially suffer from class imbalances.

Original: To deal with this, we also use a balanced sampler for training with L ce by oversampling the minority class.
Revised: To deal with this, we also use a balanced sampler for training with L ce by oversampling the minority class.

Original: The overall training objective is to minimize a weighted sum of L ce and L sc .
Revised: The overall training objective is to minimize a weighted sum of L ce and L sc .

Original: [Equation]
Revised: [Equation]

Original: while for all experiments, we ﬁx w = 1 .
Revised: while for all experiments, we ﬁx w = 1 .

Original:  E XPERIMENTS 4. O VERVIEW
Revised:  E XPERIMENTS 4. O VERVIEW

Original: In this section, we conduct extensive experiments on two standard benchmarks with artiﬁcial label noise, CIFAR-10 and CIFAR-100, and two real-world datasets, WebVision and ANIMAL-10N.
Revised: In this section, we conduct extensive experiments on two standard benchmarks with artiﬁcial label noise, CIFAR-10 and CIFAR-100, and two real-world datasets, WebVision and ANIMAL-10N (see

Original: We begin by describing the datasets and some details of the implementations and the experimental settings.
Revised: Appendix A for details).

Original: In Section 4.2, we conducted extensive ablation experiments to show the robustness of our method w.r.t its hyperparameters with different noise types, noise ratios and dataset.
Revised: In Section 4.2.1, we conduct extensive ablation experiments to show the robustness of our method w.r.t its hyperparameters with different noise types, noise ratios and dataset.

Original: In Section 4.3, we conducted extensive ablation studies to validate the beneﬁts of different modules in our method.
Revised: In Section 4.2.2, we conduct extensive ablation studies to validate the beneﬁts of different modules in our method.

Original: In Section 4.4 and 4.5, we compared with the state-of-the-art in synthetic noisy datasets and real-world noisy datasets.

Original: Datasets Following the standard practice, for CIFAR-10 and CIFAR-100, we experimented with two types of artiﬁcial noise: symmetric noise by randomly replacing labels of all samples using a uniform distribution; and asymmetric noise by randomly exchanging labels of visually similar categories, such as Horse ↔ Deer and Dog ↔ Cat. WebVision (Li et al., 2017) is a large-scale dataset of 1000 classes of images crawled from the Web. Following previous work (Jiang et al., 2018; Li et al., 2020a; Ortego et al., 2021), we compare baseline methods on the top 50 classes from Google images Subset of WebVision.
Revised: In Section 4.2.2, we compared S3 with related works in each single stage.

Original: The noise ratio that is estimated to be around 20%. ANIMAL-10N (Song et al., 2019) is a smaller and recently proposed real-world dataset consists of 10 classes of animals, that are manually labeled withan error rate that is estimated to be approximately 8%. ANIMAL-10N has similar size characteristics to the CIFAR datasets, with 50000 train images and 10000 test images.
Revised: In Section 4.3 and 4.4, we compared with the state-of-the-art in synthetic noisy datasets and real-world noisy datasets.

Original: Implementation details We used a PresActResNet-18 (He et al., 2016) as the backbone for all CIFAR10/100 experiments following previous works. Unlike previous methods that use speciﬁc warmup settings for CIFAR10/CIFAR100, we train the model from scratch with a linear raising θ s from 0 to 1 in 20 epochs. θ r is ﬁxed as 0.8 and the prediction track length L is set to 10 for all
Revised: Implementation details can be found in Appendix B.

Original: CIFAR experiments except in the corresponding ablation part. We train all modules with the same SGD optimizer for 300 epochs with a momentum of 0.9 and a weight decay of 5e-4. The initial learning rate is 0.02 and is controlled by a cosine annealing scheduler by Pytorch.
Revised: 4. A BLATIONS STUDY

Original: The batchsize is ﬁxed as 128. We set α = 4 for all noise settings in Mixup training. For WebVision, we used a standard resnet18 following Ortego et al. (2021) due to  the hardware limitation.
Revised: In this section, we conduct extensive ablation experiments to show the robustness w.r.t.

Original: We train the network with SGD optimizer for 150 epochs with momentum of 0.9 and a weight decay of 1e-4. The initial learning rate is 0.02 and is controlled by a cosine annealing scheduler. The batchsize is ﬁxed as 64. For ANIMAL-10N, we used VGG-19 (Simonyan & Zisserman, 2014) with batch-normalization following (Song et al., 2019).
Revised: the few hyperparameters with different noise types, noise ratios and dataset.

Original: We train the network with SGD optimizer for 100 epochs with momentum of 0.9 and weight decay of 5e-4.

Original: The initial learning rate is 0.02 and and is also controlled by a cosine annealing scheduler. The batchsize is ﬁxed as 128. For both datasets, we train the model from scratch with a linear raising θ s from 0 to 1 in10 epochs, while θ r is ﬁxed as 0.9 and the prediction track length L is set to 10. We set α = 1 for Mixup.
Revised: In addition, to clearly show the contributions of different components and choices in S3 we perform experiments where we replace our training or sample selection/relabeling components with those from other methods in the literature.

Original: We report averages of at least two runs on a single Nvidia RTX 3090 GPU card.

Original: 4. H YPERPARAMETERS ROBUSTNESS
Revised: 4.2. A NALYSING S AMPLE SELECTION AND RELABELLING

Original: We aim at building a framework which is robust in noise-agnostic dataset scenario and with minimal number of hyperparameters.
Revised: We aim at building a framework which is robust in noise-agnostic dataset scenario and with minimal number of hyperparameters.

Original: In this section, we conducted extensive ablation experiments to show the robustness of the few hyperparameters with different noise types, noise ratios and dataset.
Revised: In this section, we conduct extensive ablation experiments to show the robustness of the few hyperparameters with different noise types, noise ratios and dataset.

Original: Relabelling quality VS relabellingproportion The choice of θ r and L controls the sample relabelling quality and proportion.
Revised: θ r and L in sample relabelling The choice of θ r and L controls the sample relabelling quality and proportion.

Original: Roughly speaking, the lower the θ r and L , the more samples will be relabeled in the training process, which also means that possibly more errors will be introduced.In
Revised: Roughly speaking, the lower the θ r and L , the more samples will be relabeled in the training process, which also means that possibly more errors will be introduced.

Original:  Fig. 2 we reported performance with different combinations of θ r and L on the synthetic CIFAR noisy dataset.
Revised: In Fig. 3(a) we reported performance with different combinations of θ r and L on the synthetic CIFAR10 noisy dataset.

Original: Genrally, our method achieved superior performance than the state-of-the-art with different θ r and L .
Revised: Genrally, our method achieved superior performance than the state-of-the-art with different θ r and L .

Original: For example in CIFAR10 dataset with 90% sym noise, the lowest accuracy is 87.79% – this surpasses the state-of-the-art by ∼ 7%.
Revised: For example in CIFAR10 dataset with 90% sym noise, the lowest accuracy is 87.79% – this surpasses the state-of-the-art by ∼ 7%.

Original: Robustness w.r.t k  in sample selection The number k controlled the neighborhood size in the sample selection phase.
Revised: Robustness w.r.t k and θ s in sample selection The number k controls the neighborhood size in the sample selection phase.

Original: In Fig. 2, we report results with different k for the CIFAR10 dataset with 40% asym noise.
Revised: In Fig. 3(b), we report results with different k for the CIFAR10 dataset with 40% asym noise.

Original: Except for too small k which is more sensitive to the noisy samples, the performance is stable and consistently higher than the state-of-the-art.
Revised: Except for too small k which is more sensitive to the noisy samples, the performance is stable and consistently higher than the state-of-the-art.

Original: 4. A BLATIONS STUDY
Revised: θ s controls the number of selected samples, with θ s = 0 corresponding to no sample selection (i.e., all samples considered as clean subset). In Fig. 3(c), we report results with θ s = [0 , 0 . 8 , 1 . 0] .

Original: In this section, we conducted extensive ablation studies to validate the beneﬁts of different modules.
Revised: Removing sample selection leads to severe degradation especially in high noise ratio (90% symmetric noise), while a relatively high θ s giving consistently high performance. We ﬁxed θ s = 1 for experiments.

Original: Effect of augmentations Table.
Revised: Peformance of sample selection and relabelling At each iteration t , after relabelling and before sample selection, S3 creates a labeled set in which one can ﬁnd correctly labelled samples (let’s denote this subset by A t ), wrongly labeled samples that could be labeled with one of the given labels (subset B t ) and the out of set noise (subset C t ). Note, the correspondence with the sets A, B and C in the original labeled set depicted in Fig.1. After the sample selection, some will be selected and some not. Ideally, the relabelling mechanism will relabel all samples correctly (i.e., A t =

Original: 1 shows that with stronger augmentations in  themodel training stage we can achieve consistent better performance at different noise ratio/noise type.
Revised: A + B ) and, at every iteration, the selection mechanism will select only the correctly labeled samples (i.e., those in A t ). In order to show the performance of the selection mechanism, we report in Fig. 4(a)

Original: For comparison, we also refer to  the state-of-the-art result when strong augmentation applied with DivideMix (Nishi et al., 2021).
Revised: the F-score for the selection. In order to show the performance of the relabeling mechanism, we show in Fig. 4(b) the ratio of the correctly labeled samples (i.e., A t / ( A + B + C ) . It is clear that the relabelling mechanism succeeds in increasing the size of correctly labeled samples ( A t ) even in the case of heavy or out-of-set noise.

Original: It is  clear thatwe are much better in all augmentation conditions despite the simplicity of our method.
Revised: It is also clear that in all cases the sample selection mechanism achieves high scores, even in the challenging cases of heavy noise.

Original: Please note that we don’t apply cotraining. [Table]
Revised: 4.2. T RAINING MODULES ABLATIONS

Original: Effect of self-consistencyregularization and Mixup In Table.2 we report the effect of selfcontrastive regularization and Mixup.
Revised: Effects of self-consistency and Mixup In Table 1 we report the effect of self-consistency regularization and Mixup in S3 training stage.

Original: Removing Mixup decreases the performance, especially in high noise ratio and removing self-consistency also lead to degradation.
Revised: Removing Mixup decreases the performance, especially in high noise ratio and removing self-consistency also lead to degradation.

Original: Both help prevent memorization of wrong selections and to explore all samples so as to improve the model robustness.
Revised: Both help prevent memorization of wrong selections and to explore all samples so as to improve the model robustness.

Original:  Effect of balancing strategies To alleviate possible dataset class imbalance, we proposed twodataset balancing strategies in sample selection and model training phase, respectively.
Revised: [Table] [Table] Effect of balancing strategies To alleviate possible dataset class imbalance, we propose two balancing strategies in sample selection and model training phase, respectively.

Original: In Table.we investigate its effect with  controlledasymmetric noise in CIFAR10, a type of noise that is known to create class imbalanced dataset.
Revised: In Table 2 we investigate its effect with different controlled noise in CIFAR10 and also a well-known real-world imbalanced noisy dataset Clothing1M.

Original: [Table]
Revised: To demonstrate the contributions of our choices, we substitute components of our method, with components of two state of the art methods, i.e., MOIT and DivideMix, and more speciﬁcally, their sample selection and training stages. The results are summarised in Table 3. It is clear that the semi-supervised training scheme that MOIT adopts, in which all samples are (implicitly) relabelled, and the DivideMix sample selection that is based on the classiﬁer’s prediction, perform signiﬁcantly worse than ours, especially in the case of high noise ratios.

Original: [Table]
Revised: [Table]

Original: 4. S YNTHETIC NOISY DATASETS EVALUATION
Revised: 4. S YNTHETIC NOISY DATASETS EVALUATION

Original: In this section, we compared our method to several state-of-the-art methods: DivideMix Li et al. (2020a), LossModelling Arazo et al.
Revised: In this section, we compared our method to most recent state-of-the-art methods: DivideMix (Li et al., 2020a), LossModelling (Arazo et al., 2019), Coteaching+

Original: Coteaching+ Yu et al. Mixup Zhang et al. (2017), F-correction Patrini et al. SELFIE Song et al.
Revised: (Yu et al., 2019), Mixup (Zhang et al., 2017), F-correction (Patrini et al., 2017), SELFIE (Song et al., 2019), PLC (Zhang et al., 2021), PENCIL (Yi & Wu, 2019), ELR (Liu et al., 2020a), NCT (Chen et al., 2021), MOIT+ (Ortego et al., 2021),

Original: , PLC Zhang et al. PENCIL Yi & Wu (2019), ELR Liu et al. NCT (Chen et al., 2021), MOIT+ (Ortego et al., 2021), NGC (Wu et al., 2021), ProtoMix (Li et al., 2020b).
Revised: NGC (Wu et al., 2021), RRL (Li et al., 2020b), FaMUS (Xu et al., 2021), GJS (Ghosh & Lan, 2021), PDLC (Liu et al., 2020b).

Original: We show, that the proposed method achieves consistent improvements in all datasets and at all noise types and ratios.
Revised: We show, that the proposed method achieves consistent improvements in all datasets and at all noise types and ratios

Original: [Table]
Revised: [Table]

Original: Evaluation with controlled closed-set noise We compared to the most competitive works recently.
Revised: Evaluation with controlled closed-set noise In this section we compare S3 to the most competitive recent works.

Original: Please note that several of the methods require a per-dataset ﬁnetuning of the hyperparameters due to complicated structures and extra modules like model cotraining. Here we report their bestresults. Table 4 shows results on CIFAR10 andCIFAR100, where it is clear that our method performs clearly better without the use of the additional cotraining techniques.
Revised: Table 4 shows results on CIFAR10 and CIFAR100 – we note for S3 this is without the use of co-training such as in C2D (Zheltonozhskii et al., 2021) and without per-dataset ﬁnetuning of the hyperparameters as is done in several other methods. Here we report the best results of other methods and the results of S3 with ﬁxed hyperparameters.

Original: Especially at high noise ratio(90% sym noise), S3 can achieve ∼ 95% on CIFAR10 and ∼ 57% on CIFAR100, surpassing the state-of-the-art by 14.5% and 23.4%, respectively.

Original: Meanwhile, unlike several previous methods work better in  symmetric noise, our method works very well also at the more realistic asymmetric synthetic noise.This indicates that our method is more robust to real-world noise.
Revised: It is clear that our methods outperforms them, not only in the case of symmetric noise, but also that it works very well also at the more realistic asymmetric synthetic noise.

Original: Evaluation with combined open-set noise and closed-set noise Tab. 5 shows the performance of our method in a more complex combined noise scenario.
Revised: Evaluation with combined open-set noise and closed-set noise Table 5 shows the performance of our method in a more complex combined noise scenario.

Revised: The closed-set noise are generated as symmetric noise while the open-set noise are random samples from CIFAR100.

Revised: Noise ratio denotes the total noise ratio while the open ratio denotes the proportion of open-set noise.

Original: Previous methods that are specially designed for open-set noise(Lee et al., 2019; Wang et al., 2018) which degrade rapidly when the openset noise ratio is decreased from 1 to 0.5.
Revised: Previous methods that are specially designed for open-set noise degrade rapidly when the open-set noise ratio [Table] [Table] 2021) and NGC (Wu et al., 2021).

Revised: is decreased from 1 to 0.5 (Lee et al., 2019; Wang et al., 2018).

Original: The performance of method without considering open-set noise like Li et al. (2020a) will decrease when the open-set noise ratio is increased.
Revised: The performance of method without considering open-set noise like DivideMix (Li et al., 2020a) will decrease when the open-set noise ratio is increased.

Original: Sachdeva et al. (2021) modiﬁed the method of Li et al. (2020a) to deal with combined noise, however report results that are considerably lower than ours.[Table]
Revised: EDM (Sachdeva et al., 2021) modiﬁed the method of DivideMix to deal with combined noise, however report results that are considerably lower than ours.

Original: 4. R EAL - WORLD NOISY DATASETS EVALUATION
Revised: 4. R EAL - WORLD NOISY DATASETS EVALUATION

Original: [Table]

Original: Finally [Table] in Table. 6 andTable.
Revised: Finally, in Table 6, Table 7 and Table 8 we show results on the WebVision, Clothing1M and

Original: 7 we show results on the WebVision and ANIMAL-10N datasets respectively.
Revised: ANIMAL-10N datasets, respectively.

Original: To summarize, our method achieves better  performance compared to the current stateof-the-art in both large-scale web-crawled dataset and small-scale human annotated noisy dataset. 
Revised: To summarize, our method achieves better or comparable performance in relation to the current state-of-the-art in both large-scale web-crawled dataset and small-scale human annotated noisy dataset. [Table]

Original: [Table]
Revised: [Table]

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper we proposed a method for learning with noisy labels, that relies on a sample selection mechanism, a relabeling mechanism and a training strategy with multi-objective losses that enable us to learn robust features from both noisy and clean samples, and a classiﬁer from only clean or robustly relabeled ones.
Revised: In this paper we propose a method for learning with noisy labels, that relies on a sample selection mechanism, a relabeling mechanism and a training strategy with multi-objective losses that enable us to learn robust features from both noisy and clean samples, and a classiﬁer from only clean or robustly relabeled ones.

Original: The proposed method is a simple framework, does not utilize complicated mechanisms such as co-training to deal with self-conﬁrmation bias, and is shown with extensive experiments and ablation studies to be robust to the values of its few hyper-parameters, and to consistently and by large surpass the state-of-the art in both open-set and close-set noise. 
Revised: The proposed method is a simple framework, does not utilize complicated mechanisms such as co-training to deal with self-conﬁrmation bias, and is shown with extensive experiments and ablation studies to be robust to the values of its few hyper-parameters, and to consistently and by large surpass the state-of-the art in both open-set and close-set noise. A D ATASET DETAILS CIFAR10 and CIFAR100 both consist of 50K images.

Revised: Following the standard practice, for CIFAR10 and CIFAR-100, we experimented with two types of artiﬁcial noise: symmetric noise by randomly replacing labels of all samples using a uniform distribution; and asymmetric noise by randomly exchanging labels of visually similar categories, such as Horse ↔ Deer and Dog ↔ Cat.

Revised: WebVision (Li et al., 2017) is a large-scale dataset of 1000 classes of images crawled from the

Revised: Web.

Revised: Following previous work (Jiang et al., 2018; Li et al., 2020a; Ortego et al., 2021), we compare baseline methods on the top 50 classes from Google images Subset of WebVision.

Revised: The noise ratio that is estimated to be around 20%.

Revised: ANIMAL-10N (Song et al., 2019) is a smaller and recently proposed real-world dataset consists of 10 classes of animals, that are manually labeled with an error rate that is estimated to be approximately 8%.

Revised: ANIMAL-10N has similar size characteristics to the

Revised: CIFAR datasets, with 50000 train images and 10000 test images.

Revised: B E XPERIMENT D ETAILS

Revised: We used a PresActResNet-18 (He et al., 2016) as the backbone for all CIFAR10/100 experiments following previous works.

Revised: Unlike previous methods that use speciﬁc warmup settings for CIFAR10/CIFAR100, we train the model from scratch with a linear raising θ s from 0 to 1 in 20 epochs.

Revised: θ r is ﬁxed as 0.8 and the prediction track length L is set to 10 for all CIFAR experiments except in the corresponding ablation part.

Revised: We train all modules with the same SGD optimizer for 300 epochs with a momentum of 0.9 and a weight decay of 5e-4.

Revised: The initial learning rate is 0.02 and is controlled by a cosine annealing scheduler by Pytorch.

Revised: The batchsize is ﬁxed as 128.

Revised: We set α = 4 for all noise settings in Mixup training.

Revised: For WebVision, we used a standard ResNet18 following MOIT (Ortego et al., 2021) due to the hardware limitation.

Revised: We train the network with SGD optimizer for 150 epochs with momentum of 0.9 and a weight decay of 1e-4.

Revised: The initial learning rate is 0.02 and is controlled by a cosine annealing scheduler.

Revised: The batchsize is ﬁxed as 64.

Revised: For Clothing1M, we used ResNet50 following

Revised: DivideMix (Li et al., 2020a) with ImageNet pretrained weights.

Revised: We train the network with SGD optimizer for 80 epochs with momentum of 0.9 and weight decay of 1e-3.

Revised: The initial learning rate is 0.02 and and is reduced to 0.002 after 40 epochs.

Revised: For ANIMAL-10N, we used VGG-19 (Simonyan & Zisserman, 2014) with batch-normalization following (Song et al., 2019).

Revised: We train the network with SGD optimizer for 100 epochs with momentum of 0.9 and weight decay of 5e-4.

Revised: The initial learning rate is 0.02 and and is also controlled by a cosine annealing scheduler.

Revised: The batchsize is ﬁxed as 128.

Revised: For all datasets, we train the model from scratch with θ s = 1 , while θ r is ﬁxed as 0.

Revised: and the prediction track length L is set to 1.

Revised: We set α = 1 for Mixup.

Revised: We report averages of at least two runs on a single Nvidia RTX 3090 GPU card.

Revised: C E FFECT OF AUGMENTATIONS

Revised: In Table 9, we show the effect of different types of augmentations at the different stages of our method.

Revised: Our results are consistent with the ﬁndings of AugDesc (Nishi et al., 2021) that one should use weaker augmentations for sample selection and relabeling, and strong augmentations for training.

Revised: We can also see that the higher the noise ratio, the more prominent the differences are with different augmentation strategies.

Revised: [Table]

Revised: D D IFFERENT DISTANCE METRICS IN SELF - CONSISTENCY

Revised: In S3, we applied negative cosine similarity as distance metric for self-consistency head.

Revised: Here we also experimented with L2 distance.

Revised: The results are shown in Table

Revised: [Table]

Revised: where it can be seen that there are small differences

Revised: [Table]

Revised: but that the cosine similarity is in general better.

Revised: [Table]