Original: SELF: L EARNING TO F ILTER N OISY L ABELS WITH S ELF -E NSEMBLING
Revised: SELF: L EARNING TO F ILTER N OISY L ABELS WITH S ELF -E NSEMBLING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep neural networks (DNNs) have been shown to over-ﬁt a dataset when being trained with noisy labels for a long enough time.
Revised: Deep neural networks (DNNs) have been shown to over-ﬁt a dataset when being trained with noisy labels for a long enough time.

Original: To overcome this problem, we present a simple and effective method self-ensemble label ﬁltering (SELF) to progressively ﬁlter out the wrong labels during training.
Revised: To overcome this problem, we present a simple and effective method self-ensemble label ﬁltering (SELF) to progressively ﬁlter out the wrong labels during training.

Original: Our method improves the task performance by gradually allowing supervision only from the potentially non-noisy (clean) labels and stops learning on the ﬁltered noisy labels.
Revised: Our method improves the task performance by gradually allowing supervision only from the potentially non-noisy (clean) labels and stops learning on the ﬁltered noisy labels.

Original: For the ﬁltering, we form running averages of predictions over the entire training dataset using the network output at different training epochs.
Revised: For the ﬁltering, we form running averages of predictions over the entire training dataset using the network output at different training epochs.

Original: We show that these ensemble estimates yield more accurate identiﬁcation of inconsistent predictions throughout training than the single estimates of the network at the most recent training epoch.
Revised: We show that these ensemble estimates yield more accurate identiﬁcation of inconsistent predictions throughout training than the single estimates of the network at the most recent training epoch.

Original: While ﬁltered samples are removed entirely from the supervised training loss, we dynamically leverage them via semi-supervised learning in the unsupervised loss.
Revised: While ﬁltered samples are removed entirely from the supervised training loss, we dynamically leverage them via semi-supervised learning in the unsupervised loss.

Original: We demonstrate the positive effect of such an approach on various image classiﬁcation tasks under both symmetric and asymmetric label noise and at different noise ratios.
Revised: We demonstrate the positive effect of such an approach on various image classiﬁcation tasks under both symmetric and asymmetric label noise and at different noise ratios.

Original: It substantially outperforms all previous works on noise-aware learning across different datasets and can be applied to a broad set of network architectures.
Revised: It substantially outperforms all previous works on noise-aware learning across different datasets and can be applied to a broad set of network architectures.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The acquisition of large quantities of a high-quality human annotation is a frequent bottleneck in applying DNNs.
Revised: The acquisition of large quantities of a high-quality human annotation is a frequent bottleneck in applying DNNs.

Original: There are two cheap but imperfect alternatives to collect annotation at large scale: crowdsourcing from non-experts and web annotations, particularly for image data where the tags and online query keywords are treated as valid labels.
Revised: There are two cheap but imperfect alternatives to collect annotation at large scale: crowdsourcing from non-experts and web annotations, particularly for image data where the tags and online query keywords are treated as valid labels.

Original: Both these alternatives typically introduce noisy (wrong) labels.
Revised: Both these alternatives typically introduce noisy (wrong) labels.

Original: While Rolnick et al.
Revised: While Rolnick et al.

Original: (2017) empirically demonstrated that DNNs can be surprisingly robust to label noise under certain conditions, Zhang et al.
Revised: (2017) empirically demonstrated that DNNs can be surprisingly robust to label noise under certain conditions, Zhang et al.

Original: (2017) has shown that DNNs have the capacity to memorize the data and will do so eventually when being confronted with too many noisy labels.
Revised: (2017) has shown that DNNs have the capacity to memorize the data and will do so eventually when being confronted with too many noisy labels.

Original: Consequently, training DNNs with traditional learning procedures on noisy data strongly deteriorates their ability to generalize – a severe problem.
Revised: Consequently, training DNNs with traditional learning procedures on noisy data strongly deteriorates their ability to generalize – a severe problem.

Original: Hence, limiting the inﬂuence of label noise is of great practical importance.
Revised: Hence, limiting the inﬂuence of label noise is of great practical importance.

Original: A common approach to mitigate the negative inﬂuence of noisy labels is to eliminate them from the training data and train deep learning models just with the clean labels (Fr´enay & Verleysen, 2013).
Revised: A common approach to mitigate the negative inﬂuence of noisy labels is to eliminate them from the training data and train deep learning models just with the clean labels (Fr´enay & Verleysen, 2013).

Original: Employing semi-supervised learning can even counteract the noisy labels (Laine & Aila, 2016; Luo et al., 2018).
Revised: Employing semi-supervised learning can even counteract the noisy labels (Laine & Aila, 2016; Luo et al., 2018).

Original: However, the decision which labels are noisy and which are not is decisive for learning robust models.
Revised: However, the decision which labels are noisy and which are not is decisive for learning robust models.

Original: Otherwise, unﬁltered noisy labels still inﬂuence the (supervised) loss and affect the task performance as in these previous works.
Revised: Otherwise, unﬁltered noisy labels still inﬂuence the (supervised) loss and affect the task performance as in these previous works.

Original: They use the entire label set to compute the loss and severely lack a mechanism to identify and ﬁlter out the erroneous labels from the labels set.
Revised: They use the entire label set to compute the loss and severely lack a mechanism to identify and ﬁlter out the erroneous labels from the labels set.

Original: In this paper, we propose a self-ensemble label ﬁltering (SELF) framework that identiﬁes potentially noisy labels during training and keeps the network from receiving supervision from the ﬁltered noisy labels.
Revised: In this paper, we propose a self-ensemble label ﬁltering (SELF) framework that identiﬁes potentially noisy labels during training and keeps the network from receiving supervision from the ﬁltered noisy labels.

Original: This allows DNNs to gradually focus on learning from undoubtedly correct samples even with an extreme level of noise in the labels (e.g., 80% noise ratio) and leads to improved performance as the supervision become less noisy.
Revised: This allows DNNs to gradually focus on learning from undoubtedly correct samples even with an extreme level of noise in the labels (e.g., 80% noise ratio) and leads to improved performance as the supervision become less noisy.

Original: The key contribution of our work is progressive ﬁltering, i.e.,leverage the knowledge provided in the network’s output over different training iterations to form a consensus of predictions (self-ensemble predictions) to progressively identify and ﬁlter out the noisy labels from the labeled data.
Revised: The key contribution of our work is progressive ﬁltering, i.e.,

Original: [Figure]
Revised: [Figure]

Revised: leverage the knowledge provided in the network’s output over different training iterations to form a consensus of predictions (self-ensemble predictions) to progressively identify and ﬁlter out the noisy labels from the labeled data.

Original: When learning under label noise, the network receives noisy updates and hence ﬂuctuates strongly.
Revised: When learning under label noise, the network receives noisy updates and hence ﬂuctuates strongly.

Original: Such conduct of training would impede to learn stable neural representations and further mislead the consensus of the predictions.
Revised: Such conduct of training would impede to learn stable neural representations and further mislead the consensus of the predictions.

Original: Therefore, it is essential to incorporate a model with stable training behavior to obtain better estimates from the consensus.
Revised: Therefore, it is essential to incorporate a model with stable training behavior to obtain better estimates from the consensus.

Original: Concretely, we employ the semi-supervised technique as a backbone to our framework to stabilize the learning process of the model.
Revised: Concretely, we employ the semi-supervised technique as a backbone to our framework to stabilize the learning process of the model.

Original: Correctly, we maintain the running average model, such as proposed by Tarvainen & Valpola (2017), a.k.a.
Revised: Correctly, we maintain the running average model, such as proposed by Tarvainen & Valpola (2017), a.k.a.

Original: the Mean-Teacher model.
Revised: the Mean-Teacher model.

Original: This model ensemble learning provides a more stable supervisory signal than the noisy model snapshots and provides a stable ground for progressive ﬁltering to ﬁlter out potential noisy labels.
Revised: This model ensemble learning provides a more stable supervisory signal than the noisy model snapshots and provides a stable ground for progressive ﬁltering to ﬁlter out potential noisy labels.

Original: Note that this is different from just a mere combination of semi-supervised techniques with a noisy label ﬁltering method.
Revised: Note that this is different from just a mere combination of semi-supervised techniques with a noisy label ﬁltering method.

Original: We call our approach self-ensemble label ﬁltering (SELF) - that establishes model ensemble learning as a backbone to form a solid consensus of the self-ensemble predictions to ﬁlter out the noisy labels progressively.
Revised: We call our approach self-ensemble label ﬁltering (SELF) - that establishes model ensemble learning as a backbone to form a solid consensus of the self-ensemble predictions to ﬁlter out the noisy labels progressively.

Original: Our framework allows to compute supervised loss on cleaner subsets rather than the entire noisy labeled data as in previous works.
Revised: Our framework allows to compute supervised loss on cleaner subsets rather than the entire noisy labeled data as in previous works.

Original: It further leverages the entire dataset, including the ﬁltered out erroneous samples in the unsupervised loss.
Revised: It further leverages the entire dataset, including the ﬁltered out erroneous samples in the unsupervised loss.

Original: To best of our knowledge, we are the ﬁrst to identify and propose self-ensemble as a principled technique against learning under noisy labels.
Revised: To best of our knowledge, we are the ﬁrst to identify and propose self-ensemble as a principled technique against learning under noisy labels.

Original: Our motivation stems from the observation that DNNs start to learn from easy samples in initial phases and gradually adapt to hard ones during training.
Revised: Our motivation stems from the observation that DNNs start to learn from easy samples in initial phases and gradually adapt to hard ones during training.

Original: When trained on wrongly labeled data,
Revised: When trained on wrongly labeled data,

Original: DNNs learn from clean labels at ease and receive inconsistent error signals from the noisy labels before over-ﬁtting to the dataset.
Revised: DNNs learn from clean labels at ease and receive inconsistent error signals from the noisy labels before over-ﬁtting to the dataset.

Original: The network’s prediction is likely to be consistent on clean samples and inconsistent or oscillates strongly on wrongly labeled samples over different training iterations.
Revised: The network’s prediction is likely to be consistent on clean samples and inconsistent or oscillates strongly on wrongly labeled samples over different training iterations.

Original: Based on this observation, we record the outputs of a single network made on different training epochs and treat them as an ensemble of predictions obtained from different individual networks.
Revised: Based on this observation, we record the outputs of a single network made on different training epochs and treat them as an ensemble of predictions obtained from different individual networks.

Original: We call these ensembles that are evolved from a single network self-ensemble predictions .
Revised: We call these ensembles that are evolved from a single network self-ensemble predictions .

Original: Subsequently, we identify the correctly labeled samples via the agreement between the provided label set and our running average of self-ensemble predictions .
Revised: Subsequently, we identify the correctly labeled samples via the agreement between the provided label set and our running average of self-ensemble predictions .

Original: The samples of ensemble predictions that agree with the provided labels are likely to be consistent and treated as clean samples.
Revised: The samples of ensemble predictions that agree with the provided labels are likely to be consistent and treated as clean samples.

Original: In summary, our SELF framework stabilizes the training process and improves the generalization ability of DNNs.
Revised: In summary, our SELF framework stabilizes the training process and improves the generalization ability of DNNs.

Original: We evaluate the proposed technique on image classiﬁcation tasks using CIFAR10, CIFAR100 & ImageNet.
Revised: We evaluate the proposed technique on image classiﬁcation tasks using CIFAR10, CIFAR100 & ImageNet.

Original: We demonstrate that SELF consistently outperforms the existing approaches on asymmetric and symmetric noise at all noise levels, as shown in Fig.
Revised: We demonstrate that SELF consistently outperforms the existing approaches on asymmetric and symmetric noise at all noise levels, as shown in Fig.

Original: Besides,
Revised: Besides,

Original: SELF remains robust towards the choice of the network architecture.
Revised: SELF remains robust towards the choice of the network architecture.

Original: Our work is transferable to other tasks without the need to modify the architecture or the primary learning objective.
Revised: Our work is transferable to other tasks without the need to modify the architecture or the primary learning objective.

Revised: Once the best model is found, these predictions identify and ﬁlter out noisy labels using the original label set L 0 .

Revised: The model performs this progressive ﬁltering until there is no more better model.

Revised: For details see Algorithm 1.

Original:  S ELF - ENSEMBLE LABEL FILTERING 2. O VERVIEW
Revised:  S ELF - ENSEMBLE LABEL FILTERING 2. O VERVIEW

Original: Fig.
Revised: Fig.

Original: 2 shows an overview of our proposed approach.
Revised: 2 shows an overview of our proposed approach.

Original: In the beginning, we assume that the labels of the training set are noisy.
Revised: In the beginning, we assume that the labels of the training set are noisy.

Original: The model attempts to identify correct labels progressively using self- forming ensembles of models and predictions.
Revised: The model attempts to identify correct labels progressively using selfforming ensembles of models and predictions.

Original: Since wrong labels cause strong ﬂuctuations in the model’s predictions, using ensembles is a natural way to counteract noisy labels.
Revised: Since wrong labels cause strong ﬂuctuations in the model’s predictions, using ensembles is a natural way to counteract noisy labels.

Original: Concretely, in each iteration, the model learns from a detected set of potentially correct labels and maintains a running average of model snapshots (realized by the Mean Teacher model Tarvainen &
Revised: Concretely, in each iteration, the model learns from a detected set of potentially correct labels and maintains a running average of model snapshots (realized by the Mean Teacher model Tarvainen &

Original: Valpola (2017)).
Revised: Valpola (2017)).

Original: This ensemble model is evaluated on the entire dataset and provides an additional learning signal for training the single models.
Revised: This ensemble model is evaluated on the entire dataset and provides an additional learning signal for training the single models.

Original: Additionally, our framework maintains the runningaverage of the model’s predictions for the ﬁltering process.
Revised: Additionally, our framework maintains the runningaverage of the model’s predictions for the ﬁltering process.

Original: The model is trained until we ﬁnd the best model w.r.t.
Revised: The model is trained until we ﬁnd the best model w.r.t.

Original: the performance on the validation set (e.g., by early-stopping).
Revised: the performance on the validation set (e.g., by early-stopping).

Original: The set of correct labels is detected based on the strategy deﬁned in Sec.
Revised: The set of correct labels is detected based on the strategy deﬁned in Sec.

Original: In the next iteration, we again use all data and the new ﬁltered label set as input for the model training.
Revised: In the next iteration, we again use all data and the new ﬁltered label set as input for the model training.

Original: The iterative training procedure stops when no better model can be found.
Revised: The iterative training procedure stops when no better model can be found.

Original: In the following, we give more details about the combination of this training and ﬁltering procedure.
Revised: In the following, we give more details about the combination of this training and ﬁltering procedure.

Original: 2. P ROGRESSIVE LABEL FILTERING
Revised: 2. P ROGRESSIVE LABEL FILTERING

Original: Progressive detection of correctly labeled samples Our framework
Revised: Progressive detection of correctly labeled samples Our framework

Original: Self-Ensemble Label Filtering (Algorithm 1) focuses on the detection of certainly correct labels from the provided label set L 0 .
Revised: Self-Ensemble Label Filtering (Algorithm 1) focuses on the detection of certainly correct labels from the provided label set L 0 .

Original: In each iteration i , the model is trained using the label set of potentially correct labels L i .
Revised: In each iteration i , the model is trained using the label set of potentially correct labels L i .

Original: At the end of each iteration, the model determines the next correct label set L i +1 using the ﬁltering strategy described in 2.2 The model stops learning when no improvement was achieved after training on the reﬁned label set L
Revised: At the end of each iteration, the model determines the next correct label set L i +1 using the ﬁltering strategy described in 2.2 The model stops learning when no improvement was achieved after training on the reﬁned label set L

Original: i +1 .
Revised: i +1 .

Original: In other words, in each iteration, the model attempts to learn from the easy, in some sense, obviously correct labels.
Revised: In other words, in each iteration, the model attempts to learn from the easy, in some sense, obviously correct labels.

Original: However, learning from easy samples also affects similar but harder samples from the same classes.
Revised: However, learning from easy samples also affects similar but harder samples from the same classes.

Original: Therefore, by learning from these easy samples, the network can gradually distinguish between hard and wrongly-labeled samples.
Revised: Therefore, by learning from these easy samples, the network can gradually distinguish between hard and wrongly-labeled samples.

Original: Our framework does not focus on repairing all noisy labels.

Original: Although the detection of wrong labels is sometimes easy, ﬁnding their correct hidden label might be extremely challenging in case of having many classes.

Original: If the noise is sufﬁciently random, the set of correct labels will be representative to achieve high model performance.

Original: Further, in our framework, the label ﬁltering is performed on the original label set L 0 from iteration 0 .

Original: Clean labels erroneously removed in an earlier iteration (e.g., labels of hard to classify samples) can be reconsidered for model training again in later iterations.

Original: Require: D train = noisy labeled training set Require: D val =
Revised: Require: D train = noisy labeled training set Require: D val =

Original: noisy labeled validation set
Revised: noisy labeled validation set

Original: Require: ( x, y )
Revised: Require: ( x, y )

Original: = training stimuli and label
Revised: = training stimuli and label

Original: Require: α = ensembling momentum, 0 ≤ α ≤i ←(cid:46) counter to track iterations M i ← train ( D train , D val )
Revised: Require: α = ensembling momentum, 0 ≤ α ≤i ←(cid:46) counter to track iterations M i ← train ( D train , D val )

Original: (cid:46) initial Mean-Teacher ensemble model training M best ← M i (cid:46) set initial model as best model z
Revised: (cid:46) initial Mean-Teacher ensemble model training M best ← M i (cid:46) set initial model as best model z

Original: i ←(cid:46) initialize ensemble predictions of all samples (ignored sample index for simplicity) while acc ( M i , D val ) ≥ acc ( M best , D val ) do (cid:46) iterate until no best model is found on D val
Revised: i ←(cid:46) initialize ensemble predictions of all samples (ignored sample index for simplicity) while acc ( M i , D val ) ≥ acc ( M best , D val ) do (cid:46) iterate until no best model is found on D val

Original: M best ← M i
Revised: M best ← M i

Original: (cid:46) save the best model D filter ← D train (cid:46) set ﬁltered dataset as initial label set i ← i +for ( x, y ) in D filter do ˆ
Revised: (cid:46) save the best model D filter ← D train (cid:46) set ﬁltered dataset as initial label set i ← i +for ( x, y ) in D filter do ˆ

Original: z i ← M best ( x ) (cid:46) evaluate model output ˆ
Revised: z i ← M best ( x ) (cid:46) evaluate model output ˆ

Original: z i
Revised: z i

Original: z
Revised: z

Original: i ← αz
Revised: i ← αz

Original: i − 1 + (1 − α )
Revised: i − 1 + (1 − α )

Original: ˆ z i (cid:46) accumulate ensemble predictions
Revised: ˆ z i (cid:46) accumulate ensemble predictions

Original: z i if y (cid:54) =
Revised: z i if y (cid:54) =

Original: argmax ( z i ) then (cid:46) verify agreement of ensemble predictions & label y ← ∅ in D filter (cid:46) identify it as noisy label & remove from label set end if end for M i ← train ( D filter , D val )
Revised: argmax ( z i ) then (cid:46) verify agreement of ensemble predictions & label y ← ∅ in D filter (cid:46) identify it as noisy label & remove from label set end if end for M i ← train ( D filter , D val )

Original: (cid:46) train Mean-Teacher model on ﬁltered label set end while return M best
Revised: (cid:46) train Mean-Teacher model on ﬁltered label set end while return M best

Revised: Our framework does not focus on repairing all noisy labels.

Revised: Although the detection of wrong labels is sometimes easy, ﬁnding their correct hidden label might be extremely challenging in case of having many classes.

Revised: If the noise is sufﬁciently random, the set of correct labels will be representative to achieve high model performance.

Revised: Further, in our framework, the label ﬁltering is performed on the original label set L 0 from iteration 0 .

Revised: Clean labels erroneously removed in an earlier iteration (e.g., labels of hard to classify samples) can be reconsidered for model training again in later iterations.

Original: Filtering strategy
Revised: Filtering strategy

Original: The model can determine the set of potentially correct labels L i based on agreement between the label y and its maximal likelihood prediction ˆ y | x with L i
Revised: The model can determine the set of potentially correct labels L i based on agreement between the label y and its maximal likelihood prediction ˆ y | x with L i

Original: = { ( y, x ) | ˆ y x =
Revised: = { ( y, x ) | ˆ y x =

Original: y ; ∀ ( y, x ) ∈ L 0 } .
Revised: y ; ∀ ( y, x ) ∈ L 0 } .

Original: L 0 is the label set provided in the beginning, ( y, x ) are the samples and their respective noisy labels in the iteration i .
Revised: L 0 is the label set provided in the beginning, ( y, x ) are the samples and their respective noisy labels in the iteration i .

Original: In other words, the labels are only used for supervised training if in the current epoch, the model predicts the respective label to be the correct class with the highest likelihood.
Revised: In other words, the labels are only used for supervised training if in the current epoch, the model predicts the respective label to be the correct class with the highest likelihood.

Original: In practice, our framework does not use ˆ y ( x ) of model snapshots for ﬁltering but a moving-average of the ensemble models and predictions to improve the ﬁltering decision.
Revised: In practice, our framework does not use ˆ y ( x ) of model snapshots for ﬁltering but a moving-average of the ensemble models and predictions to improve the ﬁltering decision.

Original: 2. S ELF - ENSEMBLE LEARNING
Revised: 2. S ELF - ENSEMBLE LEARNING

Original: The model’s predictions for noisy samples tend to ﬂuctuate.
Revised: The model’s predictions for noisy samples tend to ﬂuctuate.

Original: For example, take a cat wrongly labeled as a tiger.
Revised: For example, take a cat wrongly labeled as a tiger.

Original: Other cat samples would encourage the model to predict the given cat image as a cat.
Revised: Other cat samples would encourage the model to predict the given cat image as a cat.

Original: Contrary, the wrong label tiger regularly pulls the model back to predict the cat as a tiger.
Revised: Contrary, the wrong label tiger regularly pulls the model back to predict the cat as a tiger.

Original: Hence, using the model’s predictions gathered in one single training epoch for ﬁltering is sub-optimal.
Revised: Hence, using the model’s predictions gathered in one single training epoch for ﬁltering is sub-optimal.

Original: Therefore, in our framework SELF , our model relies on ensembles of models and predictions.
Revised: Therefore, in our framework SELF , our model relies on ensembles of models and predictions.

Original: Model ensemble with Mean Teacher A natural way to form a model ensemble is by using an exponential running average of model snapshots (Fig. 3a).
Revised: Model ensemble with Mean Teacher A natural way to form a model ensemble is by using an exponential running average of model snapshots (Fig. 3a).

Original: This idea was proposed in Tarvainen & Valpola (2017) for semi-supervised learning and is known as the Mean Teacher model.
Revised: This idea was proposed in Tarvainen & Valpola (2017) for semi-supervised learning and is known as the Mean Teacher model.

Original: In our framework, both the mean teacher model and the normal model are evaluated on all data to preserve the consistency between both models.
Revised: In our framework, both the mean teacher model and the normal model are evaluated on all data to preserve the consistency between both models.

Original: The consistency loss between student and teacher output distribution can be realized with Mean-Square-Error loss or Kullback-Leibler-divergence.
Revised: The consistency loss between student and teacher output distribution can be realized with Mean-Square-Error loss or Kullback-Leibler-divergence.

Original: More details for training with the model ensemble can be found in Appendix A.
Revised: More details for training with the model ensemble can be found in Appendix A.

Original: Prediction ensemble Additionally, we propose to collect the sample predictions over multiple training epochs:
Revised: Prediction ensemble Additionally, we propose to collect the sample predictions over multiple training epochs:

Original: z j = αz j − 1 + (1 − α )
Revised: z j = αz j − 1 + (1 − α )

Original: ˆ z j , whereby z j depicts the moving-average prediction of sample k at epoch j , α is a momentum,
Revised: ˆ z j , whereby z j depicts the moving-average prediction of sample k at epoch j , α is a momentum,

Original: ˆ z j is the model prediction for sample k in epoch j .
Revised: ˆ z j is the model prediction for sample k in epoch j .

Original: This scheme is displayed in Fig.
Revised: This scheme is displayed in Fig.

Original: 3b.
Revised: 3b.

Original: For each sample, we store the moving-average predictions, accumulated over the past iterations.
Revised: For each sample, we store the moving-average predictions, accumulated over the past iterations.

Original: Besides having a more stable basis for the ﬁltering step, our proposed procedure also leads to negligible memory and computation overhead.
Revised: Besides having a more stable basis for the ﬁltering step, our proposed procedure also leads to negligible memory and computation overhead.

Revised: [Figure]

Original: Further, due to continuous training of the best model from the previous model, computation time can be signiﬁcantly reduced, compared to re-training the model from scratch.
Revised: Further, due to continuous training of the best model from the previous model, computation time can be signiﬁcantly reduced, compared to re-training the model from scratch.

Original: On the new ﬁltered dataset, the model must only slowly adapt to the new noise ratio contained in the training set.
Revised: On the new ﬁltered dataset, the model must only slowly adapt to the new noise ratio contained in the training set.

Original: Depending on the computation budget, a maximal number of iterations for ﬁltering can be set to save time.
Revised: Depending on the computation budget, a maximal number of iterations for ﬁltering can be set to save time.

Original: [Figure]

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Reed et al.
Revised: Reed et al.

Original: (2014); Azadi et al.
Revised: (2014); Azadi et al.

Original: (2015) performed early works on learning robustly under label noise for deep neural networks.
Revised: (2015) performed early works on learning robustly under label noise for deep neural networks.

Original: Recently, Rolnick et al.
Revised: Recently, Rolnick et al.

Original: (2017) have shown for classiﬁcation that deep neural networks come with natural robustness to label noise following a particular random distribution.
Revised: (2017) have shown for classiﬁcation that deep neural networks come with natural robustness to label noise following a particular random distribution.

Original: No modiﬁcation of the network or the training procedure is required to achieve this robustness.
Revised: No modiﬁcation of the network or the training procedure is required to achieve this robustness.

Original: Following this insight, our framework SELF relies on this natural robustness to kickstart the self-ensemble ﬁltering process to extend the robust behavior to more challenging scenarios.
Revised: Following this insight, our framework SELF relies on this natural robustness to kickstart the self-ensemble ﬁltering process to extend the robust behavior to more challenging scenarios.

Original: Laine & Aila (2016);
Revised: Laine & Aila (2016);

Original: Luo et al.
Revised: Luo et al.

Original: (2018) proposed to apply semi-supervised techniques on the data to counteract noise.
Revised: (2018) proposed to apply semi-supervised techniques on the data to counteract noise.

Original: These and other semi-supervised learning techniques learn from a static, initial set of noisy labels and have no mechanisms to repair labels.
Revised: These and other semi-supervised learning techniques learn from a static, initial set of noisy labels and have no mechanisms to repair labels.

Original: Therefore, the supervised losses in their learning objective are typically high until the model strongly overﬁts to the label noise.
Revised: Therefore, the supervised losses in their learning objective are typically high until the model strongly overﬁts to the label noise.

Original: Compared to these works, our framework performs a variant of self-supervised label corrections.
Revised: Compared to these works, our framework performs a variant of self-supervised label corrections.

Original: The network learns from a dynamic, variable set of labels, which is determined by the network itself.
Revised: The network learns from a dynamic, variable set of labels, which is determined by the network itself.

Original: Progressive ﬁltering allows the network to (1) focus on a label set with a signiﬁcantly lower noise ratio and (2) repair wrong decisions made by itself in an earlier iteration.
Revised: Progressive ﬁltering allows the network to (1) focus on a label set with a signiﬁcantly lower noise ratio and (2) repair wrong decisions made by itself in an earlier iteration.

Original: Other works assign weights to potentially wrong labels to reduce the learning signal (Jiang et al., 2017;
Revised: Other works assign weights to potentially wrong labels to reduce the learning signal (Jiang et al., 2017;

Original: Ren et al., 2018; Jenni & Favaro, 2018).
Revised: Ren et al., 2018; Jenni & Favaro, 2018).

Original: These approaches tend to assign less extreme weights or hyperparameters that are hard to set.
Revised: These approaches tend to assign less extreme weights or hyperparameters that are hard to set.

Original: Since the typical classiﬁcation loss is highly non-linear, a lower weight might still lead to learning from wrong labels.
Revised: Since the typical classiﬁcation loss is highly non-linear, a lower weight might still lead to learning from wrong labels.

Original: Compared to these works, the samples in SELF only receive extreme weights: either they are zero or one.
Revised: Compared to these works, the samples in SELF only receive extreme weights: either they are zero or one.

Original: Further, SELF focuses only on self-detecting the correct samples, instead of repairing the wrong labels.
Revised: Further, SELF focuses only on self-detecting the correct samples, instead of repairing the wrong labels.

Original: Typically, the set of correct samples are much easier to detect and are sufﬁciently representative to achieve high performance.
Revised: Typically, the set of correct samples are much easier to detect and are sufﬁciently representative to achieve high performance.

Original: Han et al.
Revised: Han et al.

Original: (2018b)
Revised: (2018b)

Original: ; Jiang et al.
Revised: ; Jiang et al.

Original: (2017) employ two collaborating and simultaneously learning networks to determine which samples to learn from and which not.
Revised: (2017) employ two collaborating and simultaneously learning networks to determine which samples to learn from and which not.

Original: However, the second network is free in its predictions and hence hard to tune.
Revised: However, the second network is free in its predictions and hence hard to tune.

Original: Compared to these works, we use ensemble learning as a principled approach to counteract model ﬂuctuations.
Revised: Compared to these works, we use ensemble learning as a principled approach to counteract model ﬂuctuations.

Original: In SELF , the second network is extremely restricted and is only composed of running averages of the ﬁrst network.
Revised: In SELF , the second network is extremely restricted and is only composed of running averages of the ﬁrst network.

Original: To realize the second network, we use the mean-teacher model (Tarvainen & Valpola, 2017) as a backbone.
Revised: To realize the second network, we use the mean-teacher model (Tarvainen & Valpola, 2017) as a backbone.

Original: Compared to their work, our self-ensemble label ﬁltering gradually detects the correct labels and learns from them, so the label set is variable.
Revised: Compared to their work, our self-ensemble label ﬁltering gradually detects the correct labels and learns from them, so the label set is variable.

Original: Further, we do use not only model ensembles but also an ensemble of predictions to detect correct labels.
Revised: Further, we do use not only model ensembles but also an ensemble of predictions to detect correct labels.

Original: Other works modify the primary loss function of the classiﬁcation tasks.
Revised: Other works modify the primary loss function of the classiﬁcation tasks.

Original: Patrini et al.
Revised: Patrini et al.

Original: (2017) estimates the noise transition matrix to correct the loss, Han et al.
Revised: (2017) estimates the noise transition matrix to correct the loss, Han et al.

Original: (2018a) uses human-in-the-loop,
Revised: (2018a) uses human-in-the-loop,

Original: Zhang & Sabuncu (2018); Thulasidasan et al.
Revised: Zhang & Sabuncu (2018); Thulasidasan et al.

Original: (2019) propose other forms of cross-entropy losses.
Revised: (2019) propose other forms of cross-entropy losses.

Original: The loss modiﬁcation impedes the transfer of these ideas to other tasks than classiﬁcation.
Revised: The loss modiﬁcation impedes the transfer of these ideas to other tasks than classiﬁcation.

Original: Compared to these works, our framework SELF does not modify the primary loss.
Revised: Compared to these works, our framework SELF does not modify the primary loss.

Revised: However, many tasks rely on the presence of clean labels such as anomaly detection (Nguyen et al., 2019a) or self-supervised and unsupervised learning (Nguyen et al., 2019b).

Original: The progressive ﬁltering procedure and self-ensemble learning  are  applicable in other tasks to counteract noise.
Revised: The progressive ﬁltering procedure and self-ensemble learning proposed are also applicable in these tasks to counteract noise effectively. [Table] USING NOISY VALIDATION SET [Table]

Original:   E VALUATION 4. E XPERIMENTS DESCRIPTIONS 4.1. S TRUCTURE OF THE ANALYSIS
Revised: [Table] [Table]  E VALUATION 4. E XPERIMENTS DESCRIPTIONS 4.1. S TRUCTURE OF THE ANALYSIS

Original: We evaluate our approach on CIFAR-10, CIFAR-100, an ImageNet-ILSVRC on different noise scenarios.
Revised: We evaluate our approach on CIFAR-10, CIFAR-100, an ImageNet-ILSVRC on different noise scenarios.

Original: For CIFAR-10, CIFAR-100, and ImageNet, we consider the typical situation with symmetric and asymmetric label noise.
Revised: For CIFAR-10, CIFAR- [Table] and ImageNet, we consider the typical situation with symmetric and asymmetric label noise.

Original: In the case of the symmetric noise, a label is randomly ﬂipped to another class with probability p .
Revised: In the case of the symmetric noise, a label is randomly ﬂipped to another class with probability p .

Original: Following previous works, we also consider label ﬂips of semantically sim- [Table] USING NOISY VALIDATION SET
Revised: Following previous works, we also consider label ﬂips of semantically similar classes on CIFAR-

Original: [Table]
Revised: [Table]

Original: CLEAN

Original: SET [Table] ilar classes on CIFAR-10, and pair-wise label ﬂips on CIFAR-100.
Revised: and pair-wise label ﬂips on CIFAR-100.

Original: Finally, we perform studies on the choice of the network architecture and the ablation of the components in our framework.
Revised: Finally, we perform studies on the choice of the network architecture and the ablation of the components in our framework.

Original: Tab.(Appendix) shows the in-deep analysis of semi-supervised learning strategies combined with recent works.
Revised: Tab.(Appendix) shows the in-deep analysis of semi-supervised learning strategies combined with recent works.

Original: Overall, the proposed framework SELF outperforms all these combinations.
Revised: Overall, the proposed framework SELF outperforms all these combinations.

Original: 4.1. C OMPARISONS TO PREVIOUS WORKS
Revised: 4.1. C OMPARISONS TO PREVIOUS WORKS

Original: We compare our work to previous methods from Reed-Hard
Revised: We compare our work to previous methods from Reed-Hard

Original: (Reed et al., 2014), S-model (Goldberger & Ben-Reuven, 2016), Wang et al.
Revised: (Reed et al., 2014), S-model (Goldberger & Ben-Reuven, 2016), Wang et al.

Original: , Rand.
Revised: , Rand.

Original: weights (Ren et al., 2018), Bi-level-model (Jenni & Favaro, 2018), D2L (Ma et al., 2018), SL (Wang et al., 2019), L q (Zhang & Sabuncu, 2018), Trunc L q (Zhang & Sabuncu, 2018), Forward ˆ T (Patrini et al., 2017), DAC (Thulasidasan et al., 2019), Random reweighting (Ren et al., 2018), and Learning to reweight (Ren et al., 2018).
Revised: weights (Ren et al., 2018), Bi-level-model (Jenni & Favaro, 2018), D2L (Ma et al., 2018), SL (Wang et al., 2019), L q (Zhang & Sabuncu, 2018), Trunc L q (Zhang & Sabuncu, 2018), Forward ˆ T (Patrini et al., 2017), DAC (Thulasidasan et al., 2019), Random reweighting (Ren et al., 2018), and Learning to reweight (Ren et al., 2018).

Original: For co-teaching (Han et al., 2018b), MentorNet (Jiang et al., 2017), JointOpt (Tanaka et al., 2018), the source codes are available and hence used for evaluation.
Revised: For co-teaching (Han et al., 2018b), MentorNet (Jiang et al., 2017), JointOpt (Tanaka et al., 2018), the source codes are available and hence used for evaluation.

Original: (Ren et al., 2018) and DAC (Thulasidasan et al., 2019) considered the setting of having a small clean validation set of 1000 and 5000 images respectively.
Revised: (Ren et al., 2018) and DAC (Thulasidasan et al., 2019) considered the setting of having a small clean validation set of 1000 and 5000 images respectively.

Original: For comparison purposes, we also experiment with a small clean set of 1000 images additionally.
Revised: For comparison purposes, we also experiment with a small clean set of 1000 images additionally.

Original: Further, we abandon oracle experiments or methods using additional information to keep the evaluation comparable.
Revised: Further, we abandon oracle experiments or methods using additional information to keep the evaluation comparable.

Original: For instance, Forward T (Patrini et al., 2017) uses the true underlying confusion matrix to correct the loss.
Revised: For instance, Forward T (Patrini et al., 2017) uses the true underlying confusion matrix to correct the loss.

Original: This information is neither known in typical scenarios nor used by other methods. 
Revised: This information is neither known in typical scenarios nor used by other methods. [Table]

Original: Whenever possible, we adopt the reported performance from the corresponding publications.
Revised: Whenever possible, we adopt the reported performance from the corresponding publications.

Original: The testing scenarios are kept as similar as possible to enable a fair comparison.
Revised: The testing scenarios are kept as similar as possible to enable a fair comparison.

Original: All tested scenarios use a noisy validation set with the same noise distribution as the training set unless stated otherwise.
Revised: All tested scenarios use a noisy validation set with the same noise distribution as the training set unless stated otherwise.

Original: All model performances are reported on the clean test set .
Revised: All model performances are reported on the clean test set .

Revised: [Table]

Revised: [Table]

Original: 4.1. N ETWORKS CONFIGURATION AND TRAINING
Revised: N ETWORKS CONFIGURATION AND TRAINING

Original: For the basic training of self-ensemble model, we use the Mean Teacher model (Tarvainen &
Revised: For the basic training of self-ensemble model [Table] we use the Mean Teacher model (Tarvainen &

Original: Valpola, 2017) available on GitHub 1 .
Revised: Valpola, 2017) available on GitHub 1 .

Original: The students and teacher networks are residual networks (

Original: He

Original: [Table]

Original: [Table] et al. [Table] with 26 layers with Shake-Shake-regularization (Gastaldi
Revised: The students and teacher networks are residual networks (He et al., 2016) with 26 layers with Shake-Shake-regularization (Gastaldi, 2017).

Original: [Table] We use the PyTorch (Paszke et al., 2017) implementation of the network and keep the training settings close to (Tarvainen & Valpola, 2017).
Revised: We use the PyTorch (Paszke et al., 2017) implementation of the network and keep the training settings close to (Tarvainen & Valpola, 2017).

Original: The network is trained with Stochastic Gradient Descent.
Revised: The network is trained with Stochastic Gradient Descent.

Original: In each ﬁltering iteration, the model is trained for a maximum of 300 epochs [Table] with patience of 50 epochs.
Revised: In each ﬁltering iteration, the model is trained for a maximum of 300 epochs, with patience of 50 epochs.

Original: For more training details, see the appendix.
Revised: For more training details, see the appendix.

Original: 4. E XPERIMENTS RESULTS 4.2. S YMMETRIC LABEL NOISE
Revised: 4. E XPERIMENTS RESULTS 4.2. S YMMETRIC LABEL NOISE

Original: CIFAR-10 and 100 Results for typical uniform noise scenarios with noise ratios on CIFARand CIFAR-100 are shown in Tab. 1.
Revised: CIFAR-10 and 100 Results for typical uniform noise scenarios with noise ratios on CIFARand CIFAR-100 are shown in Tab. 1.

Original: More results are visualized in Fig.
Revised: More results are visualized in Fig.

Original: 1a (CIFAR-10) and Fig.
Revised: 1a (CIFAR-10) and Fig.

Original: 1b (CIFAR-100).
Revised: 1b (CIFAR-100).

Original: Our approach SELF performs robustly in the case of lower noise ratios with up to 60% and outperforms previous works.
Revised: Our approach SELF performs robustly in the case of lower noise ratios with up to 60% and outperforms previous works.

Original: Although a strong performance loss occurs at 80% label noise,
Revised: Although a strong performance loss occurs at 80% label noise,

Revised: The models are trained at 40% label noise and the best model is picked based on the evaluation on noisy validation data.

Revised: Mentornet shows the best previously reported results.

Revised: Mentornet* is based on Resnet-101.

Revised: We chose the smaller Resnext50 model to reduce the run-time.

Revised: [Table]

Original: SELF still outperforms most of  the previous approaches.
Revised: The Resnet baseline was trained on the full noisy label set. Adding progressive ﬁltering improves over this baseline. The Mean Teacher maintains an ensemble of model snapshots, which helps counteract noise. Having progressive ﬁltering and model ensembles (-MVA-pred.) makes the model more robust but still fails at 80% noise.

Revised: The full

Original:  The experiment SELF* using a 1000 clean validation images shows that the performance loss mostly originates from the progressive ﬁltering relying too strongly on the extremely noisy validation set.
Revised: SELF framework additionally uses the prediction ensemble for detection of correct labels. [Table] SELF still outperforms most of the previous approaches. The experiment SELF* using a 1000 clean validation images shows that the performance loss mostly originates from the progressive ﬁltering relying too strongly on the extremely noisy validation set.

Original: ImageNet-ILSVRC Tab. 4 shows the precision@1 and @5 of various models, given 40% label noise in the training set.
Revised: ImageNet-ILSVRC Tab. 4 shows the precision@1 and @5 of various models, given 40% label noise in the training set.

Original: Our networks are based on ResNext18 and Resnext50.
Revised: Our networks are based on ResNext18 and Resnext50.

Original: Note that MentorNet (Jiang et al., 2017) uses Resnet101 (P@1: 78.25 ) (Goyal et al., 2017), which has higher performance compared to Resnext50 (P@1: 77.8 ) (Xie et al., 2017) on the standard ImageNet validation set.
Revised: Note that MentorNet (Jiang et al., 2017) uses Resnet101 (P@1: 78.25 ) (Goyal et al., 2017), which has higher performance compared to Resnext50 (P@1: 77.8 ) (Xie et al., 2017) on the standard ImageNet validation set.

Original: Despite the weaker model, SELF (ResNext50) surpasses the best previously reported results by more than 5% absolute improvement.
Revised: Despite the weaker model, SELF (ResNext50) surpasses the best previously reported results by more than 5% absolute improvement.

Original: Even the signiﬁcantly weaker model ResNext18 outperforms
Revised: Even the signiﬁcantly weaker model ResNext18 outperforms

Original: MentorNet, which is based on a more powerful ResNet101 network.
Revised: MentorNet, which is based on a more powerful ResNet101 network.

Original: The models are trained at 40% label noise and the best model is picked based on the evaluation on noisy validation data.

Original: Mentornet shows the best previously reported results.

Original: Mentornet* is based on Resnet-101.

Original: We chose the smaller Resnext50 model to reduce the run-time.

Original: [Table]

Original: Resnet baseline was trained on the full noisy label set.

Original: Adding progressive ﬁltering improves over this baseline

Original: The Mean Teacher main tains an ensemble of model snapshots which helps counteract noise

Original: Having progressive ﬁltering and model ensembles (-MVA-pred.) makes the model more robust but still fails at 80% noise framework additionally ensemble for detection of correct labels

Original: [Table]

Original:  A SYMMETRIC LABEL NOISE
Revised: 4.2. A SYMMETRIC LABEL NOISE

Original: Tab. 2 shows more challenging noise scenarios when the noise is not class-symmetric and uniform.
Revised: Tab. 2 shows more challenging noise scenarios when the noise is not class-symmetric and uniform.

Original: Concretely, labels are ﬂipped among semantically similar classes such as CAT and DOG on CIFAR10.
Revised: Concretely, labels are ﬂipped among semantically similar classes such as CAT and DOG on CIFAR10.

Original: On CIFAR-100, each label is ﬂipped to the next class with a probability p .
Revised: On CIFAR-100, each label is ﬂipped to the next class with a probability p .

Original: In these scenarios, our framework SELF also retains high performance and only shows a small performance drop at 40% noise.
Revised: In these scenarios, our framework SELF also retains high performance and only shows a small performance drop at 40% noise.

Original: The high label noise resistance of our framework indicates that the proposed self-ensemble ﬁltering process helps the network identify correct samples, even under extreme noise ratios.
Revised: The high label noise resistance of our framework indicates that the proposed self-ensemble ﬁltering process helps the network identify correct samples, even under extreme noise ratios.

Original: 4.2. E FFECTS OF DIFFERENT ARCHITECTURES
Revised: 4.2. E FFECTS OF DIFFERENT ARCHITECTURES

Original: Previous works utilize a various set of different architectures, which hinders a fair comparison.
Revised: Previous works utilize a various set of different architectures, which hinders a fair comparison.

Original: Tab. 3 shows the performance of our framework SELF compared to previous approaches.
Revised: Tab. 3 shows the performance of our framework SELF compared to previous approaches.

Original: SELF outperforms other works in all scenarios except for CIFAR-10 with 80% noise.
Revised: SELF outperforms other works in all scenarios except for CIFAR-10 with 80% noise.

Original: Typical robust learning approaches lead to signiﬁcant accuracy losses at 40% noise, while SELF still retains high performance.
Revised: Typical robust learning approaches lead to signiﬁcant accuracy losses at 40% noise, while SELF still retains high performance.

Original: Further, note that SELF allows the network’s performance to remain consistent across the different underlying architectures.
Revised: Further, note that SELF allows the network’s performance to remain consistent across the different underlying architectures.

Original: 4.2. A BLATION STUDY
Revised: 4.2. A BLATION STUDY

Original: Tab. 5 shows the importance of each component in our framework.
Revised: Tab. 5 shows the importance of each component in our framework.

Original: See Fig.
Revised: See Fig.

Original: 6a, Fig. 6b for experiments on more noise ratios.
Revised: 4a, Fig. 4b for experiments on more noise ratios.

Original: As expected, the Resnet-baseline rapidly breaks down with increasing noise ratios.
Revised: As expected, the Resnet-baseline rapidly breaks down with increasing noise ratios.

Original: Adding self-supervised ﬁltering increases the performance slightly in lower noise ratios.
Revised: Adding self-supervised ﬁltering increases the performance slightly in lower noise ratios.

Original: However, the model has to rely on extremely noisy snapshots.
Revised: However, the model has to rely on extremely noisy snapshots.

Original: Contrary, using a model ensemble alone such as in Mean-Teacher can counteract noise on the noisy dataset CIFAR-10.
Revised: Contrary, using a model ensemble alone such as in Mean-Teacher can counteract noise on the noisy dataset CIFAR-10.

Original: On the more challenging CIFAR-100, however, the performance decreases strongly.
Revised: On the more challenging CIFAR-100, however, the performance decreases strongly.

Original: With self-supervised ﬁltering and model ensembles, SELF (without MVA-pred) is more robust and only impairs performance at 80% noise.
Revised: With self-supervised ﬁltering and model ensembles, SELF (without MVA-pred) is more robust and only impairs performance at 80% noise.

Original: The last performance boost is given by using moving-average predictions so that the network can reliably detect correctly labeled samples gradually. 
Revised: The last performance boost is given by using moving-average predictions so that the network can reliably detect correctly labeled samples gradually. Fig.

Revised: 4 shows the ablation experiments on more noise ratios.

Revised: The analyses shows that each component in SELF is essential for the model to learn robustly.

Original: C ONCLUSION
Revised: [Figure] [Table] [Table] [Table]

Revised:  S EMI - SUPERVISED LEARNING FOR PROGRESSIVE FILTERING

Revised: Tab. 6 shows different semi-supervised learning strategies: entropy learning, mean-teacher combined with recent works.

Revised: Note that Co-Teaching+Mean-Teacher cannot be implemented and run in the same conﬁguration as other experiments, due to memory constraints.

Revised: The analysis indicates the semi-supervised losses mostly stabilize the baselines, compared to the model without semi-supervised learning.

Revised: However, Co-teaching and JointOpt sometimes perform worse than the purely semi-supervised model.

Revised: This result indicates that their proposed frameworks are not always compatible with semi-supervised losses.

Revised: The progressive ﬁltering technique is seamlessly compatible with different semi-supervised losses.

Revised: The ﬁltering outperforms its counterparts when combined with Entropy Learning or Mean-teacher model.

Revised: Overall, SELF outperforms all considered combinations.

Revised:  C ONCLUSION

Original: We propose a simple and easy to implement a framework to train robust deep learning models under incorrect or noisy labels.
Revised: We propose a simple and easy to implement a framework to train robust deep learning models under incorrect or noisy labels.

Original: We ﬁlter out the training samples that are hard to learn (possibly noisy labeled samples) by leveraging ensemble of predictions of the single network’s output over different training epochs.
Revised: We ﬁlter out the training samples that are hard to learn (possibly noisy labeled samples) by leveraging ensemble of predictions of the single network’s output over different training epochs.

Original: Subsequently, we allow clean supervision from the non-hard samples and further leverage additional unsupervised loss from the entire dataset.
Revised: Subsequently, we allow clean supervision from the non-hard samples and further leverage additional unsupervised loss from the entire dataset.

Original: We show that our framework results in
Revised: We show that our framework results in

Original: DNN models with superior generalization performance on CIFAR-10, CIFAR-100 & ImageNet and outperforms all previous works under symmetric (uniform) and asymmetric noises.
Revised: DNN models with superior generalization performance on CIFAR-10, CIFAR-100 & ImageNet and outperforms all previous works under symmetric (uniform) and asymmetric noises.

Original: Furthermore, our models remain robust despite the increasing noise ratio and change in network architectures.
Revised: Furthermore, our models remain robust despite the increasing noise ratio and change in network architectures.

Original: A A PPENDIX A. M EAN T EACHER MODEL FOR ITERATIVE FILTERING
Revised: A A PPENDIX A. M EAN T EACHER MODEL FOR ITERATIVE FILTERING

Original: We apply the Mean Teacher algorithm in each iteration i in the train ( D filter , D val ) procedure as follows.
Revised: We apply the Mean Teacher algorithm in each iteration i in the train ( D filter , D val ) procedure as follows.

Original: • Input: examples with potentially clean labels D filter from the ﬁltering procedure.
Revised: • Input: examples with potentially clean labels D filter from the ﬁltering procedure.

Original: In the beginning ( i = 0 ), here D filter refers to entire labeled dataset.
Revised: In the beginning ( i = 0 ), here D filter refers to entire labeled dataset.

Original: • Initialize a supervised neural network as the student model M si .
Revised: • Initialize a supervised neural network as the student model M si .

Original: • Initialize the Mean Teacher model M ti as a copy of the student model with all weights detached.
Revised: • Initialize the Mean Teacher model M ti as a copy of the student model with all weights detached.

Original: • Let the loss function be the sum of normal classiﬁcation loss of M si and the consistency loss between the outputs of M ti and M ti • Select an optimizer • In each training iteration: – Update the weights of M si using the selected optimizer – Update the weights of M ti as an exponential moving-average of the student weights – Evaluate performance of M si and M ti over D val to verify the early stopping criteria.
Revised: • Let the loss function be the sum of normal classiﬁcation loss of M si and the consistency loss between the outputs of M ti and M ti • Select an optimizer • In each training iteration: – Update the weights of M si using the selected optimizer – Update the weights of M ti as an exponential moving-average of the student weights – Evaluate performance of M si and M ti over D val to verify the early stopping criteria.

Original: • Return the best M ti
Revised: • Return the best M ti

Original: A. A SSUMPTIONS DICUSSIONS
Revised: A. A SSUMPTIONS DICUSSIONS

Original: Our method performs best when the following assumptions are hold.
Revised: Our method performs best when the following assumptions are hold.

Original: Natural robustness assumption of deep networks (Rolnick et al., 2017):
Revised: Natural robustness assumption of deep networks (Rolnick et al., 2017):

Original: The networks attempt to learn the easiest way to explain most of the data.
Revised: The networks attempt to learn the easiest way to explain most of the data.

Original: SELF uses this assumption to kickstart the learning process.
Revised: SELF uses this assumption to kickstart the learning process.

Original: Correct samples dominate over wrongly labeled samples At 80% noise on CIFAR-10, the correctly labeled cats (20% out of all cat images) still dominates over samples wrongly labeled as cat ( 8 . 8 % for each class).
Revised: Correct samples dominate over wrongly labeled samples At 80% noise on CIFAR-10, the correctly labeled cats (20% out of all cat images) still dominates over samples wrongly labeled as cat ( 8 . 8 % for each class).

Original: Independence results in less overﬁtting SELF performs best if the noises on the validation set and training set are i.i.d. .
Revised: Independence results in less overﬁtting SELF performs best if the noises on the validation set and training set are i.i.d. .

Original: SELF uses the validation data for early stopping.
Revised: SELF uses the validation data for early stopping.

Original: Hence, a high correlation of label noise between train and valid increases the chance of model overﬁtting.
Revised: Hence, a high correlation of label noise between train and valid increases the chance of model overﬁtting.

Original: Sufﬁcient label randomness assumption The subset of all correctly labeled samples capture all samples clusters.
Revised: Sufﬁcient label randomness assumption The subset of all correctly labeled samples capture all samples clusters.

Original: In fact, many works from the active learning literature show that less than% of the labeled samples are required to achieve the highest model performance.
Revised: In fact, many works from the active learning literature show that less than% of the labeled samples are required to achieve the highest model performance.

Original: SELF performs progressive expansion of the correct labels sets.
Revised: SELF performs progressive expansion of the correct labels sets.

Original: At larger noise ratios, not all clusters are covered by the identiﬁed samples.
Revised: At larger noise ratios, not all clusters are covered by the identiﬁed samples.

Original: Therefore on task containing many classes, e.g., CIFAR-100, the model performance decreases faster than on CIFAR-10.
Revised: Therefore on task containing many classes, e.g., CIFAR-100, the model performance decreases faster than on CIFAR-10.

Original: The model performance reduces when these assumptions are strongly violated.
Revised: The model performance reduces when these assumptions are strongly violated.

Original: Each assumption should have its own ”critical” threshold for violation.
Revised: Each assumption should have its own ”critical” threshold for violation.

Original: A future in-depth analysis to challenge the assumptions is an interesting future research direction.
Revised: A future in-depth analysis to challenge the assumptions is an interesting future research direction.

Original: A. T RAINING DETAILS A.3. CIFAR-10 AND CIFAR- 
Revised: A. T RAINING DETAILS A.3. CIFAR-10 AND CIFAR-

Original: Dataset Tab. 6 shows the details of CIFAR-10 and 100 datasets in our evaluation pipeline.
Revised: Dataset Tab. 7 shows the details of CIFAR-10 and 100 datasets in our evaluation pipeline.

Original: The validation set is contaminated with the same noise ratio as the training data unless stated otherwise.
Revised: The validation set is contaminated with the same noise ratio as the training data unless stated otherwise.

Original: Network training For the training our model SELF , we use the standard conﬁguration provided by Tarvainen & Valpola (2017) 2 .
Revised: Network training For the training our model SELF , we use the standard conﬁguration provided by Tarvainen & Valpola (2017) 2 .

Original: Concretely, we use the SGD-optimizer with Nesterov Sutskever et al.
Revised: Concretely, we use the SGD-optimizer with Nesterov Sutskever et al.

Original: (2013) momentum, a learning rate of 0.05 with cosine learning rate annealing Loshchilov &
Revised: (2013) momentum, a learning rate of 0.05 with cosine learning rate annealing Loshchilov &

Original: Hutter (2016), a weight decay of 2e-4, max iteration per ﬁltering step of 300, patience of 50 epochs, 
Revised: Hutter (2016), a weight decay of 2e-4, max iteration per ﬁltering step of 300, patience of 50 epochs, total epochs count of 600.

Original: Note that the noise on the training and validation set is not correlated.
Revised: Note that the noise on the training and validation set is not correlated.

Original: Hence, maximizing the  on the noisy  provides a useful (but noisy estimate for the generalization 
Revised: Hence, maximizing the accuracy on the noisy set provides a useful (but noisy) estimate for the generalization ability on unseen test data.

Original: [Table]
Revised: [Table]

Original: For basic training of baselines models without semi-supervised learning, we had to set the learning rate to 0.01.
Revised: For basic training of baselines models without semi-supervised learning, we had to set the learning rate to 0.01.

Original: In the case of higher learning rates, the loss typically explodes.
Revised: In the case of higher learning rates, the loss typically explodes.

Original: Every other option is kept the same.
Revised: Every other option is kept the same.

Original: Semi-supervised learning For the mean teacher training, additional hyperparameters are required.
Revised: Semi-supervised learning For the mean teacher training, additional hyperparameters are required.

Original: In both cases of CIFAR-10 and CIFAR-100, we again take the standard conﬁguration with the consistency loss to mean-squared-error and a consistency weight: 100.0, logit distance cost: 0.01, consistency-ramp-up:5.
Revised: In both cases of CIFAR-10 and CIFAR-100, we again take the standard conﬁguration with the consistency loss to mean-squared-error and a consistency weight: 100.0, logit distance cost: 0.01, consistency-ramp-up:5.

Original: The total batch-size is 512, with 124 samples being reserved for labeled samples, 388 for unlabeled data.
Revised: The total batch-size is 512, with 124 samples being reserved for labeled samples, 388 for unlabeled data.

Original: Each epoch is deﬁned as a complete processing of all unlabeled data.
Revised: Each epoch is deﬁned as a complete processing of all unlabeled data.

Original: When training without semi-supervised-learning, the entire batch is used for labeled data.
Revised: When training without semi-supervised-learning, the entire batch is used for labeled data.

Original: Data augmentation The data are normalized to zero-mean and standard-variance of one.
Revised: Data augmentation The data are normalized to zero-mean and standard-variance of one.

Original: Further, we use real-time data augmentation with random translation and reﬂection, subsequently random horizontal ﬂip.
Revised: Further, we use real-time data augmentation with random translation and reﬂection, subsequently random horizontal ﬂip.

Original: The standard PyTorch-library provides these transformations.
Revised: The standard PyTorch-library provides these transformations.

Original: A.3. I MAGE N ET -ILSVRC- 
Revised: A.3. I MAGE N ET -ILSVRC-

Original: Network Training The network used for evaluation were ResNet He et al.
Revised: Network Training The network used for evaluation were ResNet He et al.

Original: and
Revised: and

Original: Resnext Xie et al.
Revised: Resnext Xie et al.

Original: (2017) for training.
Revised: (2017) for training.

Original: All ResNext variants use a cardinality of 32 and base width of 4 (32x4d).
Revised: All ResNext variants use a cardinality of 32 and base width of 4 (32x4d).

Original: ResNext models follow the same structure as their Resnet counterparts, except for the cardinality and base width.
Revised: ResNext models follow the same structure as their Resnet counterparts, except for the cardinality and base width.

Original: All other conﬁgurations are kept as close as possible to Tarvainen & Valpola (2017).
Revised: All other conﬁgurations are kept as close as possible to Tarvainen & Valpola (2017).

Original: The initial learning rate to handle large batches Goyal et al.
Revised: The initial learning rate to handle large batches Goyal et al.

Original: (2017) is set to 0.1; the base learning rate is 0.
Revised: (2017) is set to 0.1; the base learning rate is 0.

Original: with a single cycle of cosine annealing.
Revised: with a single cycle of cosine annealing.

Original: Semi-supervised learning Due to the large images, the batch size is set to 40 in total with 20/ for labeled and unlabeled samples, respectively.
Revised: Semi-supervised learning Due to the large images, the batch size is set to 40 in total with 20/ for labeled and unlabeled samples, respectively.

Original: We found the Kullback-divergence leads to no meaningful network training.
Revised: We found the Kullback-divergence leads to no meaningful network training.

Original: Hence, we set the consistency loss to mean-squared-error, with a weight of 1000.
Revised: Hence, we set the consistency loss to mean-squared-error, with a weight of 1000.

Original: We use consistency ramp up of 5 epochs to give the mean teacher more time in the beginning.
Revised: We use consistency ramp up of 5 epochs to give the mean teacher more time in the beginning.

Original: Weight decay is set to 5e-5; patience is four epochs to stop training in the current ﬁltering iteration.
Revised: Weight decay is set to 5e-5; patience is four epochs to stop training in the current ﬁltering iteration.

Original: Filtering We ﬁlter noisy samples with the topk=5 strategy, instead of taking the maximumlikelihood (ML) prediction as on CIFAR-10 and CIFAR-100.
Revised: Filtering We ﬁlter noisy samples with the topk=5 strategy, instead of taking the maximumlikelihood (ML) prediction as on CIFAR-10 and CIFAR-100.

Original: That means the samples are kept for supervised training if their provided label lies within the top 5 predictions of the model.
Revised: That means the samples are kept for supervised training if their provided label lies within the top 5 predictions of the model.

Original: The main
Revised: The main

Original: [Figure]
Revised: [Figure]

Original: reason is that each image of ImageNet might contain multiple objects.
Revised: reason is that each image of ImageNet might contain multiple objects.

Original: Filtering with ML-predictions is too strict and would lead to a small recall of the detection of the correct sample.
Revised: Filtering with ML-predictions is too strict and would lead to a small recall of the detection of the correct sample.

Original: Data Augmentation For all data, we normalize the RGB-images by the mean: (0.485, 0.456, 0.406) and the standard variance (0.229, 0.224, 0.225).
Revised: Data Augmentation For all data, we normalize the RGB-images by the mean: (0.485, 0.456, 0.406) and the standard variance (0.229, 0.224, 0.225).

Original: For training data, we perform a random rotation of up to 10 degrees, randomly resize images to 224x224, apply random horizontal ﬂip and random color jittering.
Revised: For training data, we perform a random rotation of up to 10 degrees, randomly resize images to 224x224, apply random horizontal ﬂip and random color jittering.

Original: This noise is needed in regular mean-teacher training.
Revised: This noise is needed in regular mean-teacher training.

Original: The jittering setting are: brightness=0.4, contrast=0.4, saturation=0.4, hue=0.1.
Revised: The jittering setting are: brightness=0.4, contrast=0.4, saturation=0.4, hue=0.1.

Original: The validation data are resized to 256x256 and randomly cropped to 224x
Revised: The validation data are resized to 256x256 and randomly cropped to 224x

Original: A.3. S EMI - SUPERVISED LOSSES
Revised: A.3. S EMI - SUPERVISED LOSSES

Original: For the learning of wrongly labeled samples, Fig. 5 shows the relationship between the typical reweighting scheme and our baseline push-away-loss.
Revised: For the learning of wrongly labeled samples, Fig. 6 shows the relationship between the typical reweighting scheme and our baseline push-away-loss.

Original: Typically, reweighting is applied directly to the losses with samples weights w ( k ) for each sample k as shown in Eq.
Revised: Typically, reweighting is applied directly to the losses with samples weights w ( k ) for each sample k as shown in Eq.

Original: [Equation]
Revised: [Equation]

Original: D is the dataset, x ( k ) and y ( k ) label are the samples k and its noisy label.
Revised: D is the dataset, x ( k ) and y ( k ) label are the samples k and its noisy label.

Original: w ( k ) i is the samples weight for the sample k
Revised: w ( k ) i is the samples weight for the sample k

Original: at step i .
Revised: at step i .

Original: Negative samples weights w ( k ) i are often assigned to push the network away from the wrong labels.
Revised: Negative samples weights w ( k ) i are often assigned to push the network away from the wrong labels.

Original: Let w ( k ) i = − c
Revised: Let w ( k ) i = − c

Original: k ) i with c ( k ) i > 0 , then we have:
Revised: k ) i with c ( k ) i > 0 , then we have:

Original: [Equation]
Revised: [Equation]

Original: Which results in:
Revised: Which results in:

Original: [Equation]
Revised: [Equation]

Original: In other words, we perform gradient ascent for wrongly labeled samples.
Revised: In other words, we perform gradient ascent for wrongly labeled samples.

Original: However, the Negativelog-likelihood is not designed for gradient ascent.
Revised: However, the Negativelog-likelihood is not designed for gradient ascent.

Original: Hence the gradients of wrongly labeled samples vanish if the prediction is too close to the noisy label.
Revised: Hence the gradients of wrongly labeled samples vanish if the prediction is too close to the noisy label.

Original: This effect is similar to the training of
Revised: This effect is similar to the training of

Original: Generative Adversarial Network (GAN) Goodfellow et al..
Revised: Generative Adversarial Network (GAN) Goodfellow et al..

Original: In the GAN-framework, the generator loss is not simply set to the negated version of the discriminator’s loss for the same reason.
Revised: In the GAN-framework, the generator loss is not simply set to the negated version of the discriminator’s loss for the same reason.

Original: Therefore, to provide a fair comparison with our framework, we suggest the push-away-loss
Revised: Therefore, to provide a fair comparison with our framework, we suggest the push-away-loss

Original: L Push − away ( y ( k ) label , x ( k ) , D ) with improved gradients as follows:
Revised: L Push − away ( y ( k ) label , x ( k ) , D ) with improved gradients as follows:

Original: [Equation]
Revised: [Equation]

Original: Whereby Y is the set of all classes in the training set.
Revised: Whereby Y is the set of all classes in the training set.

Original: This loss has improved gradients to push the model away from the potentially wrong labels.
Revised: This loss has improved gradients to push the model away from the potentially wrong labels.

Original: [Figure] [Figure]
Revised: [Figure] [Table]

Original: Entropy minimization
Revised: Entropy minimization

Original: The typical entropy loss for semi-supervised learning is shown in Fig.
Revised: The typical entropy loss for semi-supervised learning is shown in Fig.

Original: It encourages the model to provide extreme predictions (such as 0 or 1 ) for each sample.
Revised: It encourages the model to provide extreme predictions (such as 0 or 1 ) for each sample.

Original: Over a large number of samples, the model should balance its predictions over all classes.
Revised: Over a large number of samples, the model should balance its predictions over all classes.

Original: The entropy loss can easily be applied to all samples to express the uncertainty about the provided labels.
Revised: The entropy loss can easily be applied to all samples to express the uncertainty about the provided labels.

Original: Alternatively, the loss can be combined with a strict ﬁltering strategy, as in our work, which removes the labels of potentially wrongly labeled samples.
Revised: Alternatively, the loss can be combined with a strict ﬁltering strategy, as in our work, which removes the labels of potentially wrongly labeled samples.

Original: For a large noise ratio, predictions of wrongly labeled samples ﬂuctuate strongly over previous training iterations.
Revised: For a large noise ratio, predictions of wrongly labeled samples ﬂuctuate strongly over previous training iterations.

Original: Amplifying these network decisions could lead to even noisier models model.
Revised: Amplifying these network decisions could lead to even noisier models model.

Original: Combined with iterative ﬁltering, the framework will have to rely on a single noisy model snapshot.
Revised: Combined with iterative ﬁltering, the framework will have to rely on a single noisy model snapshot.

Original: In the case of an unsuitable snapshot, the ﬁltering step will make many wrong decisions.
Revised: In the case of an unsuitable snapshot, the ﬁltering step will make many wrong decisions.

Original: Fig. 6 shows the ablation experiments on more noise ratios. The analyses shows that each component in SELF is essential for the model to learn robustly.
Revised: [Equation]

Original: A.5. C OMPLETE REMOVAL OF SAMPLES
Revised: RESULTS A.4. C OMPLETE REMOVAL OF SAMPLES

Original: Tab. 7 shows the results of deleting samples from the training set.
Revised: Tab. 8 shows the results of deleting samples from the training set.

Original: It leads to signiﬁcant performances gaps compared to our strategy ( SELF ), which considers the removed samples as unlabeled data.
Revised: It leads to signiﬁcant performances gaps compared to our strategy ( SELF ), which considers the removed samples as unlabeled data.

Original: In case of a considerable label noise of 80%, the gap is close to 9%.
Revised: In case of a considerable label noise of 80%, the gap is close to 9%.

Original: Continuously using the ﬁltered samples lead to signiﬁcantly better results.
Revised: Continuously using the ﬁltered samples lead to signiﬁcantly better results.

Original: The unsupervised-loss provides meaningful learning signals, which should be used for better model training.[Table] [Table] S
Revised: The unsupervised-loss provides meaningful learning signals, which should be used for better model training.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: S

Original: [Table]

Original: LEARNING WITH M EAN

Original: [Table]

Original: TEACHER

Original: [Table]

Original: [Table]

Original: [Table]

Original: A.

Original: [Table]

Original: S EMI - SUPERVISED LEARNING FOR PROGRESSIVE FILTERING

Original: Tab. 8 shows different semi-supervised learning strategies: entropy learning, mean-teacher combined with recent works.

Original: Note that Co-Teaching+Mean-Teacher cannot be implemented and run in the same conﬁguration as other experiments, due to memory constraints.

Original: The analysis indicates the semi-supervised losses mostly stabilize the baselines, compared to the model without semi-supervised learning.

Original: However, Co-teaching and JointOpt sometimes perform worse than the purely semi-supervised model.

Original: This result indicates that their proposed frameworks are not always compatible with semi-supervised losses.

Original: The progressive ﬁltering technique is seamlessly compatible with different semi-supervised losses.

Original: The ﬁltering outperforms its counterparts when combined with Entropy Learning or Mean-teacher model.

Original: Overall, SELF outperforms all considered combinations.

Original: [Figure]
Revised: [Figure]

Original: Further, note that the mean-teacher continously outperforms the noisy student models.
Revised: Further, note that the mean-teacher continously outperforms the noisy student models.

Original: This shows the positive effect of temporal emsembling to counter label noise.
Revised: This shows the positive effect of temporal emsembling to counter label noise.

Revised: A.4. S AMPLE TRAINING PROCESS

Original: Fig.
Revised: Fig.

Original: 7 shows the sample training processes of SELF under 60% and 80% noise on CIFAR-100.
Revised: 7 shows the sample training processes of SELF under 60% and 80% noise on CIFAR-100.

Original: The mean-teacher always outperform the student models.
Revised: The mean-teacher always outperform the student models.

Original: Further, note that regular training leads to rapid over-ﬁtting to label noise.
Revised: Further, note that regular training leads to rapid over-ﬁtting to label noise.

Original: Contrary, with our effective ﬁltering strategy, both models slowly increase their performance while the training accuracy approaches 100%.
Revised: Contrary, with our effective ﬁltering strategy, both models slowly increase their performance while the training accuracy approaches 100%.

Original: Hence, by using progressive ﬁltering, our model could erase the inconsistency in the provided labels set.
Revised: Hence, by using progressive ﬁltering, our model could erase the inconsistency in the provided labels set.