Original: P RESTOPPING : H OW D OES E ARLY S TOPPING H ELP G ENERALIZATION AGAINST L ABEL N OISE ?
Revised: P RESTOPPING : H OW D OES E ARLY S TOPPING H ELP G ENERALIZATION AGAINST L ABEL N OISE ?

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A
Revised: A

Original: BSTRACT
Revised: BSTRACT

Original: Noisy labels are very common in real-world training data, which lead to poor generalization on test data because of overﬁtting to the noisy labels.
Revised: Noisy labels are very common in real-world training data, which lead to poor generalization on test data because of overﬁtting to the noisy labels.

Original: In this paper, we claim that such overﬁtting can be avoided by “early stopping” training a deep neural network before the noisy labels are severely memorized.
Revised: In this paper, we claim that such overﬁtting can be avoided by “early stopping” training a deep neural network before the noisy labels are severely memorized.

Original: Then, we resume training the early stopped network using a “maximal safe set,” which maintains a collection of almost certainly true-labeled samples at each epoch since the early stop point.
Revised: Then, we resume training the early stopped network using a “maximal safe set,” which maintains a collection of almost certainly true-labeled samples at each epoch since the early stop point.

Original: Putting them all together, our novel two-phase training method, called
Revised: Putting them all together, our novel two-phase training method, called

Original: Prestopping , realizes noise-free training under any type of label noise for practical use.
Revised: Prestopping , realizes noise-free training under any type of label noise for practical use.

Original: Extensive experiments using four image benchmark data sets verify that our method signiﬁcantly outperforms four state-of-the-art methods in test error by 0 . 4 – 8 . 2 percent points under existence of real-world noise.
Revised: Extensive experiments using four image benchmark data sets verify that our method signiﬁcantly outperforms four state-of-the-art methods in test error by 0 . 4 – 8 . 2 percent points under existence of real-world noise.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: By virtue of massive labeled data, deep neural networks (DNNs) have achieved a remarkable success in numerous machine learning tasks, such as image classiﬁcation (Krizhevsky et al., 2012) and object detection (Redmon et al., 2016).
Revised: By virtue of massive labeled data, deep neural networks (DNNs) have achieved a remarkable success in numerous machine learning tasks, such as image classiﬁcation (Krizhevsky et al., 2012) and object detection (Redmon et al., 2016).

Original: However, owing to their high capacity to memorize any label noise, the generalization performance of DNNs drastically falls down when noisy labels are contained in the training data (Jiang et al., 2018; Han et al., 2018; Song et al., 2019).
Revised: However, owing to their high capacity to memorize any label noise, the generalization performance of DNNs drastically falls down when noisy labels are contained in the training data (Jiang et al., 2018; Han et al., 2018; Song et al., 2019).

Original: In particular, Zhang et al.
Revised: In particular, Zhang et al.

Original: (2017) have shown that a standard convolutional neural network (CNN) can easily ﬁt the entire training data with any ratio of noisy labels and eventually leads to very poor generalization on the test data.
Revised: (2017) have shown that a standard convolutional neural network (CNN) can easily ﬁt the entire training data with any ratio of noisy labels and eventually leads to very poor generalization on the test data.

Original: Thus, it is challenging to train a DNN robustly even when noisy labels exist in the training data.
Revised: Thus, it is challenging to train a DNN robustly even when noisy labels exist in the training data.

Original: A popular approach to dealing with noisy labels is “sample selection” that selects true-labeled samples from the noisy training data (Jiang et al., 2018;
Revised: A popular approach to dealing with noisy labels is “sample selection” that selects true-labeled samples from the noisy training data (Jiang et al., 2018;

Original: Ren et al., 2018; Han et al., 2018;
Revised: Ren et al., 2018; Han et al., 2018;

Original: Yu et al., 2019; Song et al., 2019).
Revised: Yu et al., 2019; Song et al., 2019).

Original: Here, (1 − τ ) × 100% of small-loss training samples are treated as true-labeled ones and then used to update a DNN robustly, where τ ∈ [0 , 1] is a noise rate.
Revised: Here, (1 − τ ) × 100% of small-loss training samples are treated as true-labeled ones and then used to update a DNN robustly, where τ ∈ [0 , 1] is a noise rate.

Original: This loss-based separation is well known to be justiﬁed by the memorization effect (Arpit et al., 2017) that DNNs tend to learn easy patterns ﬁrst and then gradually memorize all samples.
Revised: This loss-based separation is well known to be justiﬁed by the memorization effect (Arpit et al., 2017) that DNNs tend to learn easy patterns ﬁrst and then gradually memorize all samples.

Original: In practice,
Revised: In practice,

Original: Han et al.
Revised: Han et al.

Original: (2018) empirically proved that training on such small-loss samples yields a much better generalization performance under artiﬁcial noise scenarios.
Revised: (2018) empirically proved that training on such small-loss samples yields a much better generalization performance under artiﬁcial noise scenarios.

Original: Despite its great success
Revised: Despite its great success

Original: [Figure]
Revised: [Figure]

Original: Song et al.
Revised: Song et al.

Original: (2019) have recently argued that the performance of the lossbased separation becomes considerably worse depending on the type of label noise.
Revised: (2019) have recently argued that the performance of the lossbased separation becomes considerably worse depending on the type of label noise.

Original: For instance,
Revised: For instance,

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: M e m
Revised: M e m

Original: [Figure]
Revised: [Figure]

Original: r i z a ti
Revised: r i z a ti

Original: [Figure]
Revised: [Figure]

Original: R
Revised: R

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: M e m o r
Revised: M e m o r

Original: i z a ti on
Revised: i z a ti on

Original: R a
Revised: R a

Original: [Figure]
Revised: [Figure]

Original: T e s t E rr o r
Revised: T e s t E rr o r

Original: [Figure]
Revised: [Figure]

Original: e s t E rr
Revised: e s t E rr

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: in symmetric noise (Figure 1(a)), the loss-based approach well separates true-labeled samples from false-labeled ones because many true-labeled ones exhibit smaller loss than false-labeled ones.
Revised: in symmetric noise (Figure 1(a)), the loss-based approach well separates true-labeled samples from false-labeled ones because many true-labeled ones exhibit smaller loss than false-labeled ones.

Original: On the other hand, in pair and real-world noises (Figures 1(b) and 1(c)), many false-labeled samples are misclassiﬁed as true-labeled ones because the two distributions overlap closely; this overlap conﬁrms that the loss-based separation still accumulates severe label noise from many misclassiﬁed cases, especially in real-world noise or pair noise which is regarded as more realistic than symmetric noise (Ren et al., 2018;
Revised: On the other hand, in pair and real-world noises (Figures 1(b) and 1(c)), many false-labeled samples are misclassiﬁed as true-labeled ones because the two distributions overlap closely; this overlap conﬁrms that the loss-based separation still accumulates severe label noise from many misclassiﬁed cases, especially in real-world noise or pair noise which is regarded as more realistic than symmetric noise (Ren et al., 2018;

Original: Yu et al., 2019).
Revised: Yu et al., 2019).

Original: This limitation deﬁnitely calls for a new approach that supports any type of label noise for practical use.
Revised: This limitation deﬁnitely calls for a new approach that supports any type of label noise for practical use.

Original: In this regard, as shown in Figure 2(a), we thoroughly investigated the memorization effect of a DNN on the two types of noises and found two interesting properties as follows:
Revised: In this regard, as shown in Figure 2(a), we thoroughly investigated the memorization effect of a DNN on the two types of noises and found two interesting properties as follows:

Original: • A noise type affects the memorization rate for false-labeled samples:
Revised: • A noise type affects the memorization rate for false-labeled samples:

Original: The memorization rate for false-labeled samples is faster with pair noise than with symmetric noise.
Revised: The memorization rate for false-labeled samples is faster with pair noise than with symmetric noise.

Original: That is, the red portion in Figure 2(a) starts to appear earlier in pair noise than in symmetric noise.
Revised: That is, the red portion in Figure 2(a) starts to appear earlier in pair noise than in symmetric noise.

Original: This observation supports the signiﬁcant overlap of true-labeled and false-labeled samples in Figure 1(b).
Revised: This observation supports the signiﬁcant overlap of true-labeled and false-labeled samples in Figure 1(b).

Original: Thus, the loss-based separation performs well only if the false-labeled samples are scarcely learned at an early stage of training, as in symmetric noise.
Revised: Thus, the loss-based separation performs well only if the false-labeled samples are scarcely learned at an early stage of training, as in symmetric noise.

Original: • There is a period where the network accumulates the label noise severely:
Revised: • There is a period where the network accumulates the label noise severely:

Original: Regardless of the noise type, the memorization of false-labeled samples signiﬁcantly increases at a late stage of training.
Revised: Regardless of the noise type, the memorization of false-labeled samples signiﬁcantly increases at a late stage of training.

Original: That is, the red portion in Figure 2(a) increases rapidly after the dashed line, in which we call the error-prone period .
Revised: That is, the red portion in Figure 2(a) increases rapidly after the dashed line, in which we call the error-prone period .

Original: (See Section 3.2.1 for the details of estimating the error-prone period).
Revised: (See Section 3.2.1 for the details of estimating the error-prone period).

Original: We note that the training in that period brings no beneﬁt .
Revised: We note that the training in that period brings no beneﬁt .

Original: The generalization performance of “Default” deteriorates sharply, as shown in Figure 2(c).
Revised: The generalization performance of “Default” deteriorates sharply, as shown in Figure 2(c).

Original: Based on these ﬁndings, we contend that eliminating this error-prone period should make a
Revised: Based on these ﬁndings, we contend that eliminating this error-prone period should make a

Original: profound
Revised: profound

Original: impact on robust optimization.
Revised: impact on robust optimization.

Original: In this paper, we propose a novel approach, called Prestopping , that achieves noise-free training based on the early stopping mechanism.
Revised: In this paper, we propose a novel approach, called Prestopping , that achieves noise-free training based on the early stopping mechanism.

Original: Because there is no beneﬁt from the error-prone period,
Revised: Because there is no beneﬁt from the error-prone period,

Original: Prestopping early stops training before that period begins.
Revised: Prestopping early stops training before that period begins.

Original: This early stopping effectively prevents a network from overﬁtting to false-labeled samples, and the samples memorized until that point are added to a maximal safe set because they are true-labeled (i.e., blue in Figure 2(a)) with high precision.
Revised: This early stopping effectively prevents a network from overﬁtting to false-labeled samples, and the samples memorized until that point are added to a maximal safe set because they are true-labeled (i.e., blue in Figure 2(a)) with high precision.

Original: Then, Prestopping resumes training the early stopped network only using the maximal safe set in support of noise-free training.
Revised: Then, Prestopping resumes training the early stopped network only using the maximal safe set in support of noise-free training.

Original: Notably, our proposed merger of “early stopping” and “learning from the maximal safe set” indeed eliminates the error-prone period from the training process, as shown in Figure 2(b).
Revised: Notably, our proposed merger of “early stopping” and “learning from the maximal safe set” indeed eliminates the error-prone period from the training process, as shown in Figure 2(b).

Original: As a result, the generalization performance of a DNN remarkably improves in both noise types, as shown in Figure 2(c).
Revised: As a result, the generalization performance of a DNN remarkably improves in both noise types, as shown in Figure 2(c).

Original: To validate the superiority of Prestopping , DenseNet (Huang et al., 2017) and VGG-19 (Simonyan &
Revised: To validate the superiority of Prestopping , DenseNet (Huang et al., 2017) and VGG-19 (Simonyan &

Original: Zisserman, 2015) were trained on both simulated and real-world noisy data sets, including CIFAR-10,
Revised: Zisserman, 2015) were trained on both simulated and real-world noisy data sets, including CIFAR-10,

Original: CIFAR-100, ANIMAL-10N, and Food-101N.
Revised: CIFAR-100, ANIMAL-10N, and Food-101N.

Original: Compared with four state-of-the-art methods,
Revised: Compared with four state-of-the-art methods,

Original: Prestopping signiﬁcantly improved test error by up to 18 . 

Revised: Prestopping signiﬁcantly improved test error by up to 18 .

Original: pp 2 in a wide range of noise rates.
Revised: pp 2 in a wide range of noise rates.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Numerous studies have been conducted to address the problem of learning from noisy labels.
Revised: Numerous studies have been conducted to address the problem of learning from noisy labels.

Original: A typical method is using “loss correction” that estimates the label transition matrix and corrects the loss of the samples in the mini-batch.
Revised: A typical method is using “loss correction” that estimates the label transition matrix and corrects the loss of the samples in the mini-batch.

Original: Bootstrap (Reed et al., 2015) updates the network based on their own reconstruction-based objective with the notion of perceptual consistency.
Revised: Bootstrap (Reed et al., 2015) updates the network based on their own reconstruction-based objective with the notion of perceptual consistency.

Original: F-correction (Patrini et al., 2017) reweights the forward or backward loss of the training samples based on the label transition matrix estimated by a pre-trained normal network.
Revised: F-correction (Patrini et al., 2017) reweights the forward or backward loss of the training samples based on the label transition matrix estimated by a pre-trained normal network.

Original: R 3: D2L
Revised: R 3: D2L

Original: (Ma et al., 2018) employs a simple measure called local intrinsic dimensionality and then uses it to modify the forward loss in order to reduce the effects of false-labeled samples in learning.
Revised: (Ma et al., 2018) employs a simple measure called local intrinsic dimensionality and then uses it to modify the forward loss in order to reduce the effects of false-labeled samples in learning.

Original: Active Bias (Chang et al., 2017) heuristically evaluates uncertain samples with high prediction variances and then gives higher weights to their backward losses.
Revised: Active Bias (Chang et al., 2017) heuristically evaluates uncertain samples with high prediction variances and then gives higher weights to their backward losses.

Original: Ren et al.
Revised: Ren et al.

Original: (2018) include small clean validation data into the training data and re-weight the backward loss of the mini-batch samples such that the updated gradient minimizes the loss of those validation data.
Revised: (2018) include small clean validation data into the training data and re-weight the backward loss of the mini-batch samples such that the updated gradient minimizes the loss of those validation data.

Original: However, this family of methods is known to accumulate severe noise from the false correction , especially when the number of classes or the number of false-labeled samples is large (Han et al., 2018; Yu et al., 2019).
Revised: However, this family of methods is known to accumulate severe noise from the false correction , especially when the number of classes or the number of false-labeled samples is large (Han et al., 2018; Yu et al., 2019).

Original: To be free from the false correction, many recent researches have adopted “sample selection” that trains the network on selected samples.
Revised: To be free from the false correction, many recent researches have adopted “sample selection” that trains the network on selected samples.

Original: These methods attempt to select true-labeled samples from the noisy training data for use in updating the network.
Revised: These methods attempt to select true-labeled samples from the noisy training data for use in updating the network.

Original: Decouple (Malach & Shalev-Shwartz, 2017) maintains two networks simultaneously and updates the models only using the samples that have different label predictions from these two networks.
Revised: Decouple (Malach & Shalev-Shwartz, 2017) maintains two networks simultaneously and updates the models only using the samples that have different label predictions from these two networks.

Original: R 3: Wang et al.
Revised: R 3: Wang et al.

Original: (2018) proposed an iterative learning framework that learns deep discriminative features from well-classiﬁed noisy samples based on the local outlier factor algorithm (Breunig et al., 2000).
Revised: (2018) proposed an iterative learning framework that learns deep discriminative features from well-classiﬁed noisy samples based on the local outlier factor algorithm (Breunig et al., 2000).

Original: MentorNet (Jiang et al., 2018) introduces a collaborative learning paradigm that a pre-trained mentor network guides the training of a student network.
Revised: MentorNet (Jiang et al., 2018) introduces a collaborative learning paradigm that a pre-trained mentor network guides the training of a student network.

Original: Based on the small-loss criteria, the mentor network provides the student network with the samples whose label is probably correct.
Revised: Based on the small-loss criteria, the mentor network provides the student network with the samples whose label is probably correct.

Original: Co-teaching (Han et al., 2018) and Co-teaching+ (Yu et al., 2019) also maintain two networks, but each network selects a certain number of small-loss samples and feeds them to its peer network for further training.
Revised: Co-teaching (Han et al., 2018) and Co-teaching+ (Yu et al., 2019) also maintain two networks, but each network selects a certain number of small-loss samples and feeds them to its peer network for further training.

Original: Compared with Co-teaching , Co-teaching+ further employs the disagreement strategy of Decouple .
Revised: Compared with Co-teaching , Co-teaching+ further employs the disagreement strategy of Decouple .

Original: R 1: ITLM (Shen & Sanghavi, 2019) iteratively minimizes the trimmed loss by alternating between selecting a fraction of small-loss samples at current moment and retraining the network using them.
Revised: R 1: ITLM (Shen & Sanghavi, 2019) iteratively minimizes the trimmed loss by alternating between selecting a fraction of small-loss samples at current moment and retraining the network using them.

Original: INCV (Chen et al., 2019) randomly divides the noisy training data and then utilizes cross-validation to classify true-labeled samples with removing large-loss samples at each iteration.
Revised: INCV (Chen et al., 2019) randomly divides the noisy training data and then utilizes cross-validation to classify true-labeled samples with removing large-loss samples at each iteration.

Original: However, their general philosophy of selecting small-loss samples works well only in some cases such as symmetric noise.
Revised: However, their general philosophy of selecting small-loss samples works well only in some cases such as symmetric noise.

Original: RA : Differently to this family, we exploit the maximal safe set initially derived from the memorized samples at the early stop point.
Revised: RA : Differently to this family, we exploit the maximal safe set initially derived from the memorized samples at the early stop point.

Original: Most recently, a hybrid of “loss correction” and “sample selection” approaches was proposed by
Revised: Most recently, a hybrid of “loss correction” and “sample selection” approaches was proposed by

Original: Song et al.
Revised: Song et al.

Original: Their algorithm called SELFIE trains the network on selectively refurbished false-labeled samples together with small-loss samples.
Revised: Their algorithm called SELFIE trains the network on selectively refurbished false-labeled samples together with small-loss samples.

Original: SELFIE not only minimizes the number of falsely corrected samples but also exploits full exploration of the entire training data.
Revised: SELFIE not only minimizes the number of falsely corrected samples but also exploits full exploration of the entire training data.

Original: Its component for sample selection can be replaced by our method to further improve the performance.
Revised: Its component for sample selection can be replaced by our method to further improve the performance.

Original: RA : For the completeness of the survey, we mention the work that provides an empirical or  theoretical analysis of why early stopping matters in learning with the label noise.
Revised: RA : For the completeness of the survey, we mention the work that provides an empirical or a theoretical analysis on why early stopping helps learn with the label noise.

Original: Oymak et al.
Revised: Oymak et al.

Original: and Hendrycks et al.
Revised: and Hendrycks et al.

Original: (2019) have argued that early stopping is a suitable strategy because the network eventually begins to memorize all noisy samples by training for longer.
Revised: (2019) have argued that early stopping is a suitable strategy because the network eventually begins to memorize all noisy samples if it is trained too long.

Original: Li et al.
Revised: Li et al.

Original: (2019) have theoretically proved that the network memorizes false-labeled samples at a later stage of training, and thus claimed that the early stopped network is fairly robust to the label noise.
Revised: (2019) have theoretically proved that the network memorizes false-labeled samples at a later stage of training, and thus claimed that the early stopped network is fairly robust to the label noise.

Original: However, they did not mention how to take advantage of the early stopped network for further training.
Revised: However, they did not mention how to take advantage of the early stopped network for further training.

Original: Please note that Prestopping adopts early stopping to derive a seed for the maximal safe set, which is exploited to achieve noise-free training during the remaining learning period.
Revised: Please note that Prestopping adopts early stopping to derive a seed for the maximal safe set, which is exploited to achieve noise-free training during the remaining learning period.

Original: Thus, our novelty lies in the “merger” of early stopping and learning from the maximal safe set.
Revised: Thus, our novelty lies in the “merger” of early stopping and learning from the maximal safe set.

Original:  R OBUST TRAINING VIA Prestopping 3. P RELIMINARIES
Revised:  R OBUST TRAINING VIA Prestopping 3. P RELIMINARIES

Original: A k -class classiﬁcation problem requires the training data D
Revised: A k -class classiﬁcation problem requires the training data D

Original: = { x i , y ∗ i } Ni =1 , where x i is a sample and y ∗ i ∈ { 1 , 2 , . . .
Revised: = { x i , y ∗ i } Ni =1 , where x i is a sample and y ∗ i ∈ { 1 , 2 , . . .

Original: , k } is its true label.
Revised: , k } is its true label.

Original: Following the label noise scenario, let’s consider the noisy training data ˜ D
Revised: Following the label noise scenario, let’s consider the noisy training data ˜ D

Original: = { x i , ˜ y i } Ni =1 , where ˜ y i ∈ { 1 , 2 , . . .
Revised: = { x i , ˜ y i } Ni =1 , where ˜ y i ∈ { 1 , 2 , . . .

Original: , k } is a noisy label which may not be true.
Revised: , k } is a noisy label which may not be true.

Original: Then, in conventional training, when a mini-batch B t = { x i , ˜ y i } bi =1 consists of b samples randomly drawn from the noisy training data ˜ D at time t , the network parameter θ t is updated in the descent direction of the expected loss on the mini-batch B t as in Eq.
Revised: Then, in conventional training, when a mini-batch B t = { x i , ˜ y i } bi =1 consists of b samples randomly drawn from the noisy training data ˜ D at time t , the network parameter θ t is updated in the descent direction of the expected loss on the mini-batch B t as in Eq.

Original: (1), where α is a learning rate and L is a loss function.
Revised: (1), where α is a learning rate and L is a loss function.

Original: [Equation]
Revised: [Equation]

Original: As for the notion of network memorization, a sample x is deﬁned to be memorized by a network if the majority of its recent predictions at time t coincide with the given label, as in Deﬁnition 3.1.
Revised: As for the notion of network memorization, a sample x is deﬁned to be memorized by a network if the majority of its recent predictions at time t coincide with the given label, as in Deﬁnition 3.1.

Original: Deﬁnition 3.1.
Revised: Deﬁnition 3.1.

Original: (Memorized Sample)
Revised: (Memorized Sample)

Original: Let ˆ y t =
Revised: Let ˆ y t =

Original: Φ( x | θ t ) be the predicted label of a sample x at time t and H tx ( q )
Revised: Φ( x | θ t ) be the predicted label of a sample x at time t and H tx ( q )

Original: =
Revised: =

Original: { ˆ y t 1 , ˆ y t 2 , . . .
Revised: { ˆ y t 1 , ˆ y t 2 , . . .

Original: , ˆ y t q } be the history of the sample x that stores the predicted labels of the recent q times, where Φ is a neural network.
Revised: , ˆ y t q } be the history of the sample x that stores the predicted labels of the recent q times, where Φ is a neural network.

Original: Next, P ( y | x, t ; q ) is formulated such that it provides the probability of the label y ∈ { 1 , 2 , ..., k } estimated as the label of the sample x based on H tx as in
Revised: Next, P ( y | x, t ; q ) is formulated such that it provides the probability of the label y ∈ { 1 , 2 , ..., k } estimated as the label of the sample x based on H tx as in

Original: Eq.
Revised: Eq.

Original: (2), where [ · ] is the Iverson bracket 3 .
Revised: (2), where [ · ] is the Iverson bracket 3 .

Original: [Equation]
Revised: [Equation]

Original: Then, the sample x with its noisy label ˜ y is a memorized sample of the network with the parameter θ t at time t if the condition in Eq.
Revised: Then, the sample x with its noisy label ˜ y is a memorized sample of the network with the parameter θ t at time t if the condition in Eq.

Original: (3) holds.
Revised: (3) holds.

Original: [Equation]
Revised: [Equation]

Original: 3. T WO M AIN C OMPONENTS AND P HASES
Revised: 3. T WO M AIN C OMPONENTS AND P HASES

Original: The key idea of Prestopping is learning from a maximal safe set with an early stopped network.
Revised: The key idea of Prestopping is learning from a maximal safe set with an early stopped network.

Original: Thus, the two components of “early stopping” and “learning from the maximal safe set” respectively raise the questions about (Q1) when is the best point to early stop the training process?
Revised: Thus, the two components of “early stopping” and “learning from the maximal safe set” respectively raise the questions about (Q1) when is the best point to early stop the training process?

Original: and (Q2)
Revised: and (Q2)

Original: what is the maximal safe set to enable noise-free training during the remaining period?
Revised: what is the maximal safe set to enable noise-free training during the remaining period?

Original: 3.2. Q UESTION 1: B EST POINT TO EARLY STOP
Revised: 3.2. Q UESTION 1: B EST POINT TO EARLY STOP

Original: It is desirable to stop the training process at the point when the network (i) not only accumulates little noise from the false-labeled samples, (ii) but also acquires sufﬁcient information from the true-labeled ones.
Revised: It is desirable to stop the training process at the point when the network (i) not only accumulates little noise from the false-labeled samples, (ii) but also acquires sufﬁcient information from the true-labeled ones.

Original: Intuitively speaking, as indicated by the dashed line in Figure 3(a), the best stop point is the moment when label precision and label recall in Deﬁnition 3.2 cross with each other because it is the best trade-off between the two metrics.
Revised: Intuitively speaking, as indicated by the dashed line in Figure 3(a), the best stop point is the moment when label precision and label recall in Deﬁnition 3.2 cross with each other because it is the best trade-off between the two metrics.

Original: The period beyond this point is what we call the error-prone period because label precision starts decreasing rapidly.
Revised: The period beyond this point is what we call the error-prone period because label precision starts decreasing rapidly.

Original: If the ground truth y ∗ of a noisy label ˜
Revised: If the ground truth y ∗ of a noisy label ˜

Original: y is known, the best stop point can be easily calculated by these metrics.
Revised: y is known, the best stop point can be easily calculated by these metrics.

Original: Deﬁnition 3.2.
Revised: Deﬁnition 3.2.

Original: (Label Metrics)
Revised: (Label Metrics)

Original: Let M t ⊆
Revised: Let M t ⊆

Original: ˜ D be a set of memorized samples at time t .
Revised: ˜ D be a set of memorized samples at time t .

Original: Then, label precision (LP) and recall (LR) (Han et al., 2018) are formulated as Eq.
Revised: Then, label precision (LP) and recall (LR) (Han et al., 2018) are formulated as Eq.

Original: [Equation]
Revised: [Equation]

Original: However, it is not straightforward to ﬁnd the exact best stop point without the ground-truth labels.
Revised: However, it is not straightforward to ﬁnd the exact best stop point without the ground-truth labels.

Original: Hence, we present two practical heuristics of approximating the best stop point with minimal supervision .
Revised: Hence, we present two practical heuristics of approximating the best stop point with minimal supervision .

Original: These two heuristics require either a small clean validation set or a noise rate τ for the minimal supervision, where they are widely regarded as available in many studies (Veit et al., 2017; Ren et al., 2018; Han et al., 2018;
Revised: These two heuristics require either a small clean validation set or a noise rate τ for the minimal supervision, where they are widely regarded as available in many studies (Veit et al., 2017; Ren et al., 2018; Han et al., 2018;

Original: Yu et al., 2019; Song et al., 2019).
Revised: Yu et al., 2019; Song et al., 2019).

Original: [Figure]
Revised: [Figure]

Original: • Validation Heuristic: If a clean validation set is given, we stop training the network when the validation error is the lowest .
Revised: • Validation Heuristic: If a clean validation set is given, we stop training the network when the validation error is the lowest .

Original: It is reasonable to expect that the lowest validation error is achieved near the cross point; after that point (i
Revised: It is reasonable to expect that the lowest validation error is achieved near the cross point; after that point (i

Original: [Figure]
Revised: [Figure]

Original: e., in the error-prone period), the validation error likely increases because the network will be overﬁtted to the false-labeled samples.
Revised: e., in the error-prone period), the validation error likely increases because the network will be overﬁtted to the false-labeled samples.

Original: As shown in Figure 3(b), the estimated stop point is fairly close to the best stop point in both noise types.
Revised: As shown in Figure 3(b), the estimated stop point is fairly close to the best stop point in both noise types.

Original: • Noise-Rate Heuristic: If a noise rate τ is known, we stop training the network when the training error reaches τ × 100% .
Revised: • Noise-Rate Heuristic: If a noise rate τ is known, we stop training the network when the training error reaches τ × 100% .

Original: If we assume that all true-labeled samples of (1 − τ ) × 100% are memorized before any false-labeled samples of τ × 100%
Revised: If we assume that all true-labeled samples of (1 − τ ) × 100% are memorized before any false-labeled samples of τ × 100%

Original: (Arpit et al., 2017), this point indicates the cross point of label precision and label recall with their values to be all 1 .
Revised: (Arpit et al., 2017), this point indicates the cross point of label precision and label recall with their values to be all 1 .

Original: This heuristic tends to perform worse than the validation heuristic because the assumption does not hold perfectly.
Revised: This heuristic tends to perform worse than the validation heuristic because the assumption does not hold perfectly.

Original: We further discuss its quality in Appendix A.
Revised: We further discuss its quality in Appendix A.

Original: 3.2. Q UESTION 2: C RITERION OF A MAXIMAL SAFE SET
Revised: 3.2. Q UESTION 2: C RITERION OF A MAXIMAL SAFE SET

Original: Because the network is early stopped at the (estimated) best point, the set of memorized samples at that time is quantitatively sufﬁcient and qualitatively less noisy.
Revised: Because the network is early stopped at the (estimated) best point, the set of memorized samples at that time is quantitatively sufﬁcient and qualitatively less noisy.

Original: That is, it can be used as a safe and effective training set to resume the training of the early stopped network without accumulating the label noise.
Revised: That is, it can be used as a safe and effective training set to resume the training of the early stopped network without accumulating the label noise.

Original: Based on this intuition, we deﬁne a maximal safe set in Deﬁnition 3.3, which is initially derived from the memorized samples at the early stop point and gradually increased as well as puriﬁed along with the network’s learning progress.
Revised: Based on this intuition, we deﬁne a maximal safe set in Deﬁnition 3.3, which is initially derived from the memorized samples at the early stop point and gradually increased as well as puriﬁed along with the network’s learning progress.

Original: In each mini-batch B t , the network parameter θ t is updated using the current maximal safe set S t as in Eq.
Revised: In each mini-batch B t , the network parameter θ t is updated using the current maximal safe set S t as in Eq.

Original: (5), and subsequently a more reﬁned maximal safe set S t +1 is derived by the updated network.
Revised: (5), and subsequently a more reﬁned maximal safe set S t +1 is derived by the updated network.

Original: Deﬁnition 3.3.
Revised: Deﬁnition 3.3.

Original: (Maximal Safe Set) Let t stop be the early stop point.
Revised: (Maximal Safe Set) Let t stop be the early stop point.

Original: A maximal safe set S t at time t is deﬁned to be the set of the memorized samples of the network Φ( x ; θ t ) when t ≥ t stop .
Revised: A maximal safe set S t at time t is deﬁned to be the set of the memorized samples of the network Φ( x ; θ t ) when t ≥ t stop .

Original: The network Φ( x ; θ t ) at t = t stop is the early stopped network, i.e., S t stop =
Revised: The network Φ( x ; θ t ) at t = t stop is the early stopped network, i.e., S t stop =

Original: M t stop , and the network Φ( x ; θ t ) at t > t stop is obtained by Eq.
Revised: M t stop , and the network Φ( x ; θ t ) at t > t stop is obtained by Eq.

Original: [Equation]
Revised: [Equation]

Original: Figure 4 shows the main advantage of learning from the maximal safe set during the remaining period.
Revised: Figure 4 shows the main advantage of learning from the maximal safe set during the remaining period.

Original: Even when the noise rate is quite high (e.g., 40% ), this learning paradigm exploits most of true-labeled samples, in considering that the label recall of the maximal safe set was maintained very high in both CIFAR data sets regardless of the noise type.
Revised: Even when the noise rate is quite high (e.g., 40% ), this learning paradigm exploits most of true-labeled samples, in considering that the label recall of the maximal safe set was maintained very high in both CIFAR data sets regardless of the noise type.

Original: The label recall was maintained over 0 .
Revised: The label recall was maintained over 0 .

Original: 81 in symmetric noise and over 0 .
Revised: 81 in symmetric noise and over 0 .

Original: 84 in pair noise even in CIFAR-100 with pair noise after the 80th epoch.
Revised: 84 in pair noise even in CIFAR-100 with pair noise after the 80th epoch.

Original: Further, by excluding unsafe samples that might accumulate the label noise, the label precision of the maximal safe set tended to increase rapidly at the beginning of the remaining period; it was maintained over 0 .
Revised: Further, by excluding unsafe samples that might accumulate the label noise, the label precision of the maximal safe set tended to increase rapidly at the beginning of the remaining period; it was maintained over 0 .

Original: 98 in symmetric noise and over 0 .
Revised: 98 in symmetric noise and over 0 .

Original: 94 even in CIFAR-100 with pair noise after the 80th epoch, which could not be realized by the small-loss trick (Han et al., 2018; Song et al., 2019; Shen & Sanghavi, 2019; Chen et al., 2019).
Revised: 94 even in CIFAR-100 with pair noise after the 80th epoch, which could not be realized by the small-loss trick (Han et al., 2018; Song et al., 2019; Shen & Sanghavi, 2019; Chen et al., 2019).

Original: R 2: In Section 5.3, we have empirically veriﬁed that the maximal safe set is superior to the selected samples based on the small-loss trick.
Revised: R 2: In Section 5.3, we have empirically veriﬁed that the maximal safe set is superior to the selected samples based on the small-loss trick.

Original: [Figure]
Revised: [Figure]

Original: Label precision
Revised: Label precision

Original: [Figure]
Revised: [Figure]

Original: label recall on CIFAR-10 (
Revised: label recall on CIFAR-10 (

Original: [Figure]
Revised: [Figure]

Original: Validation_Error( V , θ t ); /
Revised: Validation_Error( V , θ t ); /

Original: * A validation error at time t */ 8: if isMin ( val _ err ) then θ t stop ← θ t ; /*Save the network when val _ error is the lowest*/ 9: t ← t + 1; 10: θ t ← θ t stop ; /* Load the network stopped at t stop */ 11: for i = stop _ epoch to epochs do /* Phase II: Learning from a maximal safe set */ 12: for j = 1 to | ˜ D| / |B t | do 13:
Revised: * A validation error at time t */ 8: if isMin ( val _ err ) then θ t stop ← θ t ; /*Save the network when val _ error is the lowest*/ 9: t ← t + 1; 10: θ t ← θ t stop ; /* Load the network stopped at t stop */ 11: for i = stop _ epoch to epochs do /* Phase II: Learning from a maximal safe set */ 12: for j = 1 to | ˜ D| / |B t | do 13:

Original: Draw a mini-batch B t from ˜ D ; 14:
Revised: Draw a mini-batch B t from ˜ D ; 14:

Original: S t ← { x | argmax y P ( y | x, t ; q ) = ˜ y } ; /
Revised: S t ← { x | argmax y P ( y | x, t ; q ) = ˜ y } ; /

Original: * A maximal safe set in Deﬁnition 3.3 */ 15: θ t +

Revised: * A maximal safe set in Deﬁnition 3.3 */ 15: θ t +

Original: =
Revised: =

Original: θ t − α ∇ (cid:0) 1 |S t ∩B t | (cid:80) x ∈S t ∩B t L ( x, ˜ y ; θ t ) (cid:1) ; /* Update by Eq.
Revised: θ t − α ∇ (cid:0) 1 |S t ∩B t | (cid:80) x ∈S t ∩B t L ( x, ˜ y ; θ t ) (cid:1) ; /* Update by Eq.

Original: (5) */ 16: t ← t + 1; 17: return θ t , S t ;
Revised: (5) */ 16: t ← t + 1; 17: return θ t , S t ;

Original:  A LGORITHM D ESCRIPTION 4. M AIN A LGORITHM : Prestopping WITH V ALIDATION H EURISTIC
Revised:  A LGORITHM D ESCRIPTION 4. M AIN A LGORITHM : Prestopping WITH V ALIDATION H EURISTIC

Original: Algorithm 1 describes the overall procedure of Prestopping with the validation heuristic, which is self-explanatory.
Revised: Algorithm 1 describes the overall procedure of Prestopping with the validation heuristic, which is self-explanatory.

Original: First, the network is trained on the noisy training data ˜ D in the default manner (Lines 3 – 6 ).
Revised: First, the network is trained on the noisy training data ˜ D in the default manner (Lines 3 – 6 ).

Original: During this ﬁrst phase, the validation data V is used to evaluate the best point for the early stop, and the network parameter is saved at the time of the lowest validation error (Lines 7 – 8 ).
Revised: During this ﬁrst phase, the validation data V is used to evaluate the best point for the early stop, and the network parameter is saved at the time of the lowest validation error (Lines 7 – 8 ).

Original: Then, during the second phase, Prestopping continues to train the early stopped network for the remaining learning period (Lines 10 – 12 ).
Revised: Then, during the second phase, Prestopping continues to train the early stopped network for the remaining learning period (Lines 10 – 12 ).

Original: Here, the maximal safe set S t at the current moment is retrieved, and each sample x ∈ S t ∩ B t is used to update the network parameter.
Revised: Here, the maximal safe set S t at the current moment is retrieved, and each sample x ∈ S t ∩ B t is used to update the network parameter.

Original: The mini-batch samples not included in S t are no longer used in pursuit of robust learning (Lines 14 – 15 ).
Revised: The mini-batch samples not included in S t are no longer used in pursuit of robust learning (Lines 14 – 15 ).

Original: 4. C OLLABORATION WITH S AMPLE R EFURBISHMENT : Prestopping+
Revised: 4. C OLLABORATION WITH S AMPLE R EFURBISHMENT : Prestopping+

Original: To further improve the performance of Prestopping , we introduce Prestopping+ that employs the concept of selectively refurbishing false-labeled samples in SELFIE (Song et al., 2019).
Revised: To further improve the performance of Prestopping , we introduce Prestopping+ that employs the concept of selectively refurbishing false-labeled samples in SELFIE (Song et al., 2019).

Original: In detail, the ﬁnal maximal safe set S t end is retrieved from the ﬁrst run of Prestopping , and then it is used as the true-labeled set for the next run of SELFIE with initializing the network parameter.
Revised: In detail, the ﬁnal maximal safe set S t end is retrieved from the ﬁrst run of Prestopping , and then it is used as the true-labeled set for the next run of SELFIE with initializing the network parameter.

Original: Following the update principle of SELFIE , the modiﬁed gradient update rule in Eq.
Revised: Following the update principle of SELFIE , the modiﬁed gradient update rule in Eq.

Original: (6) is used to train the network.
Revised: (6) is used to train the network.

Original: For each sample x in the mini-batch B t , the given label ˜
Revised: For each sample x in the mini-batch B t , the given label ˜

Original: y of x is selectively refurbished into y refurb if it can be corrected with high precision.
Revised: y of x is selectively refurbished into y refurb if it can be corrected with high precision.

Original: However, the mini-batch samples included in S t end are omitted from the refurbishment because their label is already highly credible.
Revised: However, the mini-batch samples included in S t end are omitted from the refurbishment because their label is already highly credible.

Original: Then, the mini-batch samples in the refurbished set R t are provided to update the network together with those in the ﬁnal maximal safe set S t end .
Revised: Then, the mini-batch samples in the refurbished set R t are provided to update the network together with those in the ﬁnal maximal safe set S t end .

Original: Refer to Song et al.
Revised: Refer to Song et al.

Original: (2019)’s work for more details.
Revised: (2019)’s work for more details.

Original: [Equation]
Revised: [Equation]

Original:  E VALUATION
Revised:  E VALUATION

Original: Data Sets : To verify the superiority of Prestopping , we performed an image classiﬁcation task on four benchmark data sets: CIFAR-10 ( 10 classes) 4 and CIFAR-100 ( 100 classes) 4 , a subset of 80 million categorical images (Krizhevsky et al., 2014); ANIMAL-10N ( 10 classes) 5 , a real-world noisy data of human-labeled online images for confusing animals (Song et al., 2019); FOOD-101N ( 101 classes) 6 , a real-world noisy data of crawled food images annotated by their search keywords in the FOODtaxonomy (Bossard et al., 2014; Lee et al., 2018).
Revised: Data Sets : To verify the superiority of Prestopping , we performed an image classiﬁcation task on four benchmark data sets: CIFAR-10 ( 10 classes) 4 and CIFAR-100 ( 100 classes) 4 , a subset of 80 million categorical images (Krizhevsky et al., 2014); ANIMAL-10N ( 10 classes) 5 , a real-world noisy data of human-labeled online images for confusing animals (Song et al., 2019); FOOD-101N ( 101 classes) 6 , a real-world noisy data of crawled food images annotated by their search keywords in the FOODtaxonomy (Bossard et al., 2014; Lee et al., 2018).

Original: We did not apply any data augmentation.
Revised: We did not apply any data augmentation.

Original: Noise Injection : As all labels in the CIFAR data sets are clean, we artiﬁcially corrupted the labels in these data sets using typical methods for the evaluation of synthetic noises (Ren et al., 2018; Han et al., 2018; Yu et al., 2019).
Revised: Noise Injection : As all labels in the CIFAR data sets are clean, we artiﬁcially corrupted the labels in these data sets using typical methods for the evaluation of synthetic noises (Ren et al., 2018; Han et al., 2018; Yu et al., 2019).

Original: For k classes, we applied the label transition matrix T : (i) symmetric noise : ∀ j (cid:54) =
Revised: For k classes, we applied the label transition matrix T : (i) symmetric noise : ∀ j (cid:54) =

Original: i T ij = τk − 1 and (ii) pair noise : ∃ j (cid:54) =
Revised: i T ij = τk − 1 and (ii) pair noise : ∃ j (cid:54) =

Original: i T ij =
Revised: i T ij =

Original: τ ∧ ∀ k (cid:54) =
Revised: τ ∧ ∀ k (cid:54) =

Original: i,k (cid:54) =
Revised: i,k (cid:54) =

Original: j T ik = 0 , where T ij is the probability of the true label i being ﬂipped to the corrupted label j and τ is the noise rate.
Revised: j T ik = 0 , where T ij is the probability of the true label i being ﬂipped to the corrupted label j and τ is the noise rate.

Original: For the pair noise, the corrupted label j was set to be the next label of the true label i following the recent work (Yu et al., 2019; Song et al., 2019).
Revised: For the pair noise, the corrupted label j was set to be the next label of the true label i following the recent work (Yu et al., 2019; Song et al., 2019).

Original: To evaluate the robustness on varying noise rates from light noise to heavy noise, we tested ﬁve noise rates τ ∈ { 0 . 0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 } .
Revised: To evaluate the robustness on varying noise rates from light noise to heavy noise, we tested ﬁve noise rates τ ∈ { 0 . 0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 } .

Original: In contrast, we did not inject any label noise into ANIMAL-10N and FOOD-101N 7 because they contain real label noise estimated at 8 . 0% and 18 .
Revised: In contrast, we did not inject any label noise into ANIMAL-10N and FOOD-101N 7 because they contain real label noise estimated at 8 . 0% and 18 .

Original: 4% respectively (Lee et al., 2018; Song et al., 2019).
Revised: 4% respectively (Lee et al., 2018; Song et al., 2019).

Original: Clean Validation Data : Recall that a clean validation set is needed for the validation heuristic in
Revised: Clean Validation Data : Recall that a clean validation set is needed for the validation heuristic in

Original: Section 3.2.1.
Revised: Section 3.2.1.

Original: As for ANIMAL-10N and Food-101N, we did exploit their own clean validation data; 5 , 000 and 3 , 824 images, respectively, were included in their validation set.
Revised: As for ANIMAL-10N and Food-101N, we did exploit their own clean validation data; 5 , 000 and 3 , 824 images, respectively, were included in their validation set.

Original: However, as no validation data exists for the CIFAR data sets, we constructed a small clean validation set by randomly selecting 1 , 000 images from the original training data of 50 , 000 images.
Revised: However, as no validation data exists for the CIFAR data sets, we constructed a small clean validation set by randomly selecting 1 , 000 images from the original training data of 50 , 000 images.

Original: Please note that the noise injection process was applied to only the rest 49 , 000 training images.
Revised: Please note that the noise injection process was applied to only the rest 49 , 000 training images.

Original: Networks and Hyperparameters :
Revised: Networks and Hyperparameters :

Original: For the classiﬁcation task, we trained DenseNet (L= 40 , k= 12 ) and VGG-19 with a momentum optimizer.
Revised: For the classiﬁcation task, we trained DenseNet (L= 40 , k= 12 ) and VGG-19 with a momentum optimizer.

Original: Speciﬁcally, we used a momentum of 0 .
Revised: Speciﬁcally, we used a momentum of 0 .

Original: 9 , a batch size of 128 , a dropout of 0 . 1 , and batch normalization.
Revised: 9 , a batch size of 128 , a dropout of 0 . 1 , and batch normalization.

Original: Prestopping has only one unique hyperparameter, the history length q , and it was set to be 10 , which was the best value found by the grid search (see
Revised: Prestopping has only one unique hyperparameter, the history length q , and it was set to be 10 , which was the best value found by the grid search (see

Original: Appendix B.1 for details).
Revised: Appendix B.1 for details).

Original: The hyperparameters used in the compared algorithms were favorably set to be the best values presented in the original papers.
Revised: The hyperparameters used in the compared algorithms were favorably set to be the best values presented in the original papers.

Original: As for the training schedule, we trained the network for 120 epochs and used an initial learning rate 0 . 1 , which was divided by 5 at 50% and 75% of the total number of epochs.
Revised: As for the training schedule, we trained the network for 120 epochs and used an initial learning rate 0 . 1 , which was divided by 5 at 50% and 75% of the total number of epochs.

Original: Algorithms : We compared Prestopping and Prestopping+ with not only a baseline algorithm but also the four state-of-the-art robust training algorithms: Default trains the network without any processing for the label noise; Active Bias re-weights the backward loss of training samples based on their prediction variance; Co-teaching selects a certain number of small-loss samples to train the network based on the co-training (Blum & Mitchell, 1998); Co-teaching+ is similar to Co-teaching , but its small-loss samples are selected from the disagreement set; SELFIE selectively refurbishes noisy samples and exploits them together with the small-loss samples.
Revised: Algorithms : We compared Prestopping and Prestopping+ with not only a baseline algorithm but also the four state-of-the-art robust training algorithms: Default trains the network without any processing for the label noise; Active Bias re-weights the backward loss of training samples based on their prediction variance; Co-teaching selects a certain number of small-loss samples to train the network based on the co-training (Blum & Mitchell, 1998); Co-teaching+ is similar to Co-teaching , but its small-loss samples are selected from the disagreement set; SELFIE selectively refurbishes noisy samples and exploits them together with the small-loss samples.

Original: All the algorithms were implemented
Revised: All the algorithms were implemented

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: CNNs on two data sets with varying symmetric noise rates.
Revised: CNNs on two data sets with varying symmetric noise rates.

Original: using TensorFlow 1 . 8 . 0 and executed using 16 NVIDIA Titan Volta GPUs.
Revised: using TensorFlow 1 . 8 . 0 and executed using 16 NVIDIA Titan Volta GPUs.

Original: For reproducibility, we provide the source code at https://bit.ly/2l3g9Jx .
Revised: For reproducibility, we provide the source code at https://bit.ly/2l3g9Jx .

Original: R 2: In support of reliable evaluation, we repeated every task thrice and reported the average and standard error of the best test error, which was evaluated using the network at the time of the lowest validation error.
Revised: R 2: In support of reliable evaluation, we repeated every task thrice and reported the average and standard error of the best test error, which was evaluated using the network at the time of the lowest validation error.

Original: Thus, the clean validation set was used to select the best model in all algorithms during the training procedure.
Revised: Thus, the clean validation set was used to select the best model in all algorithms during the training procedure.

Original: 5. P ERFORMANCE C OMPARISON
Revised: 5. P ERFORMANCE C OMPARISON

Original: Figures 5 and 6 show the test error of the seven training methods using two CNNs on two data sets with varying pair and symmetric noise rates, respectively.
Revised: Figures 5 and 6 show the test error of the seven training methods using two CNNs on two data sets with varying pair and symmetric noise rates, respectively.

Original: The results of Prestopping and Prestopping+ in Section 5 were obtained using the validation heuristic.
Revised: The results of Prestopping and Prestopping+ in Section 5 were obtained using the validation heuristic.

Original: See Appendix A for the results of the noise-rate heuristic.
Revised: See Appendix A for the results of the noise-rate heuristic.

Original: In order to highlight the improvement of Prestopping and Prestopping+ over the other methods, their lines are dark colored.
Revised: In order to highlight the improvement of Prestopping and Prestopping+ over the other methods, their lines are dark colored.

Original: Figure 7 shows the test error on two real-world noisy data sets with different noise rates.
Revised: Figure 7 shows the test error on two real-world noisy data sets with different noise rates.

Original: The best test errors are detailed in Tables 1 and 2 of Appendix C. RA :
Revised: The best test errors are detailed in Tables 1 and 2 of Appendix C. RA :

Original: In addition, more experimental results on Tiny-ImageNet and Clothings  data sets are discussed in Appendix B.2 and B.3.
Revised: In addition, more experimental results on Tiny-ImageNet and Clothings (Xiao et al., 2015) data sets are discussed in Appendix B.2 and B.3.

Original: 5.1. R ESULT WITH P AIR N OISE (F IGURE 5)
Revised: 5.1. R ESULT WITH P AIR N OISE (F IGURE 5)

Original: The performance trend in the two network architectures was similar with each other.
Revised: The performance trend in the two network architectures was similar with each other.

Original: In general, either Prestopping or Prestopping+ achieved the lowest test error in a wide range of noise rates on both CIFAR data sets.
Revised: In general, either Prestopping or Prestopping+ achieved the lowest test error in a wide range of noise rates on both CIFAR data sets.

Original: With help of the refurbished samples, Prestopping+ achieved a slightly better performance than Prestopping in CIFAR-10.
Revised: With help of the refurbished samples, Prestopping+ achieved a slightly better performance than Prestopping in CIFAR-10.

Original: However, an opposite trend was observed in CIFAR-100; this phenomenon was due to a large number of falsely corrected labels under the pair noise, especially when the number of classes is large (see Appendix B.4 for details).
Revised: However, an opposite trend was observed in CIFAR-100; this phenomenon was due to a large number of falsely corrected labels under the pair noise, especially when the number of classes is large (see Appendix B.4 for details).

Original: Although SELFIE achieved relatively lower test error among the existing methods, the test error of SELFIE was still worse than that of Prestopping .
Revised: Although SELFIE achieved relatively lower test error among the existing methods, the test error of SELFIE was still worse than that of Prestopping .

Original: Co-teaching did not work well because many false-labeled samples were misclassiﬁed as clean ones; Co-teaching+ was shown to be even worse than Co-teaching despite it being an improvement of Co-teaching (see Section 5.4 for details).
Revised: Co-teaching did not work well because many false-labeled samples were misclassiﬁed as clean ones; Co-teaching+ was shown to be even worse than Co-teaching despite it being an improvement of Co-teaching (see Section 5.4 for details).

Original: The test error of Active Bias was not comparable to that of Prestopping .
Revised: The test error of Active Bias was not comparable to that of Prestopping .

Original: The performance improvement of ours over the others increased as the label noise became heavier.
Revised: The performance improvement of ours over the others increased as the label noise became heavier.

Original: In particular, at a heavy noise rate of 40% , Prestopping or Prestopping+ signiﬁcantly reduced the absolute test error by 2 . 2 pp – 18 . 1 pp compared with the other robust methods.
Revised: In particular, at a heavy noise rate of 40% , Prestopping or Prestopping+ signiﬁcantly reduced the absolute test error by 2 . 2 pp – 18 . 1 pp compared with the other robust methods.

Original: 5.1. R ESULT WITH S YMMETRIC N OISE (F IGURE 6)
Revised: 5.1. R ESULT WITH S YMMETRIC N OISE (F IGURE 6)

Original: Similar to the pair noise, both Prestopping and Prestopping+ generally outperformed the other methods.
Revised: Similar to the pair noise, both Prestopping and Prestopping+ generally outperformed the other methods.

Original: In particular, the performance of Prestopping+ was the best at any noise rate on all data sets, because the synergistic effect was higher in symmetric noise than in pair noise.
Revised: In particular, the performance of Prestopping+ was the best at any noise rate on all data sets, because the synergistic effect was higher in symmetric noise than in pair noise.

Original: Quantitatively, at a heavy noise rate of 40% , our methods showed signiﬁcant reduction in the absolute test error
Revised: Quantitatively, at a heavy noise rate of 40% , our methods showed signiﬁcant reduction in the absolute test error

Original: [Figure]
Revised: [Figure]

Original: Co
Revised: Co

Original: [Figure]
Revised: [Figure]

Original: teaching
Revised: teaching

Original: Co
Revised: Co

Original: [Figure]
Revised: [Figure]

Original: teaching+
Revised: teaching+

Original: SELFIE
Revised: SELFIE

Original: LESSON
Revised: LESSON

Original: LESSON+
Revised: LESSON+

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: by 0 . 3 pp – 17 . 5 pp compared with the other robust methods.
Revised: by 0 . 3 pp – 17 . 5 pp compared with the other robust methods.

Original: Unlike the pair noise, Co-teaching and
Revised: Unlike the pair noise, Co-teaching and

Original: SELFIE achieved a low test error comparable to Prestopping because true-labeled samples could be well separated from false-labeled ones by their small-loss criteria; hence, the dominance between
Revised: SELFIE achieved a low test error comparable to Prestopping because true-labeled samples could be well separated from false-labeled ones by their small-loss criteria; hence, the dominance between

Original: Co-teaching and Active Bias was reversed so that the former slightly outperformed the latter.
Revised: Co-teaching and Active Bias was reversed so that the former slightly outperformed the latter.

Original: 5.1. R ESULT WITH R EAL -W ORLD N OISE (F IGURE 7)
Revised: 5.1. R ESULT WITH R EAL -W ORLD N OISE (F IGURE 7)

Original: Both Prestopping and Prestopping+ maintained their dominance over the other methods under real-world label noise as well.
Revised: Both Prestopping and Prestopping+ maintained their dominance over the other methods under real-world label noise as well.

Original: Prestopping+ achieved the lowest test error when the number of classes is small (e.g., ANIMAL-10N), while Prestopping was the best when the number of classes is large (e.g., FOOD-101N) owing to the difﬁculty in label correction of Prestopping+ .
Revised: Prestopping+ achieved the lowest test error when the number of classes is small (e.g., ANIMAL-10N), while Prestopping was the best when the number of classes is large (e.g., FOOD-101N) owing to the difﬁculty in label correction of Prestopping+ .

Original: (Thus, practitioners are recommended to choose between Prestopping and Prestopping+ depending on the number of classes in hand.)
Revised: (Thus, practitioners are recommended to choose between Prestopping and Prestopping+ depending on the number of classes in hand.)

Original: Speciﬁcally, they improved the absolute test error by 0 .
Revised: Speciﬁcally, they improved the absolute test error by 0 .

Original: 4 pp – 4 . 

Revised: 4 pp – 4 .

Original: pp and 0 .
Revised: pp and 0 .

Original: 5 pp – 8 .
Revised: 5 pp – 8 .

Original: pp in ANIMAL-10N and FOOD-101N, respectively.
Revised: pp in ANIMAL-10N and FOOD-101N, respectively.

Original: Therefore, we believe that the advantage of our methods is unquestionable even in the real-world scenarios.
Revised: Therefore, we believe that the advantage of our methods is unquestionable even in the real-world scenarios.

Original: 5. C ONVERGENCE A NALYSIS ON C LEAN V ALIDATION D ATA (F IGURE 8)
Revised: 5. C ONVERGENCE A NALYSIS ON C LEAN V ALIDATION D ATA (F IGURE 8)

Original: To verify that the error-prone period is eliminated, we plot the convergence of validation error for the seven training methods.
Revised: To verify that the error-prone period is eliminated, we plot the convergence of validation error for the seven training methods.

Original: The validation error of Default , Active Bias , and Co-teaching+ ﬁrst reached their lowest values before the error-prone period, and then increased rapidly in the error-prone period.
Revised: The validation error of Default , Active Bias , and Co-teaching+ ﬁrst reached their lowest values before the error-prone period, and then increased rapidly in the error-prone period.

Original: On the other hand, the remaining methods kept reducing the validation error even in that period.
Revised: On the other hand, the remaining methods kept reducing the validation error even in that period.

Original: Notably, Prestopping+ achieved the lowest validation error at the end, owing to the successful merger of the advanced sample selection in Prestopping and the sample refurbishment in SELFIE .
Revised: Notably, Prestopping+ achieved the lowest validation error at the end, owing to the successful merger of the advanced sample selection in Prestopping and the sample refurbishment in SELFIE .

Original: In addition, the performance of Prestopping was shown to be almost the same as and comparable to that of Prestopping+ in pair and symmetric noises, respectively.
Revised: In addition, the performance of Prestopping was shown to be almost the same as and comparable to that of Prestopping+ in pair and symmetric noises, respectively.

Original: The validation error of SELFIE also converged faster than those of the other existing methods, but it was still inferior to Prestopping+ .
Revised: The validation error of SELFIE also converged faster than those of the other existing methods, but it was still inferior to Prestopping+ .

Original: Thus, these results conﬁrm that our two methods effectively overcome the error-prone period.
Revised: Thus, these results conﬁrm that our two methods effectively overcome the error-prone period.

Original: 5. A CCURACY OF S AMPLE S ELECTION (F IGURE 9)
Revised: 5. A CCURACY OF S AMPLE S ELECTION (F IGURE 9)

Original: RA : Figure 9 shows the average accuracy of sample selection for two different selection strategies using DenseNet.
Revised: RA : Figure 9 shows the average accuracy of sample selection for two different selection strategies using DenseNet.

Original: Average accuracy is deﬁned as the average ratio of the true-labeled samples in selected mini-batch samples during the training process.
Revised: Average accuracy is deﬁned as the average ratio of the true-labeled samples in selected mini-batch samples during the training process.

Original: As mentioned earlier in Section 1, because the loss distribution of false-labeled samples was overlapped closely with that of true-labeled ones in pair noise, the average accuracy of the small-loss trick was worse with pair noise than with symmetric noise; it decreased to 75 .
Revised: As mentioned earlier in Section 1, because the loss distribution of false-labeled samples was overlapped closely with that of true-labeled ones in pair noise, the average accuracy of the small-loss trick was worse with pair noise than with symmetric noise; it decreased to 75 .

Original: 0% as the noise rate increased in CIFAR-100 with pair noise.
Revised: 0% as the noise rate increased in CIFAR-100 with pair noise.

Original: On the other hand, the maximal safe set maintained its dominance over the small-loss trick in all data sets regardless of the noise type.
Revised: On the other hand, the maximal safe set maintained its dominance over the small-loss trick in all data sets regardless of the noise type.

Original: In particular, the average accuracy of the maximal safe set was 93 .
Revised: In particular, the average accuracy of the maximal safe set was 93 .

Original: Default
Revised: Default

Original: Active Bias
Revised: Active Bias

Original: Co-teaching
Revised: Co-teaching

Original: Co-teaching+
Revised: Co-teaching+

Original: SELFIE
Revised: SELFIE

Original: LESSON
Revised: LESSON

Original: LESSON+
Revised: LESSON+

Original: Default
Revised: Default

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: even in CIFAR-100 with pair noise of 40% , which led to the huge performance improvement of
Revised: even in CIFAR-100 with pair noise of 40% , which led to the huge performance improvement of

Original: Prestopping over Co-teaching in Figure 5(b).
Revised: Prestopping over Co-teaching in Figure 5(b).

Original: 5. A NATOMY OF Co-teaching+ (F IGURE 10)
Revised: 5. A NATOMY OF Co-teaching+ (F IGURE 10)

Original: Although Co-teaching+ is the latest method, its performance was worse than expected, as shown in
Revised: Although Co-teaching+ is the latest method, its performance was worse than expected, as shown in

Original: Section 5.1.
Revised: Section 5.1.

Original: Thus, we looked into Co-teaching+ in more detail.
Revised: Thus, we looked into Co-teaching+ in more detail.

Original: A poor performance of Co-teaching+ was attributed to the fast consensus of the label predictions for true-labeled samples, especially when training a complex network.
Revised: A poor performance of Co-teaching+ was attributed to the fast consensus of the label predictions for true-labeled samples, especially when training a complex network.

Original: In other words, because two complex networks in Co-teaching+ start making the same predictions for true-labeled samples too early, these samples are excluded too early.
Revised: In other words, because two complex networks in Co-teaching+ start making the same predictions for true-labeled samples too early, these samples are excluded too early.

Original: As shown in Figures 10(a) and 10(b), the disagreement ratio with regard to the true-labeled samples dropped faster with a complex network than with a simple network, in considering that the ratio drastically decreased to 49 .
Revised: As shown in Figures 10(a) and 10(b), the disagreement ratio with regard to the true-labeled samples dropped faster with a complex network than with a simple network, in considering that the ratio drastically decreased to 49 .

Original: 8% during the ﬁrst 5 epochs.
Revised: 8% during the ﬁrst 5 epochs.

Original: Accordingly, it is evident that Co-teaching+ with a complex network causes a narrow exploration of the true-labeled samples, and the selection accuracy of Co-teaching+ naturally degraded from 60 .
Revised: Accordingly, it is evident that Co-teaching+ with a complex network causes a narrow exploration of the true-labeled samples, and the selection accuracy of Co-teaching+ naturally degraded from 60 .

Original: 4% to 44 .
Revised: 4% to 44 .

Original: 8% for that reason, as shown in
Revised: 8% for that reason, as shown in

Original: Figure 10(c).
Revised: Figure 10(c).

Original: Therefore, we conclude that Co-teaching+ may not suit a complex network.
Revised: Therefore, we conclude that Co-teaching+ may not suit a complex network.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we proposed a novel two-phase training strategy for the noisy training data, which we call Prestopping .
Revised: In this paper, we proposed a novel two-phase training strategy for the noisy training data, which we call Prestopping .

Original: The ﬁrst phase, “early stopping,” retrieves an initial set of true-labeled samples as many as possible, and the second phase, “learning from a maximal safe set,” completes the rest training process only using the true-labeled samples with high precision.
Revised: The ﬁrst phase, “early stopping,” retrieves an initial set of true-labeled samples as many as possible, and the second phase, “learning from a maximal safe set,” completes the rest training process only using the true-labeled samples with high precision.

Original: Prestopping can be easily applied to many real-world cases because it additionally requires only either a small clean validation set or a noise rate.
Revised: Prestopping can be easily applied to many real-world cases because it additionally requires only either a small clean validation set or a noise rate.

Original: Furthermore, we combined this novel strategy with sample refurbishment to develop Prestopping+ .
Revised: Furthermore, we combined this novel strategy with sample refurbishment to develop Prestopping+ .

Original: Through extensive experiments using various real-world and simulated noisy data sets, we veriﬁed that either Prestopping or Prestopping+ achieved the lowest test error among the seven compared methods, thus signiﬁcantly improving the robustness to diverse types of label noise.
Revised: Through extensive experiments using various real-world and simulated noisy data sets, we veriﬁed that either Prestopping or Prestopping+ achieved the lowest test error among the seven compared methods, thus signiﬁcantly improving the robustness to diverse types of label noise.

Original: Overall, we believe that our work of dividing the training process into two phases by early stopping is a new direction for robust training and can trigger a lot of subsequent studies.
Revised: Overall, we believe that our work of dividing the training process into two phases by early stopping is a new direction for robust training and can trigger a lot of subsequent studies.

Original: A Prestopping WITH N OISE -R ATE H EURISTIC
Revised: A Prestopping WITH N OISE -R ATE H EURISTIC

Original: Algorithm 2 describes the overall procedure of Prestopping with the noise-rate heuristic, which is also self-explanatory.
Revised: Algorithm 2 describes the overall procedure of Prestopping with the noise-rate heuristic, which is also self-explanatory.

Original: Compared with Algorithm 1, only the way of determining the best stop point in Lines 7 – 9 has changed.
Revised: Compared with Algorithm 1, only the way of determining the best stop point in Lines 7 – 9 has changed.

Original: I NPUT : ˜ D : data, epochs : total number epochs, q :
Revised: I NPUT : ˜ D : data, epochs : total number epochs, q :

Original: O UTPUT : θ t : network parameters 1: t ← 1; θ t ← Initialize the network parameter ; 2: θ t stop ← ∅ ; /
Revised: O UTPUT : θ t : network parameters 1: t ← 1; θ t ← Initialize the network parameter ; 2: θ t stop ← ∅ ; /

Original: * The parameter of the stopped network */ 3: for i = 1 to epochs do /
Revised: * The parameter of the stopped network */ 3: for i = 1 to epochs do /

Original: * Phase I: Learning from a noisy training data set */ 4: for j = 1 to | ˜ D| / |B t | do 5:
Revised: * Phase I: Learning from a noisy training data set */ 4: for j = 1 to | ˜ D| / |B t | do 5:

Original: Draw a mini-batch B t from ˜ D ; 6: θ t +1 =
Revised: Draw a mini-batch B t from ˜ D ; 6: θ t +1 =

Original: θ t − α ∇ (cid:0) 1 |B t | (cid:80) x ∈B t L ( x, ˜ y ; θ t ) (cid:1) ; /* Update by Eq.
Revised: θ t − α ∇ (cid:0) 1 |B t | (cid:80) x ∈B t L ( x, ˜ y ; θ t ) (cid:1) ; /* Update by Eq.

Original: (1) */ 7: train _ err ← Get_Training_Error( ˜ D , θ t ); /
Revised: (1) */ 7: train _ err ← Get_Training_Error( ˜ D , θ t ); /

Original: * A training error at time t */ 8: if train _ err ≤ τ then /*Save the network when train _ err ≤ τ */ 9: θ t stop ← θ t ; break; 10: t ← t + 1; 11: θ t ← θ t stop ; /* Load the network stopped at t stop */ 12: for i = stop _ epoch to epochs do /* Phase II: Learning from a maximal safe set */ 13: for j = 1 to | ˜ D| / |B t | do 14:
Revised: * A training error at time t */ 8: if train _ err ≤ τ then /*Save the network when train _ err ≤ τ */ 9: θ t stop ← θ t ; break; 10: t ← t + 1; 11: θ t ← θ t stop ; /* Load the network stopped at t stop */ 12: for i = stop _ epoch to epochs do /* Phase II: Learning from a maximal safe set */ 13: for j = 1 to | ˜ D| / |B t | do 14:

Original: Draw a mini-batch B t from ˜ D ; 15:
Revised: Draw a mini-batch B t from ˜ D ; 15:

Original: S t ← { x | argmax y P ( y | x, t ; q ) = ˜ y } ; /
Revised: S t ← { x | argmax y P ( y | x, t ; q ) = ˜ y } ; /

Original: * A maximal safe set in Deﬁnition 3.3 */ 16: θ t +1 =
Revised: * A maximal safe set in Deﬁnition 3.3 */ 16: θ t +1 =

Original: θ t − α ∇ (cid:0) 1 |
Revised: θ t − α ∇ (cid:0) 1 |

Original: [Equation]
Revised: [Equation]

Original: (cid:80)
Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: A. R ESULT WITH S YNTHETIC N OISE (F IGURE 11)
Revised: A. R ESULT WITH S YNTHETIC N OISE (F IGURE 11)

Original: To verify the performance of Prestopping and Prestopping+ with the noise-rate heuristic in Section 3.2.1, we trained a VGG-19 network on two simulated noisy data sets with the same conﬁguration as in Section 5.
Revised: To verify the performance of Prestopping and Prestopping+ with the noise-rate heuristic in Section 3.2.1, we trained a VGG-19 network on two simulated noisy data sets with the same conﬁguration as in Section 5.

Original: Figure 11 shows the test error of our two methods using the noise-rate heuristic as well as those of the other ﬁve training methods.
Revised: Figure 11 shows the test error of our two methods using the noise-rate heuristic as well as those of the other ﬁve training methods.

Original: Again, the test error of either Prestopping or Prestopping+ was the lowest at most error rates with any noise type.
Revised: Again, the test error of either Prestopping or Prestopping+ was the lowest at most error rates with any noise type.

Original: The trend of the noise-rate heuristic here was almost the same as that of the validation heuristic in Section 5.1.
Revised: The trend of the noise-rate heuristic here was almost the same as that of the validation heuristic in Section 5.1.

Original: Especially when the noise rate was 40% , Prestopping and Prestopping+ signiﬁcantly improved the test error by 5 . 1 pp – 17 . 

Revised: Especially when the noise rate was 40% , Prestopping and Prestopping+ signiﬁcantly improved the test error by 5 . 1 pp – 17 .

Original: pp in the pair noise (Figure 11(a)) and 0 .
Revised: pp in the pair noise (Figure 11(a)) and 0 .

Original: 3 pp – 17 .
Revised: 3 pp – 17 .

Original: pp in the symmetric noise (Figure 11(b))
Revised: pp in the symmetric noise (Figure 11(b))

Original: compared with the other robust methods.
Revised: compared with the other robust methods.

Original: [Table]
Revised: [Table]

Original: e s t T e s t
Revised: e s t T e s t

Original: [Table]
Revised: [Table]

Original: rr o r
Revised: rr o r

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: A. C OMPARISON WITH V ALIDATION H EURISTIC (F IGURE 12)
Revised: A. C OMPARISON WITH V ALIDATION H EURISTIC (F IGURE 12)

Original: Figure 12 shows the difference in test error caused by the two heuristics.
Revised: Figure 12 shows the difference in test error caused by the two heuristics.

Original: Overall, the performance with the noise-rate heuristic was worse than that with validation heuristic, even though the worse one outperformed the other training methods as shown in Figure 11.
Revised: Overall, the performance with the noise-rate heuristic was worse than that with validation heuristic, even though the worse one outperformed the other training methods as shown in Figure 11.

Original: As the assumption of the noise-rate heuristic does not hold perfectly, a lower performance of the noise-rate heuristic looks reasonable.
Revised: As the assumption of the noise-rate heuristic does not hold perfectly, a lower performance of the noise-rate heuristic looks reasonable.

Original: However, we expect that the performance with this heuristic can be improved by stopping a little earlier than the estimated point, and we leave this extension as the future work.
Revised: However, we expect that the performance with this heuristic can be improved by stopping a little earlier than the estimated point, and we leave this extension as the future work.

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: B S UPPLEMENTARY E VALUATION B. H YPERPARAMETER S ELECTION (F IGURE 13)
Revised: B S UPPLEMENTARY E VALUATION B. H YPERPARAMETER S ELECTION (F IGURE 13)

Original: Prestopping requires one additional hyperparameter, the history length q in Deﬁnition 3.1.
Revised: Prestopping requires one additional hyperparameter, the history length q in Deﬁnition 3.1.

Original: For hyperparameter tuning, we trained DenseNet (L=40, k=12) on CIFAR-10 and CIFAR-100 with a noise rate of 40% , and the history length q was chosen in a grid ∈ { 1 , 5 , 10 , 15 , 20 } .
Revised: For hyperparameter tuning, we trained DenseNet (L=40, k=12) on CIFAR-10 and CIFAR-100 with a noise rate of 40% , and the history length q was chosen in a grid ∈ { 1 , 5 , 10 , 15 , 20 } .

Original: Figure 13 shows the test error of Prestopping obtained by the grid search on two noisy CIFAR data sets.
Revised: Figure 13 shows the test error of Prestopping obtained by the grid search on two noisy CIFAR data sets.

Original: Regardless of the noise type, the lowest test error was typically achieved when the value of q was 10 in both data sets.
Revised: Regardless of the noise type, the lowest test error was typically achieved when the value of q was 10 in both data sets.

Original: Therefore, the history length q was set to be 10 in all experiments.
Revised: Therefore, the history length q was set to be 10 in all experiments.

Original: [Figure]
Revised: [Figure]

Original: B. E XPERIMENT USING T INY -I MAGE N ET D ATA S ET (F IGURE 14)R
Revised: B. E XPERIMENT USING T INY -I MAGE N ET D ATA S ET (F IGURE 14)

Original:  3: For a larger scale experiment, we repeated the image classiﬁcation task on Tiny-ImageNet (classes), a subset of ImageNet (Krizhevsky et al., 2012), with 100 , 000 training and 10 , 000 validation images.
Revised: R 3: For a larger-scale experiment, we repeated the image classiﬁcation task on Tiny-ImageNet (classes), a subset of ImageNet (Krizhevsky et al., 2012), with 100 , 000 training and 10 , 000 validation images.

Original: Because no test set exists,the randomly selected 9 , 000 images from the validation set were used as the test set, and the rest 1 , 000 validation images were used as the clean validation set for the validation heuristic.
Revised: Because no test set exists, randomly selected 9 , 000 images from the validation set were used as the test set, and the rest 1 , 000 validation images were used as the clean validation set for the validation heuristic.

Original: The experimental conﬁgurations were the same as those in Section 5.
Revised: The experimental conﬁgurations were the same as those in Section 5.

Original: Figure 14 shows the test error of the seven training methods using VGG-19 on Tiny-ImageNet with varying noise rates.
Revised: Figure 14 shows the test error of the seven training methods using VGG-19 on Tiny-ImageNet with varying noise rates.

Original: Similar to CIFAR data sets in Figures 5 and 6, both Prestopping and
Revised: Similar to CIFAR data sets in Figures 5 and 6, both Prestopping and

Original: Prestopping+ outperformed the other robust methods in both noise types.
Revised: Prestopping+ outperformed the other robust methods in both noise types.

Original: The only difference was Default
Revised: The only difference was Default

Original: [Figure]
Revised: [Figure]

Original: a) Pair Noise
Revised: a) Pair Noise

Original: [Figure]
Revised: [Figure]

Original: (b) Symmetric Noise.
Revised: (b) Symmetric Noise.

Original: [Figure]
Revised: [Figure]

Original: that there was no synergistic effect from collaboration with sample refurbishment because of the larger number of classes (i.e., 200 ).
Revised: that there was no synergistic effect from collaboration with sample refurbishment because of the larger number of classes (i.e., 200 ).

Original: In particular, compared with other robust methods, Prestopping showed signiﬁcant reduction in the absolute test error by 7 . 8 pp – 12 .
Revised: In particular, compared with other robust methods, Prestopping showed signiﬁcant reduction in the absolute test error by 7 . 8 pp – 12 .

Original: pp at pair noise of 40% and 1 .
Revised: pp at pair noise of 40% and 1 .

Original: 7 pp – 13 .
Revised: 7 pp – 13 .

Original: 9 pp at symmetric noise of 40% .
Revised: 9 pp at symmetric noise of 40% .

Original: B. E XPERIMENT USING C LOTHING D ATA S ET (F IGURE 15)
Revised: B. E XPERIMENT USING C LOTHING D ATA S ET (F IGURE 15)

Original: R 2: We additionally evaluated the superiority of Prestopping (+) onClothing70k (14 classes), a subset of another challenging real-world noisy data set called Clothing1M (Xiao et al., 2015), where its noise rate was estimated at 38 .
Revised: R 2: We additionally evaluated the superiority of Prestopping (+) on another challenging real-world noisy data set Clothing70k (14 classes), a subset 8 of Clothing1M (Xiao et al., 2015), where its noise rate was estimated at 38 .

Original: For  Clothing70k data set, we randomly selected 70 , 000 images from 1 M noisy training images in Clothing1M and used them as its noisy training set. As for the validation and test set, we did exploit the original clean validationset and test set; 14 , 313 and 10 ,images, respectively.
Revised: For the Clothing70k data set, we randomly selected 70 , 000 images from 1 million noisy training images in Clothing1M and used them as its noisy training set; we did exploit the original clean validation and test sets consisting of 14 , 313 and 10 , 526 images, respectively.

Original: The experimental conﬁgurations were the same as those in Section 5.
Revised: The experimental conﬁgurations were the same as those in Section 5.

Original: Figure 15 shows the test error of the seven training methods using two CNNs on Clothing70k with real-world noise of 38 .
Revised: Figure 15 shows the test error of the seven training methods using two CNNs on Clothing70k with real-world noise of 38 .

Original: Overall, Prestopping and Prestopping+ signiﬁcantly outperformed  other robust methods.
Revised: Overall, Prestopping and Prestopping+ signiﬁcantly outperformed the other robust methods.

Original: In particular, Prestopping+ achieved the lowest test error because the number of classes was not that large (i.e., 14 classes).
Revised: In particular, Prestopping+ achieved the lowest test error because the number of classes was not that large (i.e., 14 classes).

Original: Compared with other state-of-the-art methods, it improved the absolute test error by 1 . 1 pp – 3 . 3 pp using DenseNet and 0 .
Revised: Compared with other state-of-the-art methods, it improved the absolute test error by 1 . 1 pp – 3 . 3 pp using DenseNet and 0 .

Original: 7 pp – 5 . 

Revised: 7 pp – 5 .

Original: pp using VGG.
Revised: pp using VGG-19.

Original: [Table]
Revised: [Table]

Original: B e s t T e s t E rr 
Revised: B e s t T e s t E rr o r

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: B. I MPACT OF N UMBER OF C LASSES ON Prestopping+ (F IGURE 16)
Revised: B. I MPACT OF N UMBER OF C LASSES ON Prestopping+ (F IGURE 16)

Original: We investigate the impact of the number of classes on Prestopping+ under the pair noise of 40% .
Revised: We investigate the impact of the number of classes on Prestopping+ under the pair noise of 40% .

Original: Figure 16 shows the ratio of refurbished samples used for training (hatched bar) with the refurbishing accuracy (solid line) on two data sets with 10 and 100 classes.
Revised: Figure 16 shows the ratio of refurbished samples used for training (hatched bar) with the refurbishing accuracy (solid line) on two data sets with 10 and 100 classes.

Original: When the number of classes is small (e.g., 10), Prestopping+ refurbished most of false-labeled samples very accurately; the refurbishing accuracy was consistently over 91 .
Revised: When the number of classes is small (e.g., 10), Prestopping+ refurbished most of false-labeled samples very accurately; the refurbishing accuracy was consistently over 91 .

Original: 6% during the entire training period.
Revised: 6% during the entire training period.

Original: On the other hand, when the number of classes is large (e.g., 100), the refurbishing accuracy drastically dropped from 92 .
Revised: On the other hand, when the number of classes is large (e.g., 100), the refurbishing accuracy drastically dropped from 92 .

Original: 0% to 76 .
Revised: 0% to 76 .

Original: 7% as more samples were refurbished.
Revised: 7% as more samples were refurbished.

Original: Thus, as shown in Figure 16(b), collaboration with sample refurbishment is not always synergistic because such a signiﬁcant drop in the refurbishing accuracy could induce many falsely corrected samples, especially when there are many classes
Revised: Thus, as shown in Figure 16(b), collaboration with sample refurbishment is not always synergistic because such a signiﬁcant drop in the refurbishing accuracy could induce many falsely corrected samples, especially when there are many classes

Original: [Figure]
Revised: [Figure]

Original: C C OMPLETE R ESULTS ON B EST T EST E RROR
Revised: C C OMPLETE R ESULTS ON B EST T EST E RROR

Original: Table 1 shows the test error of seven training methods using two CNNs on two simulated noisy data sets with varying noise rates.
Revised: Table 1 shows the test error of seven training methods using two CNNs on two simulated noisy data sets with varying noise rates.

Original: Table 2 shows the test error of seven training methods using two CNNs on two real-world noisy data sets with different noise rates.
Revised: Table 2 shows the test error of seven training methods using two CNNs on two real-world noisy data sets with different noise rates

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: The best test errors under pair and symmetric noises of 30% ( τ = 0 . 3 )
Revised: The best test errors under pair and symmetric noises of 20% ( τ = 0 . 2 )

Original: [Table]
Revised: [Table]

Revised: in Figure

Original: [Table] 
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: D C ASE S TUDY : N OISY L ABELS IN O RIGINAL CIFAR- 
Revised: D C ASE S TUDY : N OISY L ABELS IN O RIGINAL CIFAR-

Original: One interesting observation is a noticeable improvement of Prestopping+ even when the noise rate was 0% , as shown in Table 1(a).
Revised: One interesting observation is a noticeable improvement of Prestopping+ even when the noise rate was 0% , as shown in Table 1(a).

Original: It was turned out that Prestopping+ sometimes refurbished the labels of the false-labeled samples which were originally contained in the CIFAR data sets.
Revised: It was turned out that Prestopping+ sometimes refurbished the labels of the false-labeled samples which were originally contained in the CIFAR data sets.

Original: Figureshows a few successful refurbishment cases.
Revised: Figureshows a few successful refurbishment cases.

Original: For example, an image falsely annotated as a “Boy” was refurbished as a “Baby” (Figure 17(a))
Revised: For example, an image falsely annotated as a “Boy” was refurbished as a “Baby” (Figure 17(a))

Original: [Table]
Revised: [Table]

Original: and an image falsely annotated as a “Mouse” was refurbished as a “Hamster” (Figure 17(c)).
Revised: and an image falsely annotated as a “Mouse” was refurbished as a “Hamster” (Figure 17(c)).

Original: Thus, this sophisticated label correction of Prestopping+ helps overcome the residual label noise in well-known benchmark data sets, which are misconceived to be clean
Revised: Thus, this sophisticated label correction of Prestopping+ helps overcome the residual label noise in well-known benchmark data sets, which are misconceived to be clean

Original: [Table]
Revised: [Table]

Original: and ultimately further improves the generalization performance of a network.
Revised: and ultimately further improves the generalization performance of a network.

Original: [Equation] [Figure]
Revised: [Equation] [Figure]