Original: A UTO C LEANSING : U NBIASED E STIMATION OF D EEP L EARNING WITH M ISLABELED D ATA
Revised: A UTO C LEANSING : U NBIASED E STIMATION OF D EEP L EARNING WITH M ISLABELED D ATA

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Mislabeled samples cause prediction errors.
Revised: Mislabeled samples cause prediction errors.

Original: This study proposes a solution to the problem of incorrect labels, known as AutoCleansing , to automatically capture the effect of incorrect labels and mitigate it without removing the mislabeled samples.
Revised: This study proposes a solution to the problem of incorrect labels, known as AutoCleansing , to automatically capture the effect of incorrect labels and mitigate it without removing the mislabeled samples.

Original: AutoCleansing consists of a base network model and sample-category speciﬁc constants.
Revised: AutoCleansing consists of a base network model and sample-category speciﬁc constants.

Original: Both parameters of the base model and sample-category constants are estimated simultaneously using the training data.
Revised: Both parameters of the base model and sample-category constants are estimated simultaneously using the training data.

Original: Thereafter, predictions for test data are made using a base model without the constants capturing the mislabeled effects.
Revised: Thereafter, predictions for test data are made using a base model without the constants capturing the mislabeled effects.

Original: A theoretical model for AutoCleansing is developed and theoretical analysis shows that the  proposed method can estimate true parameters with mislabeled data if the model is correctly constructed.
Revised: A theoretical model for AutoCleansing is developed and theoretical analysis shows that the loss function of the proposed method can be minimized locally at true parameters with mislabeled data if the model is correctly constructed.

Original: Experimental results show that AutoCleansing has better performance in test accuracy than previous studies for CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets
Revised: Experimental results show that AutoCleansing has better performance in test accuracy than previous studies for CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The prediction performance of supervised machine learning depends on the quality of the training data.
Revised: The prediction performance of supervised machine learning depends on the quality of the training data.

Original: For classiﬁcation tasks, the dataset is assumed to have a correct label for each object.
Revised: For classiﬁcation tasks, the dataset is assumed to have a correct label for each object.

Original: However, real-world datasets may contain some mislabeled samples.
Revised: However, real-world datasets may contain some mislabeled samples.

Original: For instance, Pleiss et al.
Revised: For instance, Pleiss et al.

Original: (2020) analyzed incorrect labels in the CIFAR-10 and CIFAR-100 datasets (Krizhevsky & Hinton, 2009).
Revised: (2020) analyzed incorrect labels in the CIFAR-10 and CIFAR-100 datasets (Krizhevsky & Hinton, 2009).

Original: They reported that the mislabeled sample was 3 % of CIFAR-10 and 13 % of the CIFAR-100 datasets.
Revised: They reported that the mislabeled sample was 3 % of CIFAR-10 and 13 % of the CIFAR-100 datasets.

Original: Original label
Revised: Original label

Original: DOG
Revised: DOG

Original: TRUCK
Revised: TRUCK

Original: DEER
Revised: DEER

Original: Alternative label
Revised: Alternative label

Original: CAT
Revised: CAT

Original: PERSON
Revised: PERSON

Original: DEER and PERSON
Revised: DEER and PERSON

Original: [Figure]
Revised: Image

Original: Figure 1 shows typical examples of incorrect labels in the CIFAR-10 dataset.
Revised: Figure 1 shows typical examples of incorrect labels in the CIFAR-10 dataset.

Original: It consists of 60, images in 10 category classes.
Revised: It consists of 60, images in 10 category classes.

Original: Each image was assigned one of 10 classes.
Revised: Each image was assigned one of 10 classes.

Original: In this ﬁgure, the original label of #1 is DOG; however, it appears to be the image of CAT.
Revised: In this ﬁgure, the original label of #1 is DOG; however, it appears to be the image of CAT.

Original: As the category set of CIFARincludes both DOG and CAT, #1 is considered to be an example of an incorrect label within the category set .
Revised: As the category set of CIFARincludes both DOG and CAT, #1 is considered to be an example of an incorrect label within the category set .

Original: The image of #2 has TRUCK as the original label.
Revised: The image of #2 has TRUCK as the original label.

Original: However, it shows the image of
Revised: However, it shows the image of

Original: PERSON, which does not belong to the category set.
Revised: PERSON, which does not belong to the category set.

Original: Thus, this is an example of an incorrect label outside the category set .
Revised: Thus, this is an example of an incorrect label outside the category set .

Original: For the image of #3, there are two objects in this image; however, it has only one label of DEER.
Revised: For the image of #3, there are two objects in this image; however, it has only one label of DEER.

Original: It can be considered as an example of an incorrect label with multiple objects .
Revised: It can be considered as an example of an incorrect label with multiple objects .

Original: Incorrect labels in the training dataset may cause prediction errors.
Revised: Incorrect labels in the training dataset may cause prediction errors.

Original: The most intuitive way to address the problem of incorrect labels is by removing mislabeled samples from the training dataset.
Revised: The most intuitive way to address the problem of incorrect labels is by removing mislabeled samples from the training dataset.

Original: However, to identify mislabeled samples, it is necessary to measure the correctness of labels and deﬁne the threshold determining whether the label is correct or not.
Revised: However, to identify mislabeled samples, it is necessary to measure the correctness of labels and deﬁne the threshold determining whether the label is correct or not.

Original: Deleting excess data may reduce the efﬁciency of estimation by decreasing the sample size.
Revised: Deleting excess data may reduce the efﬁciency of estimation by decreasing the sample size.

Original: Finding an optimal threshold requires several runs of learning by removing mislabeled samples with different thresholds.
Revised: Finding an optimal threshold requires several runs of learning by removing mislabeled samples with different thresholds.

Original: This study proposes an alternative solution to the problem of incorrect labels, called AutoCleansing , to automatically capture the effect of incorrect labels and mitigate it without removing mislabeled samples.
Revised: This study proposes an alternative solution to the problem of incorrect labels, called AutoCleansing , to automatically capture the effect of incorrect labels and mitigate it without removing mislabeled samples.

Original: AutoCleansing consists of a base network model and sample-category speciﬁc constants.
Revised: AutoCleansing consists of a base network model and sample-category speciﬁc constants.

Original: Both parameters of the base model and sample-category constants are estimated simultaneously using the training data.
Revised: Both parameters of the base model and sample-category constants are estimated simultaneously using the training data.

Original: Thereafter, predictions for test data are made using a base model without the constants capturing the mislabeled effects.
Revised: Thereafter, predictions for test data are made using a base model without the constants capturing the mislabeled effects.

Original: Figure 2 shows the concept of AutoCleansing.
Revised: Figure 2 shows the concept of AutoCleansing.

Original: Let x be the input, y be the output, and y
Revised: Let x be the input, y be the output, and y

Original: = m ( x, θ ) be the base network model, where θ denotes the parameter of the base model.
Revised: = m ( x, θ ) be the base network model, where θ denotes the parameter of the base model.

Original: Consider ﬁve observations of A, B, · · · , E .
Revised: Consider ﬁve observations of A, B, · · · , E .

Original: The red line is the true model deﬁned as, y
Revised: The red line is the true model deﬁned as, y

Original: = m ( x, θ ∗ ) where θ ∗ denotes the true parameter.
Revised: = m ( x, θ ∗ ) where θ ∗ denotes the true parameter.

Original: B is a mislabeled sample, as the observed label B differs signiﬁcantly from the true label B ∗ .
Revised: B is a mislabeled sample, as the observed label B differs signiﬁcantly from the true label B ∗ .

Original: The dotted line is the estimated model y
Revised: The dotted line is the estimated model y

Original: = m ( x, ˆ θ ) using incorrect data, where ˆ θ denotes the estimated parameter.
Revised: = m ( x, ˆ θ ) using incorrect data, where ˆ θ denotes the estimated parameter.

Original: As can be observed, overﬁtting occurs owing to the mislabeled sample.
Revised: As can be observed, overﬁtting occurs owing to the mislabeled sample.

Original: In this ﬁgure, ˆ y denotes the prediction for x = 3 using the estimated model; however, the true label is y ∗ .
Revised: In this ﬁgure, ˆ y denotes the prediction for x = 3 using the estimated model; however, the true label is y ∗ .

Original: Thus, the incorrect label causes the prediction error.
Revised: Thus, the incorrect label causes the prediction error.

Original: Consider the cleansing model, y
Revised: Consider the cleansing model, y

Original: = m ( x, θ ) + α , where α denotes the constant parameter for each observation.
Revised: = m ( x, θ ) + α , where α denotes the constant parameter for each observation.

Original: If the constant α B captures the effect of an incorrect label, as shown in this ﬁgure, removing the constant from the cleansing model may mitigate the overﬁtting problem.
Revised: If the constant α B captures the effect of an incorrect label, as shown in this ﬁgure, removing the constant from the cleansing model may mitigate the overﬁtting problem.

Original: As shown in the section of the theoretical analysis the proposed AutoCleansing can address the prediction errors due to the incorrect labels within the category set, outside the category set and with multiple objects.
Revised: As shown in the section of the theoretical analysis, the proposed AutoCleansing can address the prediction errors due to the incorrect labels within the category set, outside the category set and with multiple objects.

Original: AutoCleansing can use any network model as the base model with any augmentation method.
Revised: AutoCleansing can use any network model as the base model with any augmentation method.

Original: For example, the experimental section presents the estimation results of AutoCleansing with base models of ResNet (He et al., 2016), WideResNet (Zagoruyko & Komodaki, 2016), Shake-
Revised: For example, the experimental section presents the estimation results of AutoCleansing with base models of ResNet (He et al., 2016), WideResNet (Zagoruyko & Komodaki, 2016), ShakeShake (Gastaldi, 2017), and PyramidNet + ShakeDrop

Original: Shake (Gastaldi, 2017), and PyramidNet + ShakeDrop (Yamada et al., 2018) using AutoAugment (Cubuk et al., 2018).
Revised: (Yamada et al., 2018) using AutoAugment (Cubuk et al., 2018).

Original: Furthermore, AutoCleansing does not require iterative runs to identify the incorrect labels because the effects of the incorrect labels are automatically captured in a single run by the sample-category speciﬁc constant.
Revised: Furthermore, AutoCleansing does not require iterative runs to identify the incorrect labels because the effects of the incorrect labels are automatically captured in a single run by the sample-category speciﬁc constant.

Original: The contribution of this study can be summarized as follows:
Revised: The contribution of this study can be summarized as follows:

Original: • It provides a theoretical model for AutoCleansing.
Revised: • It provides a theoretical model for AutoCleansing.

Original: The incorrect labels in the training data cause a prediction error.
Revised: The incorrect labels in the training data cause a prediction error.

Original: The proposed method can capture the biased effects of incorrect labels automatically and address the problem of prediction error due to incorrect labels.
Revised: The proposed method can capture the biased effects of incorrect labels automatically and address the problem of prediction error due to incorrect labels.

Original: • The proposed method can be implemented with any network model or augmentation method.
Revised: • The proposed method can be implemented with any network model or augmentation method.

Original: This study considers experiments of AutoCleansing with ResNet, WideResNet, Shake-Shake, and PyramidNet + ShakeDrop using AutoAugment.
Revised: This study considers experiments of AutoCleansing with ResNet, WideResNet, Shake-Shake, and PyramidNet + ShakeDrop using AutoAugment.

Original: • Experimental results show that the proposed AutoCleansing method can improve the validation accuracy for the CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets.
Revised: • Experimental results show that the proposed AutoCleansing method can improve the validation accuracy for the CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets.

Original: • The additional training cost of AutoCleansing relative to the base network model can be negligible.
Revised: • The additional training cost of AutoCleansing relative to the base network model can be negligible.

Original: AutoCleansing can remove the biased effect of incorrect labels automatically in a single run of leraning.
Revised: AutoCleansing can remove the biased effect of incorrect labels automatically in a single run of learning.

Revised: For example, the additional learning time of AutoCleansing with CIFAR-10/100 datasets is only 0.5 % of the learning time of the base network models.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: There are several studies on noisy datasets in the literature on machine learning.
Revised: There are several studies on noisy datasets in the literature on machine learning.

Original: Frnay & Verleysen (2014) provide a comprehensive review of label noise in classiﬁcation.
Revised: Frnay & Verleysen (2014) provide a comprehensive review of label noise in classiﬁcation.

Original: Algan & Ulusoy (2019) provide a complete overview of deep learning with noisy datasets.
Revised: Algan & Ulusoy (2019) provide a complete overview of deep learning with noisy datasets.

Original: There are three approaches to dealing with mislabeled datasets as follows: (1) robust learning with label noise, (2) identiﬁcation of mislabeled data, and (3) utilization of a small dataset without incorrect labels.
Revised: There are three approaches to dealing with mislabeled datasets as follows: (1) robust learning with label noise, (2) identiﬁcation of mislabeled data, and (3) utilization of a small dataset without incorrect labels.

Original: For robust learning with label noise, after the early works of Reed et al. (2015) and Azadi et al.
Revised: For robust learning with label noise, after the early works of Reed et al. (2015) and Azadi et al.

Original: (2015), several algorithms using deep neural networks have been proposed including the S-model (Goldberger & Ben-Reuven, 2016), MentorNet (Jiang et al., 2018), decoupling (Malach & ShalevShwartz, 2017), F-correction (Patrini et al., 2017),
Revised: (2015), several algorithms using deep neural networks have been proposed including the S-model (Goldberger & Ben-Reuven, 2016), MentorNet (Jiang et al., 2018), decoupling (Malach & ShalevShwartz, 2017), F-correction (Patrini et al., 2017),

Original: Open-set
Revised: Open-set

Original: (Wang et al., 2018), Bi-level-model (Jenni & Favaro, 2018), Lq (Zhang & Sabuncu, 2018), co-teaching (Han et al., 2018), random reweighting (Ren et al., 2018), joint optimization (Tanaka et al., 2018), DAC (Thulasidasan et al., 2019), SELF 
Revised: (Wang et al., 2018), Bi-level-model (Jenni & Favaro, 2018), Lq (Zhang & Sabuncu, 2018), co-teaching (Han et al., 2018), random reweighting (Ren et al., 2018), joint optimization (Tanaka et al., 2018), DAC (Thulasidasan et al., 2019), SELF (Nguyen et al., 2020), dynamic bootstrapping (Arazo et al., 2019), and DivideMix (Li et al., 2020). Goldberger & Ben-Reuven (2016) and Patrini et al.

Original: (Nguyen et al., 2020), and DivideMix (Li et al., 2020).
Revised: (2017) estimate the noise transition matrix to correct the loss function. However, it might to be difﬁcult to estimate the transition matrix correctly. Jiang et al. and Ren et al.

Original: These studies used a dataset with artiﬁcial label noise added.
Revised: propose weighted samples to adapt the noise samples. However, estimation of correctly weighted samples could be challenging tasks. Arazo et al. (2019) proposes a beta mixture model to the cross-entropy loss of each sample and model label noise.

Revised: Their approach shows outstanding performance for high level noise.

Revised: For the linear regression, the consistent robust regression is propsed for the corrupted data (Bhatia et al., 2017).

Revised: However, the consistency of robust lerning with nonlinear estimation using the classiﬁcation model is not clear.

Revised: These studies of robust learning with label noise used a dataset with synthetic label noise added.

Original: Label noise is generated by replacing one label with another with a given probability within a category set.
Revised: Label noise is generated by replacing one label with another with a given probability within a category set.

Revised: These studies show good performance for the artiﬁcially generated label noise.

Revised: However, most of these studies do not provide experiments ﬁnding mislabeled samples in the realworld datasets.

Original: Meanwhile, the approach proposed in this study can consider the incorrect labels within and outside the category set as well as the multiple objects.
Revised: Meanwhile, the approach proposed in this study can consider the incorrect labels within and outside the category set as well as the multiple objects in the real-world datasets.

Original: For the identiﬁcation of mislabeled data, some studies found incorrect labels in famous datasets for deep learning.
Revised: For the identiﬁcation of mislabeled data, some studies found incorrect labels in famous datasets for deep learning.

Original: For instance, Ekambaram et al.
Revised: For instance, Ekambaram et al.

Original: (2017) found 92 mislabeled examples in 18 of the image classes, after reviewing more than 15 % of the images in the ImageNet dataset.
Revised: (2017) found 92 mislabeled examples in 18 of the image classes, after reviewing more than 15 % of the images in the ImageNet dataset.

Original: AlRawi & Karatzasu (2018) reported 9 incorrect samples on CIFAR10 and 15 mislabeled samples on CIFAR100.
Revised: AlRawi & Karatzasu (2018) reported 9 incorrect samples on CIFAR10 and 15 mislabeled samples on CIFAR100.

Original: M¨uller & Markert (2019) reported 4 mislabeled samples on MNIST, 7 samples on
Revised: M¨uller & Markert (2019) reported 4 mislabeled samples on MNIST, 7 samples on

Original: CIFAR100, and 64 samples on Fashion-MNIST.
Revised: CIFAR100, and 64 samples on Fashion-MNIST.

Original: Pleiss et al.
Revised: Pleiss et al.

Original: identiﬁed incorrect labels using the area under the margin (AUM) statistic.
Revised: identiﬁed incorrect labels using the area under the margin (AUM) statistic.

Original: They showed that incorrect labels are 3 % on CIFAR10, 13 % on CIFAR100, and 24 % on Tiny ImageNet.
Revised: They showed that incorrect labels are 3 % on CIFAR10, 13 % on CIFAR100, and 24 % on Tiny ImageNet.

Original: On CIFAR100, they reported that removing 13 % of the data leads to a 1.2 % drop in error.
Revised: On CIFAR100, they reported that removing 13 % of the data leads to a 1.2 % drop in error.

Original: Identifying mislabeled data requires some criteria to determine whether it is correct or incorrect.
Revised: Identifying mislabeled data requires some criteria to determine whether it is correct or incorrect.

Original: Several runs of learning may be required to search for optimal criteria for the identiﬁcation of incorrect labels.
Revised: Several runs of learning may be required to search for optimal criteria for the identiﬁcation of incorrect labels.

Original: The method proposed in this study, in contrast, allows us to capture and remove the effects of incorrect labels in a single run of learning.
Revised: The method proposed in this study, in contrast, allows us to capture and remove the effects of incorrect labels in a single run of learning.

Original: For the utilization of a small dataset without incorrect labels, it is assumed that we have a small set of clean data, namely, free of mislabeled samples.
Revised: For the utilization of a small dataset without incorrect labels, it is assumed that we have a small set of clean data, namely, free of mislabeled samples.

Original: Sukhbaatar et al.
Revised: Sukhbaatar et al.

Original: and Hendrycks et al.
Revised: and Hendrycks et al.

Original: (2018) used clean data to estimate the noise-transition matrix for incorrect labels.
Revised: (2018) used clean data to estimate the noise-transition matrix for incorrect labels.

Original: Other studies on the utilization of clean data include Ren et al.
Revised: Other studies on the utilization of clean data include Ren et al.

Original: , Li et al.
Revised: , Li et al.

Original: (2017), and Zhang et al.
Revised: (2017), and Zhang et al.

Original: (2019b).
Revised: (2019b).

Revised: However, it might be difﬁcult to ﬁnd a small set of clean data in the real-world datasets.

Original: Meanwhile, the method proposed in this study does not require clean data.
Revised: Meanwhile, the method proposed in this study does not require clean data.

Original:  A UTO C LEANSING
Revised:  A UTO C LEANSING

Original: Consider the classiﬁcation task with K categories for the N training data points X
Revised: Consider the classiﬁcation task with K categories for the N training data points X

Original: = { x 1 , x 2 , · · · , x N } and labels Y =
Revised: = { x 1 , x 2 , · · · , x N } and labels Y =

Original: { y 1 , y 2 , · · · , y N } .
Revised: { y 1 , y 2 , · · · , y N } .

Original: Let the learning network model be, M ( x, θ ) =
Revised: Let the learning network model be, M ( x, θ ) =

Original: { m 1 ( x, θ ) , · · · , m K ( x, θ ) } that assigns an input x to an output m with a given parameter θ .
Revised: { m 1 ( x, θ ) , · · · , m K ( x, θ ) } that assigns an input x to an output m with a given parameter θ .

Original: The predicted probability of output y , given the input x using this model, is assumed to be calculated by the following softmax function:
Revised: The predicted probability of output y , given the input x using this model, is assumed to be calculated by the following softmax function:

Original: [Equation]
Revised: [Equation]

Original: where m i
Revised: where m i

Original: = m i ( x, θ ) denotes the i th element of output and K = { 1 , · · · , K } the category set.
Revised: = m i ( x, θ ) denotes the i th element of output and K = { 1 , · · · , K } the category set.

Original: If the training data has mislabeled samples, the estimated parameter might be biased, which could cause a prediction error.
Revised: If the training data has mislabeled samples, the estimated parameter might be biased, which could cause a prediction error.

Original: To address the problem of incorrect labels, consider the cleansing network model, C ( x, θ )
Revised: To address the problem of incorrect labels, consider the cleansing network model, C ( x, θ )

Original: =
Revised: =

Original: M ( x, θ ) + α , where α ∈ R N × K is a parameter of the sample-category speciﬁc constant.
Revised: M ( x, θ ) + α , where α ∈ R N × K is a parameter of the sample-category speciﬁc constant.

Original: Let α k be the constant for input x of the sample and category k .
Revised: Let α k be the constant for input x of the sample and category k .

Original: Therefore, the prediction probability P C with the constant can be expressed as follows:
Revised: Therefore, the prediction probability P C with the constant can be expressed as follows:

Original: [Equation]
Revised: [Equation]

Original: If the effect of the mislabeled samples is captured by the sample-category-speciﬁc constant α of the cleansing network model C ( x, θ, α )
Revised: If the effect of the mislabeled samples is captured by the sample-category-speciﬁc constant α of the cleansing network model C ( x, θ, α )

Original: =
Revised: =

Original: M ( x, θ ) + α , the base network model M ( x, θ ) could avoid the biased problem due to incorrect labels.
Revised: M ( x, θ ) + α , the base network model M ( x, θ ) could avoid the biased problem due to incorrect labels.

Original: The learning process with AutoCleansing is as follows: (1) Learning with the training data using a cleansing network model C ( x, θ, α )
Revised: The learning process with AutoCleansing is as follows: (1) Learning with the training data using a cleansing network model C ( x, θ, α )

Original: =
Revised: =

Original: M ( x, θ ) + α , (2) deleting the sample-category-speciﬁc constant ˆ α estimated in the learning process, and (3) testing with the validation data using a cleansed network model, C ( x, ˆ θ, ˆ α ) − ˆ α , where ˆ θ denotes the estimated parameter of the base model.
Revised: M ( x, θ ) + α , (2) deleting the sample-category-speciﬁc constant ˆ α estimated in the learning process, and (3) testing with the validation data using a cleansed network model, C ( x, ˆ θ, ˆ α ) − ˆ α , where ˆ θ denotes the estimated parameter of the base model.

Original: 3. T HEORETICAL A NALYSIS OF A UTO C LEANSING
Revised: 3. T HEORETICAL A NALYSIS OF A UTO C LEANSING

Original: To investigate the performance of the proposed AutoCleansing for mislabeled data, we consider the following deﬁnition of a general case:
Revised: To investigate the performance of the proposed AutoCleansing for mislabeled data, we consider the following deﬁnition of a general case:

Original: Deﬁnition 1 (Incorrect labels and outside of the category set).
Revised: Deﬁnition 1 (Incorrect labels and outside of the category set).

Original: Let K ∗ be a full set of true categories and K ⊂ K ∗ be the category set of the model.
Revised: Let K ∗ be a full set of true categories and K ⊂ K ∗ be the category set of the model.

Original: Let π (ˆ y | y ∗ x ) be the probability such that the input x with the true category y ∗ ∈ K ∗ has label ˆ y ∈ K .
Revised: Let π (ˆ y | y ∗ x ) be the probability such that the input x with the true category y ∗ ∈ K ∗ has label ˆ y ∈ K .

Original: Let P ( y ∗ | x, K ∗ , θ ∗ ) be the true probability of the category y ∗ from the category set K ∗ with input x given true parameter θ ∗ .
Revised: Let P ( y ∗ | x, K ∗ , θ ∗ ) be the true probability of the category y ∗ from the category set K ∗ with input x given true parameter θ ∗ .

Original: Thus, the observed probability of category ˆ y ∈ K for the input x with incorrect labels within and outside of the category set is deﬁned as follows:
Revised: Thus, the observed probability of category ˆ y ∈ K for the input x with incorrect labels within and outside of the category set is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: This deﬁnition includes incorrect labels, outside of the category set, and multiple objects.
Revised: This deﬁnition includes incorrect labels, outside of the category set, and multiple objects.

Original: If K ∗ =
Revised: If K ∗ =

Original: K , this deﬁnition is equivalent to that within the category set.
Revised: K , this deﬁnition is equivalent to that within the category set.

Original: If incorrect labels occur outside the category set, namely π (ˆ y | y ∗ , x ) = 0 ∀
Revised: If incorrect labels occur outside the category set, namely π (ˆ y | y ∗ , x ) = 0 ∀

Original: ˆ y (cid:54) =
Revised: ˆ y (cid:54) =

Original: y ∗ ∈ K and π (ˆ y | y ∗ , x ) ≥ 0 ∀
Revised: y ∗ ∈ K and π (ˆ y | y ∗ , x ) ≥ 0 ∀

Original: y ∗ ∈ K ∗ \ K , it deﬁnes the observed probability outside the category set.
Revised: y ∗ ∈ K ∗ \ K , it deﬁnes the observed probability outside the category set.

Original: Let S be the combination of categories in multiple objects (for example, DEER and PERSON) and K + = K (cid:83) S be the true category set including the original categories and combination of categories in multiple objects.
Revised: Let S be the combination of categories in multiple objects (for example, DEER and PERSON) and K + = K (cid:83) S be the true category set including the original categories and combination of categories in multiple objects.

Original: If K ∗ =
Revised: If K ∗ =

Original: K + , deﬁnitiondescribes multiple objects.
Revised: K + , deﬁnitiondescribes multiple objects.

Original: If the sample data has incorrect labels within the category set, incorrect labels outside of the category set, or multiple objects, the estimated parameter using the minimum loss function does not converge to the true parameter; thus, the incorrect labels cause the prediction error.
Revised: If the sample data has incorrect labels within the category set, incorrect labels outside of the category set, or multiple objects, the estimated parameter using the minimum loss function does not converge to the true parameter; thus, the incorrect labels cause the prediction error.

Original: However, the following theorem shows that AutoCleansing can address the biased estimation due to mislabeled data:
Revised: However, the following theorem shows that AutoCleansing can address the biased estimation due to mislabeled data:

Original: Theorem 1 (AutoCleansing for incorrect labels within and outside the category set).
Revised: Theorem 1 (AutoCleansing for incorrect labels within and outside the category set).

Original: Let K ∗ be a full set of true categories and K be the category set of the model.
Revised: Let K ∗ be a full set of true categories and K be the category set of the model.

Original: Let π (ˆ y | y ∗ , x ) be a probability such that the input x with the true category y ∗ ∈ K ∗ has label ˆ y ∈ K .
Revised: Let π (ˆ y | y ∗ , x ) be a probability such that the input x with the true category y ∗ ∈ K ∗ has label ˆ y ∈ K .

Original: Assume that the sample has incorrect labels and the outside of the category set is deﬁned as Deﬁnition 1.
Revised: Assume that the sample has incorrect labels and the outside of the category set is deﬁned as Deﬁnition 1.

Original: If the model is correctly constructed, the probability distribution of output is the softmax function, and theparameters are identiﬁed; the estimated parameters using the minimum loss function with AutoCleansing converge to the true parameter ˆ θ → θ ∗ , as N → ∞ .
Revised: If the model is correctly constructed, the probability distribution of output is the softmax function; the loss function with AutoCleansing is minimized locally at the true parameter values θ ∗ , as N → ∞ .

Original: The proof can be found in Appendix A.1.
Revised: The proof can be found in Appendix A.1.

Revised: Note that while this theorem states that the loss function with AutoCleansing is minimized locally at true values, it does not guarantee that minimization of the loss function will converge to the true value, if the loss function has more than one local minimum.

Original: Theorem 1 shows that implementation of the samplecategory speciﬁc constant α can capture the biased effect of incorrect labels.
Revised: Theorem 1 shows that implementation of the sample-category speciﬁc constant α can capture the biased effect of incorrect labels.

Original: This suggests that the value of α may reﬂect the effects of incorrect labels.
Revised: This suggests that the value of α may reﬂect the effects of incorrect labels.

Original: The following theorem conﬁrms this:
Revised: The following theorem conﬁrms this:

Original: Theorem 2 (Sample-category speciﬁc constants and incorrect labels).
Revised: Theorem 2 (Sample-category speciﬁc constants and incorrect labels).

Original: Assume that the sample has incorrect labels and outside of the category set deﬁned as Deﬁnition 1.
Revised: Assume that the sample has incorrect labels and outside of the category set deﬁned as Deﬁnition 1.

Original: Consider that the true label t is assigned a false label f .
Revised: Consider that the true label t is assigned a false label f .

Original: Let ˆ α t and ˆ α f be parameters of the sample-category-speciﬁc constants of the true and false labels, respectively, which are estimated by the minimum loss function with
Revised: Let ˆ α t and ˆ α f be parameters of the sample-category-speciﬁc constants of the true and false labels, respectively, which are estimated by the minimum loss function with

Original: AutoCleansing.
Revised: AutoCleansing.

Original: Assume that the probability of observing a false label is greater than or equal to that of the true label, namely Q ( f | x, K ∗ , θ ∗ ) ≥ Q ( t | x, K ∗ , θ ∗ ) .
Revised: Assume that the probability of observing a false label is greater than or equal to that of the true label, namely Q ( f | x, K ∗ , θ ∗ ) ≥ Q ( t | x, K ∗ , θ ∗ ) .

Original: If the model is correctly constructed and the probability distribution of the output is the softmax function,  the estimated parameter of the sample-category speciﬁc constants ˆ α using the minimum loss function with AutoCleansing has the following sign :
Revised: If the model is correctly constructed, the probability distribution of the output is the softmax function, and the loss function is minimized at true parameter values, the estimated parameter of the sample-category speciﬁc constants ˆ α using the minimum loss function with AutoCleansing has the following sign :

Original: General case: The sample-category-speciﬁc constants of the true label are equal to or less than that of the false label: ˆ α t ≤ ˆ α f . 2.
Revised: General case: The sample-category-speciﬁc constants of the true label are equal to or less than that of the false label: ˆ α t ≤ ˆ α f . 2.

Original: Symmetric case: Assume π
Revised: Symmetric case: Assume π

Original: ( f | t, x ) is symmetric and independent of x , namely, π ( f | t, x )
Revised: ( f | t, x ) is symmetric and independent of x , namely, π ( f | t, x )

Original: =
Revised: =

Original: π ( f | t ) =
Revised: π ( f | t ) =

Original: π ( t | f ) .
Revised: π ( t | f ) .

Original: For this case, the sample-category speciﬁc constants of the true label are equal to or less than zero: ˆ α t ≤ 0 .
Revised: For this case, the sample-category speciﬁc constants of the true label are equal to or less than zero: ˆ α t ≤ 0 .

Original: Single symmetric case: Assume π ( f | t, x ) is symmetric, independent of x , and incorrect labels occur between t and f only.
Revised: Single symmetric case: Assume π ( f | t, x ) is symmetric, independent of x , and incorrect labels occur between t and f only.

Original: Consequently, π ( f | t, x ) =
Revised: Consequently, π ( f | t, x ) =

Original: π ( f | t ) =
Revised: π ( f | t ) =

Original: π ( t | f ) and π ( j | j, x ) = 1 , ∀ j (cid:54) =
Revised: π ( t | f ) and π ( j | j, x ) = 1 , ∀ j (cid:54) =

Original: t, f .
Revised: t, f .

Original: For this case, the sample-category-speciﬁc constants of labels except the true and false labels are equal to zero: ˆ α j = 0 , ∀ j (cid:54) =
Revised: For this case, the sample-category-speciﬁc constants of labels except the true and false labels are equal to zero: ˆ α j = 0 , ∀ j (cid:54) =

Original: t, f 
Revised: t, f .

Original: Appendix A.2 provides the proof of this theorem.
Revised: Appendix A.2 provides the proof of this theorem.

Revised: Table 1 provides numerical examples of the theorem.

Original: E XPERIMENTS
Revised: Assume that the category set has three categories K = { 1 , 2 , 3 } . Example (A) is the incorrect label within the category set. Assume the true model output is { m ∗ 1 , m ∗ 2 , m ∗ 3 } = { 0 . 1 , 0 . Namely, the correct category is 3 that has the highest output value.

Revised: If the ﬁrst label is observed, the incorrect label causes the biased learning: { c 1 , c 2 , c 3 } = { 0 . 8 , 0 . 1 , 0 . 1 } .

Revised: The estimated samplecategory-speciﬁc constants are assumed to be optimal

Revised: such that α

Revised: = m ∗ − c .

Revised: The constant of the true label ( α 3 = − 0 .

Revised: 7 ) is less than that of the false label ( α 1 = 0 .

Revised: such that the output value of the biased model ( c = m ∗ + α ) of observed category (0.8) is larger than one of true category (0.1).

Revised: Example (B) is the incorrect label outside the category set.

Revised: Consider the true category set is K ∗

Revised: = { 1 , 2 , 3 , 4 } .

Revised: Assume the true category is 4 that is the outside of the observed category set.

Revised: The optimal constant of observed label ( α 1 = 0 .

Revised: 7 ) has the largest value.

Revised: Note that the constant of true category ( α 4 = − 0 .

Revised: 7 ) is not estimated, because this category is the outside of category set.

Revised: Example (C) is the multiple objects.

Revised: Assume the observed label is 1, while the true categories are 1 and 3.

Revised: The constant of unobserved true category ( α 3 = − 0 .

Revised: 7 ) is less than the one of observed true label ( α 1 = 0 . 0 ),

Original:  This section provides the experiments investigating the performance of the proposed AutoCleansing on the CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009), SVHN (Netzer et al., 2011), and
Revised: [Table]  E XPERIMENTS This section provides the experiments investigating the performance of the proposed AutoCleansing on the CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009), SVHN (Netzer et al., 2011), and

Original: ImageNet (Russakovsky et al., 2015) datasets.
Revised: ImageNet (Russakovsky et al., 2015) datasets.

Original: The proposed AutoCleansing method requires base network models.
Revised: The proposed AutoCleansing method requires base network models.

Original: In these experiments, the base network models are Wide-ResNet 40-2 and WideResNet 28-10 (Zagoruyko & Komodaki, 2016), Shake-Shake 26 2x32d, 26 2x96d and 26 2x112d (Gastaldi, 2017), and PyramidNet + ShakeDrop (Yamada et al., 2018).
Revised: In these experiments, the base network models are Wide-ResNet 40-2 and WideResNet 28-10 (Zagoruyko & Komodaki, 2016), Shake-Shake 26 2x32d, 26 2x96d and 26 2x112d (Gastaldi, 2017), and PyramidNet + ShakeDrop (Yamada et al., 2018).

Original: All hyperparameters of base network models are the same as those used in AutoAugment (Cubuk et al., 2018), FastAutoAugment (Lim et al., 2019), and PBA (Ho et al., 2019) 1 .
Revised: All hyperparameters of base network models are the same as those used in AutoAugment (Cubuk et al., 2018), FastAutoAugment (Lim et al., 2019), and PBA (Ho et al., 2019) 1 .

Original: A cosine learning decay with one annealing cycle was applied to all models except ImageNet.
Revised: A cosine learning decay with one annealing cycle was applied to all models except ResNet.

Original: For AutoCleansing, the  weight decay of the sample-category speciﬁc constant is 5 ×10 − 5 , except for PyramidNet, which uses 1 ×10 − 5 .
Revised: For AutoCleansing, the sample-category speciﬁc constatns could not convergence without regularization. The weight decay of the sample-category speciﬁc constant is 5 ×10 − 5 , except for PyramidNet, which uses 1 ×10 − 5 .

Original: AutoCleansing has learning parameters of the sample-category speciﬁc constant α that consists of
Revised: AutoCleansing has learning parameters of the sample-category speciﬁc constant α that consists of

Original: K variables for each sample.
Revised: K variables for each sample.

Original: Note that all K variables cannot be identiﬁed; therefore, the constants for the ﬁrst category are set to zero for all samples ( α 1 = 0 ).
Revised: Note that all K variables cannot be identiﬁed; therefore, the constants for the ﬁrst category are set to zero for all samples ( α 1 = 0 ).

Original: Thus, the estimated sample-category speciﬁc constant α ∈ R N × K has N ( K − 1) estimable parameters.
Revised: Thus, the estimated sample-category speciﬁc constant α ∈ R N × K has N ( K − 1) estimable parameters.

Original: In this study, the AutoCleansing with the sample-category-speciﬁc constant of N ( K − 1) parameters is called as AC1.
Revised: In this study, the AutoCleansing with the sample-category-speciﬁc constant of N ( K − 1) parameters is called as AC1.

Original: However, it might be difﬁcult to estimate all parameters of α for large datasets.
Revised: However, it might be difﬁcult to estimate all parameters of α for large datasets.

Original: For example, ImageNet has more than 1.2 million images with 1,000 categories for training data.
Revised: For example, ImageNet has more than 1.2 million images with 1,000 categories for training data.

Original: Therefore, AC1 needs to estimate more than 1.2 billion parameters.
Revised: Therefore, AC1 needs to estimate more than 1.2 billion parameters.

Original: Instead, consider the sample speciﬁc constant α ∈ R N such that all categories except the observed label are set to zero for all samples ( α j = 0 ∀ j (cid:54)
Revised: Instead, consider the sample speciﬁc constant α ∈ R N such that all categories except the observed label are set to zero for all samples ( α j = 0 ∀ j (cid:54)

Original: =
Revised: =

Original: ˆ y )
Revised: ˆ y )

Original: ; that is, α has N estimable parameters.
Revised: ; that is, α has N estimable parameters.

Original: The AutoCleansing with the sample-speciﬁc constant of N parameters is called as AC2.
Revised: The AutoCleansing with the sample-speciﬁc constant of N parameters is called as AC2.

Revised: Note that AC2 corresponds to the special case of the single symmetric case in

Revised: Theorem 2.

Revised: For the single symmetric case, α nj = 0 ∀ n, ∀ j (cid:54) =

Revised: t, f .

Revised: If the true category belongs to the outside of the category set, we can not estimate the α nt of the true category, therefore, all categories except the observed label have zero values of α nj .

Original: The experiments compare the results with baseline preprocessing, Cutout (DeVries & Taylor, 2017),
Revised: The experiments compare the results with baseline preprocessing, Cutout (DeVries & Taylor, 2017),

Original: AutoAugment (AA), FastAutoAugment (FAA), and Population Based Augmentation (PBA).
Revised: AutoAugment (AA), FastAutoAugment (FAA), and Population Based Augmentation (PBA).

Original: The baseline preprocessing is conventional augmentation as follows: standardizing the data, horizontal ﬂipping with 50 % probability, zero-padding, and randomly cropping.
Revised: The baseline preprocessing is conventional augmentation as follows: standardizing the data, horizontal ﬂipping with 50 % probability, zero-padding, and randomly cropping.

Original: In the proposed AutoCleansing, we follow the procedure of AutoAugment, which ﬁrst applies the baseline preprocessing, then applies the AutoAugment policy, and ﬁnally applies the Cutout.
Revised: In the proposed AutoCleansing, we follow the procedure of AutoAugment, which ﬁrst applies the baseline preprocessing, then applies the AutoAugment policy, and ﬁnally applies the Cutout.

Original: 4. E XPERIMENTAL R ESULTS
Revised: 4. E XPERIMENTAL R ESULTS

Original: The CIFAR-10 dataset has a total of 60,000 images, including 50,000 for training set and 10,000 for test sets.
Revised: The CIFAR-10 dataset has a total of 60,000 images, including 50,000 for training set and 10,000 for test sets.

Original: The number of categories was 10.
Revised: The number of categories was 10.

Original: Thus, the sample-category speciﬁc constant for AC1 has 0.45 million estimable parameters, whereas the sample speciﬁc constant for AC2 has 0.05 million estimable parameters.
Revised: Thus, the sample-category speciﬁc constant for AC1 has 0.45 million estimable parameters, whereas the sample speciﬁc constant for AC2 has 0.05 million estimable parameters.

Original: Table 1 shows the results of the test accuracy for different network models using the CIFAR-10 dataset.
Revised: Table 2 shows the results of the test accuracy for different network models using the CIFAR-10 dataset.

Original: For all models, the proposed AutoCleansing with AutoAugment can achieve better performance compared to previous models 2 .
Revised: For all models, the proposed AutoCleansing with AutoAugment can achieve better performance compared to previous models 2 .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: The CIFAR-100 dataset also has a total of 60,000 images, including 50,000 for training set and 10,000 for test sets.
Revised: The CIFAR-100 dataset also has a total of 60,000 images, including 50,000 for training set and 10,000 for test sets.

Original: The number of categories was 100.
Revised: The number of categories was 100.

Original: Thus, the sample-category speciﬁc constant for AC1 has 4.95 million estimable parameters, whereas the sample speciﬁc constant for AC2 has 0.05 million estimable parameters.
Revised: Thus, the sample-category speciﬁc constant for AC1 has 4.95 million estimable parameters, whereas the sample speciﬁc constant for AC2 has 0.05 million estimable parameters.

Original: Table 2 provides the results for the CIFAR-100 dataset.
Revised: Table 3 provides the results for the CIFAR-100 dataset.

Original: Similarly, for CIFAR-10, the proposed model has better accuracy than previous models.
Revised: Similarly, for CIFAR-10, the proposed model has better accuracy than previous models.

Original: The SVHN dataset has 73,257 digit images for the core training set, 531,131 for the additional training set [Table] and 26,032 for the test set.
Revised: The SVHN dataset has 73,257 digit images for the core training set, 531,131 for the additional training set, and 26,032 for the test set.

Original: In this experiment, both core and additional training sets were used.
Revised: In this experiment, both core and additional training sets were used.

Original: The number of categories was 10.
Revised: The number of categories was 10.

Original: Thus, the sample-category speciﬁc constant for
Revised: Thus, the sample-category speciﬁc constant for

Original: AC1 has 5.44 million estimable parameters, whereas the sample speciﬁc constant for AC2 has 0.
Revised: AC1 has 5.44 million estimable parameters, whereas the sample speciﬁc constant for AC2 has 0.

Original: million estimable parameters.
Revised: million estimable parameters.

Original: Table 3 reports the results for the SVHN dataset.
Revised: Table 4 reports the results for the SVHN dataset.

Original: For the SVHN dataset, the proposed model has better accuracy than previous models.
Revised: For the SVHN dataset, the proposed model has better accuracy than previous models.

Original: [Table]

Original: The ImageNet dataset has more than 1.2 million images for the training set and 0.15 million images for the validation and test sets.
Revised: The ImageNet dataset has more than 1.2 million images for the training set and 0.15 million images for the validation and test sets.

Original: The number of categories was 1,000.
Revised: The number of categories was 1,000.

Original: Thus, the sample-category speciﬁc constant for AC1 has more than 1.28 billion parameters that may not be feasible to estimate.
Revised: Thus, the sample-category speciﬁc constant for AC1 has more than 1.28 billion parameters that may not be feasible to estimate.

Original: Therefore, this experiment uses the sample speciﬁc constant for AC2, which has 1.28 million estimable parameters.
Revised: Therefore, this experiment uses the sample speciﬁc constant for AC2, which has 1.28 million estimable parameters.

Original: Table 4 shows the results for the ImageNet dataset.
Revised: Table 5 shows the results for the ImageNet dataset.

Original: The proposed model of
Revised: The proposed model of

Original: AC2 has better Top1 accuracy than previous models, whereas the Top5 accuracy of AC2 is less than that of AA and FAA. 
Revised: AC2 has better Top1 accuracy than previous models, whereas the Top5 accuracy of AC2 is less than that of AA and FAA. Pleiss et al.

Original: AutoCleansing can be implemented with different augments.
Revised: (2020) proposed the the area under the margin (AUM) statistic for the robust learning with label noise. They provide the experiments of the label noise in the real-world datasets as well as the artiﬁcially generated label noise. Their experimental results show that the AUM has better performance than the previous studies for dataset with synthetic label noise added.

Original: Table 5 shows the test accuracy of
Revised: Table 6 shows [Table] [Table]

Original:  AutoCleansing  using the Wide-ResNet 40-2 model and different augmentation methods.
Revised: the test accuracy of AutoCleansing and AUM using the ResNet 32 model.

Original: AutoCleansing has better performance for all augmentation of baseline, Cutout, and AutoAugment on both CIFAR-10 and CIFAR-100.[Table]
Revised: All hyperparameters of base network models are the same as those used in Pleiss et al. AutoCleansing demonstrates outperforming the AUM on both CIFAR-10 and CIFAR-100.

Original: 4. D ETECTION OF I NCORRECT L ABELS USING A UTO C LEANSING
Revised: 4. D ETECTION OF I NCORRECT L ABELS USING A UTO C LEANSING

Original: As shown in the theoretical analysis section [Table] AutoCleansing can capture the effect of incorrect labels using the sample-category speciﬁc constant α .
Revised: As shown in the theoretical analysis section, AutoCleansing can capture the effect of incorrect labels using the sample-category speciﬁc constant α .

Original: If there is no mislabeled sample, α → 0 , as N → ∞ .
Revised: If there is no mislabeled sample, α → 0 , as N → ∞ .

Original: A large value of | α | indicates the existence of mislabeled samples in the data.
Revised: A large value of | α | indicates the existence of mislabeled samples in the data.

Original: Therefore, it might be possible to identify incorrect labels using the estimated sample-category speciﬁc constant ˆ α .
Revised: Therefore, it might be possible to identify incorrect labels using the estimated sample-category speciﬁc constant ˆ α .

Original: Figure 4-6 in Appendix show the sample-category speciﬁc constants ˆ α for the example of mislabeled images in CIFAR-10 and CIFAR-100.
Revised: Figure 4-6 in Appendix show the sample-category speciﬁc constants ˆ α for the example of mislabeled images in CIFAR-10 and CIFAR-100.

Original: The estimation model is Wide-ResNet 40-2 with
Revised: The estimation model is Wide-ResNet 40-2 with

Original: AutoCleansing.
Revised: AutoCleansing.

Original: Because ˆ α 1 is ﬁxed to zero, these ﬁgures show the standardized values of ˆ α k − Mean { ˆ α 1 , · · · , ˆ α K } .
Revised: Because ˆ α 1 is ﬁxed to zero, these ﬁgures show the standardized values of ˆ α k − Mean { ˆ α 1 , · · · , ˆ α K } .

Original: Let ˆ α Maxn be the maximum value and ˆ α Minn be the minimum value of the standardized sample-category-speciﬁc constants for n th data.
Revised: Let ˆ α Maxn be the maximum value and ˆ α Minn be the minimum value of the standardized sample-category-speciﬁc constants for n th data.

Original: MaxRank is the percentile rank of sorted ˆ α Maxn in descending order, and MinRank is sorted ˆ α Minn in ascending order.
Revised: MaxRank is the percentile rank of sorted ˆ α Maxn in descending order, and MinRank is sorted ˆ α Minn in ascending order.

Original: #1 in Figure 4 is an example of incorrect labels within the category set.
Revised: #1 in Figure 4 is an example of incorrect labels within the category set.

Original: The original label of #1 is
Revised: The original label of #1 is

Original: DOG and an alternative to this image is CAT.
Revised: DOG and an alternative to this image is CAT.

Original: The estimated ˆ α of an original label of #1 is 0.233, whereas ˆ α of an alternative label of #1 is -0.227.
Revised: The estimated ˆ α of an original label of #1 is 0.233, whereas ˆ α of an alternative label of #1 is -0.227.

Original: Similarly, the original labels are positive ˆ α , whereas alternative labels are negative ˆ α for images #2-#5 in this ﬁgure.
Revised: Similarly, the original labels are positive ˆ α , whereas alternative labels are negative ˆ α for images #2-#5 in this ﬁgure.

Original: #6 in Figure 5 shows an example of incorrect labels outside the category set.
Revised: #6 in Figure 5 shows an example of incorrect labels outside the category set.

Original: The original label of # is TRUCK, whereas the correct label might be PERSON, which does not belong to the category set for the CIFAR-10 dataset.
Revised: The original label of # is TRUCK, whereas the correct label might be PERSON, which does not belong to the category set for the CIFAR-10 dataset.

Original: For this image, ˆ α of the original labels have positive.
Revised: For this image, ˆ α of the original labels have positive.

Original: For these examples of #6-#10, the estimated ˆ α of the original labels are positive.
Revised: For these examples of #6-#10, the estimated ˆ α of the original labels are positive.

Original: #11 in Figure 6 presents an example of multiple objects.
Revised: #11 in Figure 6 presents an example of multiple objects.

Original: The original label of #11 is DEER; however, this image includes an additional object of PERSON that does not belong to the category set of the
Revised: The original label of #11 is DEER; however, this image includes an additional object of PERSON that does not belong to the category set of the

Original: CIFAR-10 dataset.
Revised: CIFAR-10 dataset.

Original: Images of #12-#15 are examples of multiple objects in the CIFAR-100 datasets.
Revised: Images of #12-#15 are examples of multiple objects in the CIFAR-100 datasets.

Original: For example, the original label of #12 is PLANE; however, this image also has the object of SEA.
Revised: For example, the original label of #12 is PLANE; however, this image also has the object of SEA.

Original: Both PLANE and SEA belong to the category set.
Revised: Both PLANE and SEA belong to the category set.

Original: For these images, ˆ α of the original labels have positive, ˆ α and additional labels have negative labels.
Revised: For these images, ˆ α of the original labels have positive, ˆ α and additional labels have negative labels.

Original: Note that the values of MaxRank or MinRank are less than 0.1 % for all images in these ﬁgures.
Revised: Note that the values of MaxRank or MinRank are less than 0.1 % for all images in these ﬁgures.

Original: This suggests that the mislabeled samples can be identiﬁed using the high or low value of the samplecategory-speciﬁc constants ˆ α estimated by AutoCleansing.
Revised: This suggests that the mislabeled samples can be identiﬁed using the high or low value of the samplecategory-speciﬁc constants ˆ α estimated by AutoCleansing.

Original: To identify the mislabeled samples, we must specify the threshold criteria between the correct and incorrect labels.
Revised: To identify the mislabeled samples, we must specify the threshold criteria between the correct and incorrect labels.

Original: Let τ be the percentage of mislabeled samples in the dataset.
Revised: Let τ be the percentage of mislabeled samples in the dataset.

Original: Algorithm 2 in Appendix A.4 provides the procedure for searching for mislabeled samples given τ using AutoCleansing.
Revised: Algorithm 2 in Appendix A.4 provides the procedure for searching for mislabeled samples given τ using AutoCleansing.

Original: After searching for the mislabeled samples, we can remove the mislabeled data from the sample and run the learning models using the trimmed sample.
Revised: After searching for the mislabeled samples, we can remove the mislabeled data from the sample and run the learning models using the trimmed sample.

Original: Figure 3 shows the test accuracies of the base model using trimmed data with different threshold criteria of incorrect labels on the CIFARand CIFAR-100 datasets.
Revised: Figure 3 shows the test accuracies of the base model using trimmed data with different threshold criteria of incorrect labels on the CIFARand CIFAR-100 datasets.

Original: The base model is a Wide-ResNet 40-2 model with AutoAugmentation.
Revised: The base model is a Wide-ResNet 40-2 model with AutoAugmentation.

Original: As can be observed, the test accuracies are highest when 0.2 % of incorrect labels are dropped in both datasets.
Revised: As can be observed, the test accuracies are highest when 0.2 % of incorrect labels are dropped in both datasets.

Original: This ﬁgure shows that dropping mislabeled samples at an appropriate drop rate can improve the classiﬁcation accuracy.
Revised: This ﬁgure shows that dropping mislabeled samples at an appropriate drop rate can improve the classiﬁcation accuracy.

Original: However, it is necessary to repeat learning several times with different criteria to determine the optimum drop rate of mislabeled samples.
Revised: However, it is necessary to repeat learning several times with different criteria to determine the optimum drop rate of mislabeled samples.

Original: If an excessively small sample is dropped, the effects of bias due to incorrect labels might remain.
Revised: If an excessively small sample is dropped, the effects of bias due to incorrect labels might remain.

Original: If excess samples are removed, the estimation efﬁciency could be reduced because of the decreasing sample size.
Revised: If excess samples are removed, the estimation efﬁciency could be reduced because of the decreasing sample size.

Original: Notably, the maximum test accuracies using trimmed data are very close to the accuracies of AutoCleansing in this ﬁgure.
Revised: Notably, the maximum test accuracies using trimmed data are very close to the accuracies of AutoCleansing in this ﬁgure.

Original: This suggests that AutoCleansing can remove the biased effects of incorrect labels without dropping the mislabeled samples from the datasets.
Revised: This suggests that AutoCleansing can remove the biased effects of incorrect labels without dropping the mislabeled samples from the datasets.

Original: Furthermore, AutoCleansing does not need to repeat learning because it does not require the threshold criteria of the drop rate for the mislabeled samples.[Figure] [Table] [Table]
Revised: Furthermore, AutoCleansing does not need to repeat learning because it does not require the threshold criteria of the drop rate for the mislabeled samples.

Original: [Table] [Table] [Table] [Table]
Revised: Instead of dropping the mislabeled samples that requires the threshold criteria of incorrect labels, AutoCleansing automatically drops the sample-category speciﬁc constants capturing the mislabeled bias.

Original: [Table]
Revised: [Figure]

Original: t es t acc u r acypercent of mislabeled samples
Revised: t es t acc u r acy

Original: CIFARAutoCleansing Trimmed Data
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This study introduces AutoCleansing to address the biased problem due to incorrect labels.
Revised: This study introduces AutoCleansing to address the biased problem due to incorrect labels.

Original: The proposed method is appealing in that it can automatically capture the effect of incorrect labels and mitigate it without removing mislabeled samples.
Revised: The proposed method is appealing in that it can automatically capture the effect of incorrect labels and mitigate it without removing mislabeled samples.

Original: As shown in the theoretical analysis, if the model is correctly constructed,  AutoCleansing can estimate true parameters using mislabeled samples for incorrect labels within or outside the category set as well as multiple objects.
Revised: As shown in the theoretical analysis, if the model is correctly constructed, the loss function of AutoCleansing can be minimized locally at true parameter values using mislabeled samples for incorrect labels within or outside the category set as well as multiple objects.

Original: Furthermore, AutoCleansing can be implemented with any network model and any augmentation method.
Revised: Furthermore, AutoCleansing can be implemented with any network model and any augmentation method.

Original: Experimental results show that the proposed AutoCleansing has better performance than previous studies on CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets.
Revised: Experimental results show that the proposed AutoCleansing has better performance than previous studies on CIFAR-10, CIFAR-100, SVHN, and ImageNet datasets.

Original: Additional topics for future investigation into AutoCleansing include applications to artiﬁcial label noise (Algan & Ulusoy, 2019), the use of other network models such as EfﬁcientNet (Tan & Le, 2019), and the use of recent augmentation methods such as adversarial AutoAugment (Zhang et al., 2019a).
Revised: Additional topics for future investigation into AutoCleansing include applications to artiﬁcial label noise (Algan & Ulusoy, 2019), the use of other network models such as EfﬁcientNet (Tan & Le, 2019), and the use of recent augmentation methods such as adversarial AutoAugment (Zhang et al., 2019a).

Original: A A PPENDIX A. P ROOF OF T HEOREM
Revised: A A PPENDIX A. P ROOF OF T HEOREM

Original: Consider following loss function:
Revised: Consider following loss function:

Original: [Equation]
Revised: [Equation]

Original: where 1[ y n = i ] is an indicator function equal to one when y n =
Revised: where 1[ y n = i ] is an indicator function equal to one when y n =

Original: i and zero otherwise.
Revised: i and zero otherwise.

Original: Let L ( θ ) be the probability limit of the loss function L ( θ ) .
Revised: Let L ( θ ) be the probability limit of the loss function L ( θ ) .

Original: Consider that the sample has incorrect labels and the outside of the category set is deﬁned as deﬁnition 1.
Revised: Consider that the sample has incorrect labels and the outside of the category set is deﬁned as deﬁnition 1.

Original: If the model is correctly constructed [Equation] by the strong law of large numbers, as N → ∞ , L ( θ ) is as follows:
Revised: If the model is correctly constructed, by the strong law of large numbers, as N → ∞ , L ( θ ) is as follows:

Original: [Equation]
Revised: [Equation]

Original: The derivative of L ( θ ) is as follows:
Revised: The derivative of L ( θ ) is as follows:

Original: [Equation]
Revised: [Equation]

Original: If the distribution function is the softmax function (1)
Revised: If the distribution function is the softmax function (1)

Original: [Equation]
Revised: [Equation]

Original: the derivative of L ( θ ) can be expressed as follows:
Revised: the derivative of L ( θ ) can be expressed as follows:

Original: [Equation]
Revised: [Equation]

Original:  where m ∗ k
Revised:  where m ∗ k

Original: = m k
Revised: = m k

Original: ( x, θ ∗ ) denotes the k th element of the output for the base model given the true parameter and the true category set.
Revised: ( x, θ ∗ ) denotes the k th element of the output for the base model given the true parameter and the true category set.

Original: First, consider the case of no incorrect label, namely, π ( i | i, x ) =
Revised: First, consider the case of no incorrect label, namely, π ( i | i, x ) =

Original: for all i ∈ K and K
Revised: for all i ∈ K and K

Original: = K ∗ .
Revised: = K ∗ .

Original: For this case, the derivative of L ( θ ) is equal to zero at the true parameter θ = θ ∗ , as follows:
Revised: For this case, the derivative of L ( θ ) is equal to zero at the true parameter θ = θ ∗ , as follows:

Original: [Equation]
Revised: [Equation]

Original: Note that (cid:80) i ∈ K P ( i | x, K , θ ∗ )
Revised: Note that (cid:80) i ∈ K P ( i | x, K , θ ∗ )

Original: = 1 .
Revised: = 1 .

Original: From the assumption of identiﬁcation for the parameter, L ( θ ) (cid:54) =
Revised: From the assumption of identiﬁcation for the parameter, L ( θ ) (cid:54) =

Original: L ( θ ∗ ) for all θ (cid:54)
Revised: L ( θ ∗ ) for all θ (cid:54)

Original: = θ ∗ .
Revised: = θ ∗ .

Original: Thus, if there is no incorrect label and the model is constructed correctly, the estimated parameter ˆ θ using the minimum loss function is consistent as follows:
Revised: Thus, if there is no incorrect label and the model is constructed correctly, the estimated parameter ˆ θ using the minimum loss function is consistent as follows:

Original: ˆ θ → θ ∗ as
Revised: ˆ θ → θ ∗ as

Original: N
Revised: N

Original: → ∞ .
Revised: → ∞ .

Original: This is a well-known property of consistency in the maximum likelihood estimator of the logit model (Amemiya, 1985).
Revised: This is a well-known property of consistency in the maximum likelihood estimator of the logit model (Amemiya, 1985).

Original: However, in general, the derivative of L ( θ ) is not always equal to zero at the true parameter value if the sample has an incorrect label.
Revised: However, in general, the derivative of L ( θ ) is not always equal to zero at the true parameter value if the sample has an incorrect label.

Original: If there are mislabeled samples, the estimated parameter ˆ θ using the minimum loss function may not converge to the true value.
Revised: If there are mislabeled samples, the estimated parameter ˆ θ using the minimum loss function may not converge to the true value.

Original: Consider the estimation using the AutoCleansing model, c j ( x, Θ)
Revised: Consider the estimation using the AutoCleansing model, c j ( x, Θ)

Original: =
Revised: =

Original: m j ( x, θ )
Revised: m j ( x, θ )

Original: + α j where Θ = { θ, α } .
Revised: + α j where Θ = { θ, α } .

Original: The probability limit of the loss function for the AutoCleansing model is as follows:
Revised: The probability limit of the loss function for the AutoCleansing model is as follows:

Original: [Equation]
Revised: [Equation]

Original: Assume that the probability distribution is a softmax function (2).
Revised: Assume that the probability distribution is a softmax function (2).

Original: Therefore, the derivative of
Revised: Therefore, the derivative of

Original: L C (Θ) can be expressed as follows:
Revised: L C (Θ) can be expressed as follows:

Revised: [Equation]

Revised: [Equation]

Revised:   (cid:80)

Revised: [Equation]

Revised: e

Original: [Equation]
Revised: [Equation]

Original:  where α ∗ i =
Revised: j (cid:80) j ∈ K [Equation] e m ∗ j e m ∗ i + α ∗ i e m i + α j ∂P C ( i | x, K , Θ) ∂θ   where α ∗ i =

Original: ln
Revised: ln

Original: [Equation]
Revised: [Equation]

Original: − m ∗ i .
Revised: − m ∗ i .

Original: Thus, the derivative of L C (Θ) is equal to zero at Θ =Θ ∗
Revised: Thus, the derivative of L C (Θ) is equal to zero at Θ =Θ ∗

Original: =
Revised: =

Original: { θ ∗ , α ∗ } . From the assumption of identiﬁcation for the parameter, L C (Θ) (cid:54) = L C (Θ ∗ ) for all Θ (cid:54) = Θ ∗ .
Revised: { θ ∗ , α ∗ } , as N → ∞ .

Original: Thus, the estimated parameter ˆΘ using the minimum loss function with the AutoCleansing model is consistent as follows: ˆΘ → Θ ∗ , as N → ∞ .

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: From the assumption, Q ( f | x, K ∗ , θ ∗ ) ≥ Q ( t | x, K ∗ , θ ∗ ) .
Revised: From the assumption, Q ( f | x, K ∗ , θ ∗ ) ≥ Q ( t | x, K ∗ , θ ∗ ) .

Original: If the model is correctly constructed, the output of the model for the true category is higher than that of the other category.
Revised: If the model is correctly constructed, the output of the model for the true category is higher than that of the other category.

Original: Therefore, m ∗ t ≥ m ∗ j , ∀ j .
Revised: Therefore, m ∗ t ≥ m ∗ j , ∀ j .

Original: For the general case, from Theorem 1, the difference between the sample-categoryspeciﬁc constants of the false label and that of the true label is as follows:
Revised: For the general case, from Theorem 1, the difference between the sample-categoryspeciﬁc constants of the false label and that of the true label is as follows:

Original: [Equation]
Revised: [Equation]

Original: For the symmetric case, π ( t | k, x ) =
Revised: For the symmetric case, π ( t | k, x ) =

Original: π ( k | t, x ) for all k
Revised: π ( k | t, x ) for all k

Original: Therefore, the sample-category-speciﬁc constants of the true label are as follows:
Revised: Therefore, the sample-category-speciﬁc constants of the true label are as follows:

Original: [Equation]
Revised: [Equation]

Original: For the single-symmetric case, π ( j | j ) = 1 ∀ j (cid:54) = f, t .
Revised: For the single-symmetric case, π ( j | j ) = 1 ∀ j (cid:54) = f, t .

Original: Therefore, the sample-category-speciﬁc constants of category j (cid:54) =
Revised: Therefore, the sample-category-speciﬁc constants of category j (cid:54) =

Original: f, t are as follows:
Revised: f, t are as follows:

Original: [Equation]
Revised: [Equation]

Original: A. A DDITIONAL T ABLES
Revised: A. A DDITIONAL T ABLES

Original: [Table]

Original: [Table]

Original: [Table]

Original: ±

Original: [Table]

Original: [Table]

Original: ±

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: ±

Original: [Table]

Original: [Table]

Original: ±

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: ±

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: ±

Original: [Table]

Original: [Table]

Original: ±

Original: [Table]
Revised: [Table]

Original: A. A LGORITHMS
Revised: A. A LGORITHMS

Original: Require:
Revised: Require:

Original: Initial parameter α ∈ R N × K of the sample-category speciﬁc constant.
Revised: Initial parameter α ∈ R N × K of the sample-category speciﬁc constant.

Original: N : The number of samples.
Revised: N : The number of samples.

Original: K :
Revised: K :

Original: The number of categories.
Revised: The number of categories.

Original: B : The number of minibatches.
Revised: B : The number of minibatches.

Original: Learning: for b = 1 , . . .
Revised: Learning: for b = 1 , . . .

Original: B do
Revised: B do

Original: Sample a minibatch of x ( b ) from training data set X .
Revised: Sample a minibatch of x ( b ) from training data set X .

Original: Let d ( b ) be a vector of index of the sampling such that x ( b ) =
Revised: Let d ( b ) be a vector of index of the sampling such that x ( b ) =

Original: X
Revised: X

Original: [ d ( b ) ] .
Revised: [ d ( b ) ] .

Original: Compute gradient using the cleansing model for the minibatch C ( x ( b ) , θ, α ) =
Revised: Compute gradient using the cleansing model for the minibatch C ( x ( b ) , θ, α ) =

Original: M ( x ( b ) , θ )
Revised: M ( x ( b ) , θ )

Original: + α ( b ) , where α ( b ) =
Revised: + α ( b ) , where α ( b ) =

Original: α [ d ( b ) ] ∈ R N b × K is the constant for the minibatch and N b is the batch size.
Revised: α [ d ( b ) ] ∈ R N b × K is the constant for the minibatch and N b is the batch size.

Original: Update the parameter Θ = { θ, α } .
Revised: Update the parameter Θ = { θ, α } .

Original: end for
Revised: end for

Original: Testing: Predict the category using the cleansed model M ( x, ˆ θ ) =
Revised: Testing: Predict the category using the cleansed model M ( x, ˆ θ ) =

Original: C ( x, ˆ θ, ˆ α ) − ˆ α .
Revised: C ( x, ˆ θ, ˆ α ) − ˆ α .

Original: Require: ˆ α ∈ R N × K : the sample-category speciﬁc constants of n th sample and k th category,
Revised: Require: ˆ α ∈ R N × K : the sample-category speciﬁc constants of n th sample and k th category,

Original: N : the number of samples, K : the number of categories, and τ : the percent of mislabeled samples.
Revised: N : the number of samples, K : the number of categories, and τ : the percent of mislabeled samples.

Original: Require:
Revised: Require:

Original: Standardize ˆ α nk ← ˆ α nk − Mean { ˆ α n 1 , · · · , ˆ α nK } for all n = { 1 , · · · , N } .
Revised: Standardize ˆ α nk ← ˆ α nk − Mean { ˆ α n 1 , · · · , ˆ α nK } for all n = { 1 , · · · , N } .

Original: Require:
Revised: Require:

Original: Set ˆ
Revised: Set ˆ

Original: α Maxn = Max
Revised: α Maxn = Max

Original: { ˆ α n 1 , · · · , ˆ α nK } and ˆ
Revised: { ˆ α n 1 , · · · , ˆ α nK } and ˆ

Original: α Minn =
Revised: α Minn =

Original: Min { ˆ α n 1 , · · · , ˆ α nK } for all n = { 1 , · · · , N } .
Revised: Min { ˆ α n 1 , · · · , ˆ α nK } for all n = { 1 , · · · , N } .

Original: Require: Set L miss =
Revised: Require: Set L miss =

Original: [] : mislabeled list.
Revised: [] : mislabeled list.

Original: N miss =
Revised: N miss =

Original: τN/ 100 : the number of mislabeled samples.
Revised: τN/ 100 : the number of mislabeled samples.

Original: while the size of L miss <N miss do
Revised: while the size of L miss <N miss do

Original: Find the sample index n such as ˆ α Maxn = Max(ˆ α Max ) .
Revised: Find the sample index n such as ˆ α Maxn = Max(ˆ α Max ) .

Original: Delete ˆ α Maxn from ˆ α Max .
Revised: Delete ˆ α Maxn from ˆ α Max .

Original: Add the sample index n to the mislabeled list L miss if n / ∈ L miss .
Revised: Add the sample index n to the mislabeled list L miss if n / ∈ L miss .

Original: if the size of L miss <N miss do
Revised: if the size of L miss <N miss do

Original: Find the sample index n such as ˆ α Minn = Min(ˆ α Min ) .
Revised: Find the sample index n such as ˆ α Minn = Min(ˆ α Min ) .

Original: Delete ˆ α Minn from ˆ α Min .
Revised: Delete ˆ α Minn from ˆ α Min .

Original: Add the sample index n to the mislabeled list L miss if n / ∈ L miss end if end while
Revised: Add the sample index n to the mislabeled list L miss if n / ∈ L miss end if end while

Original: Return: the mislabeled list L miss .
Revised: Return: the mislabeled list L miss .

Original: [Figure]
Revised: [Figure]

Original: A DDITIONAL F IGURES
Revised: A DDITIONAL F IGURES

Original: Dataset
Revised: Dataset

Original: Origianl labels
Revised: Origianl labels

Original: Alternative labels
Revised: Alternative labels

Original: ID
Revised: ID

Original: CIFARdog cat 0. CIFARcat dog 0.
Revised: CIFARdog cat 0. CIFARcat dog 0.

Original: CIFARleopard tiger 0.
Revised: CIFARleopard tiger 0.

Original: CIFARtulip rose 0.
Revised: CIFARtulip rose 0.

Original: CIFARcloud can 0.
Revised: CIFARcloud can 0.

Original: Image
Revised: Image

Original: MaxRank
Revised: MaxRank

Original: MinRank 1.328% 0.038% # # # 0.072% 0.268% 0.020% 0.002% 0.020% # # 2.002% 0.028% 0.438% (cid:127) (cid:129) (cid:127)
Revised: MinRank 1.328% 0.038% # # # 0.072% 0.268% 0.020% 0.002% 0.020% # # 2.002% 0.028% 0.438% (cid:127) (cid:129) (cid:127)

Original: Dataset
Revised: Dataset

Original: Origianl labels
Revised: Origianl labels

Original: Alternative labels
Revised: Alternative labels

Original: ID
Revised: ID

Original: CIFARtruck person 0.
Revised: CIFARtruck person 0.

Original: CIFARlion giraffe 0.
Revised: CIFARlion giraffe 0.

Original: CIFARplate cutlery 0.
Revised: CIFARplate cutlery 0.

Original: CIFARcup glass 0.
Revised: CIFARcup glass 0.

Original: CIFARmotorcycle raincoat 0.
Revised: CIFARmotorcycle raincoat 0.

Original: Image
Revised: Image

Original: MaxRank
Revised: MaxRank

Original: MinRank # 0.002% 7.970% # 0.016% 43.730% # 0.054% 13.646% # 0.044% 39.838% # 0.002% 1.242% (cid:127) (cid:127) !
Revised: MinRank # 0.002% 7.970% # 0.016% 43.730% # 0.054% 13.646% # 0.044% 39.838% # 0.002% 1.242% (cid:127) (cid:127) !

Original: (cid:127)
Revised: (cid:127)

Original: Dataset
Revised: Dataset

Original: Origianl labels
Revised: Origianl labels

Original: Alternative labels
Revised: Alternative labels

Original: ID
Revised: ID

Original: CIFARdeer person 0.
Revised: CIFARdeer person 0.

Original: CIFARplain sea 0.
Revised: CIFARplain sea 0.

Original: CIFARcat dog 0.
Revised: CIFARcat dog 0.

Original: CIFARbus road 0.
Revised: CIFARbus road 0.

Original: CIFARhouse road 0.
Revised: CIFARhouse road 0.

Original: MaxRank
Revised: MaxRank

Original: MinRank 0.450% 0.052% # 5.558% 0.088% # 0.112% 0.006% # 1.356% 0.060% # # 0.024% 0.734%
Revised: MinRank 0.450% 0.052% # 5.558% 0.088% # 0.112% 0.006% # 1.356% 0.060% # # 0.024% 0.734%

Original: Image (cid:127) (cid:127) ! (cid:127)
Revised: Image (cid:127) (cid:127) ! (cid:127)