Original: R OBUST T RAINING WITH E NSEMBLE C ONSENSUS
Revised: R OBUST T RAINING WITH E NSEMBLE C ONSENSUS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Since deep neural networks are over-parametrized, they may memorize noisy examples.
Revised: Since deep neural networks are over-parametrized, they may memorize noisy examples.

Original: We address such memorizing issue in the presence of annotation noise.
Revised: We address such memorizing issue in the presence of annotation noise.

Original: From the fact that deep neural networks cannot generalize neighborhoods of the features acquired via memorization, we observe that noisy examples do not consistently incur small losses on the network under a certain perturbation.
Revised: From the fact that deep neural networks cannot generalize neighborhoods of the features acquired via memorization, we observe that noisy examples do not consistently incur small losses on the network under a certain perturbation.

Original: Based on this observation, we propose a novel training method called Learning with Ensemble Consensus (LEC) that prevents overﬁtting noisy examples by eliminating them using the consensus of an ensemble of perturbed networks.
Revised: Based on this observation, we propose a novel training method called Learning with Ensemble Consensus (LEC) that prevents overﬁtting noisy examples by eliminating them using the consensus of an ensemble of perturbed networks.

Original: One of the proposed LECs, LTEC outperforms the current state-of-the-art methods on MNIST,
Revised: One of the proposed LECs, LTEC outperforms the current state-of-the-art methods on MNIST,

Original: CIFAR-10, and CIFAR-100 with efﬁcient memory usage.
Revised: CIFAR-10, and CIFAR-100 with efﬁcient memory usage.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks (DNNs) have shown excellent performance (Krizhevsky et al., 2012;
Revised: Deep neural networks (DNNs) have shown excellent performance (Krizhevsky et al., 2012;

Original: He et al., 2016) on visual recognition datasets (Deng et al., 2009).
Revised: He et al., 2016) on visual recognition datasets (Deng et al., 2009).

Original: However, it is difﬁcult to obtain annotated datasets of such high quality in practice (Wang et al., 2018a).
Revised: However, it is difﬁcult to obtain annotated datasets of such high quality in practice (Wang et al., 2018a).

Original: Even worse, DNNs may not generalize training data in the presence of noisy examples (Zhang et al., 2016).
Revised: Even worse, DNNs may not generalize training data in the presence of noisy examples (Zhang et al., 2016).

Original: Therefore, there is an increasing demand for robust training methods.
Revised: Therefore, there is an increasing demand for robust training methods.

Original: In general, DNNs trained on noisy datasets ﬁrst generalize clean examples (Arpit et al., 2017).
Revised: In general, DNNs trained on noisy datasets ﬁrst generalize clean examples (Arpit et al., 2017).

Original: Based on this, recent studies consider examples that incur small training losses in the early stage of training as being clean (Han et al., 2018; Shen & Sanghavi, 2019; Yu et al., 2019).
Revised: Based on this, recent studies consider examples that incur small training losses in the early stage of training as being clean (Han et al., 2018; Shen & Sanghavi, 2019; Yu et al., 2019).

Original: However, small-loss examples may be highly corrupted especially when the training set contains a high level of annotation noise.
Revised: However, small-loss examples may be highly corrupted especially when the training set contains a high level of annotation noise.

Original: Hence, choosing safe examples from the noisy training set with small-loss criteria can be impractical.
Revised: Hence, choosing safe examples from the noisy training set with small-loss criteria can be impractical.

Original: To address this, we ﬁnd the method of screening out noisy examples among small-loss examples by focusing on well-known observations: (i) noisy examples are learned via memorization rather than via generalization and (ii) under a certain perturbation, network predictions for memorized features easily ﬂuctuate, while those for generalized features do not.
Revised: To address this, we ﬁnd the method of screening out noisy examples among small-loss examples by focusing on well-known observations: (i) noisy examples are learned via memorization rather than via generalization and (ii) under a certain perturbation, network predictions for memorized features easily ﬂuctuate, while those for generalized features do not.

Original: Based on these two observations, we hypothesize that out of small-loss examples, training losses of noisy examples would change dynamically by injecting certain perturbation, while those of clean examples would not.
Revised: Based on these two observations, we hypothesize that out of small-loss examples, training losses of noisy examples would change dynamically by injecting certain perturbation, while those of clean examples would not.

Original: This suggests that clean examples can be found out of small-loss examples by selecting examples consistently incurring small losses on an ensemble of perturbed networks, i.e. , ensemble consensus on small-loss examples.
Revised: This suggests that clean examples can be found out of small-loss examples by selecting examples consistently incurring small losses on an ensemble of perturbed networks, i.e. , ensemble consensus on small-loss examples.

Original: Since this idea comes from the difference between generalization and memorization, it can be applied to any architecture optimized with the gradient-based method.
Revised: Since this idea comes from the difference between generalization and memorization, it can be applied to any architecture optimized with the gradient-based method.

Original: In this work, we introduce simple perturbation methods to generate the ensemble for discriminating clean and noisy examples.
Revised: In this work, we introduce simple perturbation methods to generate the ensemble for discriminating clean and noisy examples.

Original: By embedding those perturbation methods into training, we propose a new robust training scheme termed learning with ensemble consensus (LEC).
Revised: By embedding those perturbation methods into training, we propose a new robust training scheme termed learning with ensemble consensus (LEC).

Original: In LEC, the network is ﬁrst trained on the entire training set for a while and then trained on examples selected based on whether training losses are consistently small across the ensemble of perturbed networks.
Revised: In LEC, the network is ﬁrst trained on the entire training set for a while and then trained on examples selected based on whether training losses are consistently small across the ensemble of perturbed networks.

Original: We present three LECs with different perturbations and evaluate their effectiveness on three benchmark datasets with random label noise (Goldberger & Ben-Reuven, 2016;
Revised: We present three LECs with different perturbations and evaluate their effectiveness on three benchmark datasets with random label noise (Goldberger & Ben-Reuven, 2016;

Original: Ma et al., 2018), open-set noise (Wang et al., 2018b), and semantic noise.
Revised: Ma et al., 2018), open-set noise (Wang et al., 2018b), and semantic noise.

Original: The proposed LEC outperforms existing robust training methods by efﬁciently removing noisy examples from training batches.
Revised: The proposed LEC outperforms existing robust training methods by efﬁciently removing noisy examples from training batches.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Generalization of DNNs.
Revised: Generalization of DNNs.

Original: Although DNNs are over-parametrized, they have impressive generalization ability (Krizhevsky et al., 2012;
Revised: Although DNNs are over-parametrized, they have impressive generalization ability (Krizhevsky et al., 2012;

Original: He et al., 2016).
Revised: He et al., 2016).

Original: Some studies argue that gradient-based optimization plays an important role in regularizing DNNs (Neyshabur et al., 2014; Zhang et al., 2016).
Revised: Some studies argue that gradient-based optimization plays an important role in regularizing DNNs (Neyshabur et al., 2014; Zhang et al., 2016).

Original: As a result, DNNs optimized with SGD ﬁrst generalize and then memorize the training set (Krueger et al., 2017; Arpit et al., 2017).
Revised: As a result, DNNs optimized with SGD ﬁrst generalize and then memorize the training set (Krueger et al., 2017; Arpit et al., 2017).

Original: Since mislabeling reduces the correlation with other training examples, it is likely that noisy examples are learned via memorization.
Revised: Since mislabeling reduces the correlation with other training examples, it is likely that noisy examples are learned via memorization.

Original: Therefore, we analyze the difference between generalized and memorized features to discriminate clean and noisy examples.
Revised: Therefore, we analyze the difference between generalized and memorized features to discriminate clean and noisy examples.

Original: Training DNNs with Noisy datasets.
Revised: Training DNNs with Noisy datasets.

Original: Label noise issues can be addressed by reducing the impact of noisy examples.
Revised: Label noise issues can be addressed by reducing the impact of noisy examples.

Original: One direction is to train with a modiﬁed loss function based on the noise distribution.
Revised: One direction is to train with a modiﬁed loss function based on the noise distribution.

Original: Most studies of this direction estimate the noise distribution prior to training as it is not accessible in general (Sukhbaatar et al., 2014; Goldberger & Ben-Reuven, 2016;
Revised: Most studies of this direction estimate the noise distribution prior to training as it is not accessible in general (Sukhbaatar et al., 2014; Goldberger & Ben-Reuven, 2016;

Original: Patrini et al., 2017;
Revised: Patrini et al., 2017;

Original: Hendrycks et al., 2018).
Revised: Hendrycks et al., 2018).

Original: Another direction is to train with modiﬁed labels using the current model prediction (Reed et al., 2014; Ma et al., 2018).
Revised: Another direction is to train with modiﬁed labels using the current model prediction (Reed et al., 2014; Ma et al., 2018).

Original: Aside from these directions, recent work suggests a method of exploiting small-loss examples (Jiang et al., 2017; Han et al., 2018; Yu et al., 2019; Shen & Sanghavi, 2019).
Revised: Aside from these directions, recent work suggests a method of exploiting small-loss examples (Jiang et al., 2017; Han et al., 2018; Yu et al., 2019; Shen & Sanghavi, 2019).

Original: Their effectiveness comes from an artifact of SGD that a considerable portion of small-loss examples during generalization is well-annotated (Han et al., 2018).
Revised: Their effectiveness comes from an artifact of SGD that a considerable portion of small-loss examples during generalization is well-annotated (Han et al., 2018).

Original: However, it is still hard to ﬁnd clean examples based on training losses.
Revised: However, it is still hard to ﬁnd clean examples based on training losses.

Original: This study presents a simple method to overcome such a problem of small-loss criteria.
Revised: This study presents a simple method to overcome such a problem of small-loss criteria.

Original:  R OBUST TRAINING WITH E NSEMBLE C ONSENSUS 3. P ROBLEM STATEMENT
Revised:  R OBUST TRAINING WITH E NSEMBLE C ONSENSUS 3. P ROBLEM STATEMENT

Original: In a situation where (cid:15) % of examples in a dataset D := D clean ∪ D noisy are noisy, let S (cid:15), D ,θ denote the set of (100- (cid:15) )
Revised: In a situation where (cid:15) % of examples in a dataset D := D clean ∪ D noisy are noisy, let S (cid:15), D ,θ denote the set of (100- (cid:15) )

Original: % small-loss examples of the network parametrized by θ out of examples in D .
Revised: % small-loss examples of the network parametrized by θ out of examples in D .

Original: Since it is generally hard to learn only all clean examples especially on the highly corrupted training set, it may be problematic to regard all examples in S (cid:15), D ,θ as being clean.
Revised: Since it is generally hard to learn only all clean examples especially on the highly corrupted training set, it may be problematic to regard all examples in S (cid:15), D ,θ as being clean.

Original: To mitigate this, we suggest a simple idea: to ﬁnd noisy examples among examples in S (cid:15), D ,θ .
Revised: To mitigate this, we suggest a simple idea: to ﬁnd noisy examples among examples in S (cid:15), D ,θ .

Original: 3. L EARNING WITH E NSEMBLE C ONSENSUS (LEC)
Revised: 3. L EARNING WITH E NSEMBLE C ONSENSUS (LEC)

Original: Since noisy examples are little correlated with other training examples, they are likely to be learned via memorization.
Revised: Since noisy examples are little correlated with other training examples, they are likely to be learned via memorization.

Original: However, DNNs cannot generalize neighborhoods of the memorized features.
Revised: However, DNNs cannot generalize neighborhoods of the memorized features.

Original: This means that even if training losses of noisy examples are small, they can be easily increased under a certain perturbation δ , i.e. , for ( x, y ) ∈
Revised: This means that even if training losses of noisy examples are small, they can be easily increased under a certain perturbation δ , i.e. , for ( x, y ) ∈

Original: D noisy ,
Revised: D noisy ,

Original: [Equation]
Revised: [Equation]

Original: Unlike noisy examples, some clean examples can be generalized even in the early stage of training with the entire set D .
Revised: Unlike noisy examples, some clean examples can be generalized even in the early stage of training with the entire set D .

Original: Thus, training losses of those clean examples are consistently small in the presence of the certain perturbation δ , i.e. , for ( x, y ) ∈
Revised: Thus, training losses of those clean examples are consistently small in the presence of the certain perturbation δ , i.e. , for ( x, y ) ∈

Original: D clean 
Revised: D clean ,

Original: [Equation]
Revised: [Equation]

Original: This suggests that noisy examples can be identiﬁed by using the inconsistency of losses under certain perturbation δ .
Revised: This suggests that noisy examples can be identiﬁed by using the inconsistency of losses under certain perturbation δ .

Original: Based on this, we regard examples in the intersection of (100- (cid:15) )
Revised: Based on this, we regard examples in the intersection of (100- (cid:15) )

Original: % small-loss examples of an ensemble of M networks generated by perturbing parameters M times, i.e. , ∩ Mm =1 S (cid:15), D ,θ + δ m as being clean.
Revised: % small-loss examples of an ensemble of M networks generated by perturbing parameters M times, i.e. , [Equation] as being clean.

Original: We call it ensemble consensus ﬁltering because examples are selected via ensemble consensus.
Revised: We call it ensemble consensus ﬁltering because examples are selected via ensemble consensus.

Original: Here, we can obtain such θ by training on the entire set for a while.
Revised: Here, we can obtain such θ by training on the entire set for a while.

Original: Considering all this, we propose a new robust training method termed learning with ensemble consensus (LEC) as described in Algorithms 1 and 2.
Revised: Considering all this, we propose a new robust training method termed learning with ensemble consensus (LEC) as described in Algorithms 1 and 2.

Original: Both algorithms consist of warming-up and ﬁltering processes.
Revised: Both algorithms consist of warming-up and ﬁltering processes.

Original: The difference between these two lies in the selection of small-loss examples during the ﬁltering process, i.e. , a mini batch vs. a full batch.
Revised: The difference between these two lies in the selection of small-loss examples during the ﬁltering process, i.e. , a mini batch vs. a full batch.

Original: In Algorithm 1, the network is trained on the intersection of (100- (cid:15) )
Revised: In Algorithm 1, the network is trained on the intersection of (100- (cid:15) )

Original: % small-loss examples of M networks within a mini batch, thus the number of examples updated at once is changing.
Revised: % small-loss examples of M networks within a mini batch, thus the number of examples updated at once is changing.

Original: We can encourage more stable training with ﬁxed batchsize: At each epoch, we obtain the intersection of small-loss examples of M networks within a full batch.
Revised: We can encourage more stable training with ﬁxed batchsize as described in Algorithm 2. At each epoch, we obtain the intersection of small-loss examples of M networks within a full batch.

Original: At each update, we sample a subset of batchsize from the intersection and train them like a normal SGD.
Revised: At each update, we sample a subset of batchsize from the intersection and train them like a normal SGD.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 3. P ERTURBATION TO IDENTIFY NOISY EXAMPLES
Revised: 3. P ERTURBATION TO IDENTIFY NOISY EXAMPLES

Original: Now the goal is to ﬁnd a certain perturbation δ that will be injected to distinguish between generalized and memorized features.
Revised: Now the goal is to ﬁnd a certain perturbation δ that will be injected to distinguish between generalized and memorized features.

Original: We present three LECs with different perturbations in the following.
Revised: We present three LECs with different perturbations in the following.

Original: The pseudocodes can be found in Section A.1.3.
Revised: The pseudocodes can be found in Section A.1.3.

Original: • Network-Ensemble Consensus (LNEC) :
Revised: • Network-Ensemble Consensus (LNEC) :

Original: Inspired by the observation that an ensemble of networks with the same architecture is correlated during generalization and is decorrelated during memorization (Morcos et al., 2018), the perturbation of prediction δ comes from the difference between M networks .
Revised: Inspired by the observation that an ensemble of networks with the same architecture is correlated during generalization and is decorrelated during memorization (Morcos et al., 2018), the perturbation of prediction δ comes from the difference between M networks .

Original: During the warming-up process, M networks are trained independently.
Revised: During the warming-up process, M networks are trained independently.

Original: During ﬁltering process, M networks are trained on the intersection of (100- (cid:15) )
Revised: During ﬁltering process, M networks are trained on the intersection of (100- (cid:15) )

Original: % small-loss examples of M networks.
Revised: % small-loss examples of M networks.

Original: • Self-Ensemble Consensus (LSEC) :
Revised: • Self-Ensemble Consensus (LSEC) :

Original: We focus on the relationship between Morcos et al.
Revised: We focus on the relationship between Morcos et al.

Original: (2018) and Lakshminarayanan et al.
Revised: (2018) and Lakshminarayanan et al.

Original: (2017): network predictions for memorized features are uncertain and those for generalized features are certain.
Revised: (2017): network predictions for memorized features are uncertain and those for generalized features are certain.

Original: Since the uncertainty of predictions also can be captured by multiple stochastic predictions (Gal & Ghahramani, 2016), the perturbation of prediction δ comes from the difference between M stochastic predictions of a single network .
Revised: Since the uncertainty of predictions also can be captured by multiple stochastic predictions (Gal & Ghahramani, 2016), the perturbation of prediction δ comes from the difference between M stochastic predictions of a single network .

Original: During the ﬁltering process, the network is trained on the intersection of (100- (cid:15) )% small-loss examples obtained with M stochastic predictions.
Revised: During the ﬁltering process, the network is trained on the intersection of (100- (cid:15) )% small-loss examples obtained with M stochastic predictions.

Original: • Temporal-Ensemble Consensus (LTEC) :
Revised: • Temporal-Ensemble Consensus (LTEC) :

Original: Inspired by the observation that during training, atypical features are more easily forgetful compared to typical features (Toneva et al., 2018), the perturbation of prediction δ comes from the difference between predictions at the current and the preceding epochs .
Revised: Inspired by the observation that during training, atypical features are more easily forgetful compared to typical features (Toneva et al., 2018), the perturbation of prediction δ comes from the difference between predictions at the current and the preceding epochs .

Original: During the ﬁltering process, the network is trained on the intersection of (100- (cid:15) )% small-loss examples at the current epoch t and preceding min( M − 1 , t − 1) epochs.
Revised: During the ﬁltering process, the network is trained on the intersection of (100- (cid:15) )% small-loss examples at the current epoch t and preceding min( M − 1 , t − 1) epochs.

Original: We collect (100- (cid:15) )
Revised: We collect (100- (cid:15) )

Original: % small-loss examples at the preceding epochs, rather than network parameters to reduce memory usage.
Revised: % small-loss examples at the preceding epochs, rather than network parameters to reduce memory usage.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we will show (i) the effectiveness of three perturbations at removing noisy examples from small-loss examples and (ii) the comparison of LEC and other existing methods under various annotation noises.
Revised: In this section, we will show (i) the effectiveness of three perturbations at removing noisy examples from small-loss examples and (ii) the comparison of LEC and other existing methods under various annotation noises.

Original: 4. E XPERIMENTAL SETUP
Revised: 4. E XPERIMENTAL SETUP

Original: Annotation noise.
Revised: Annotation noise.

Original: We study random label noise (Goldberger & Ben-Reuven, 2016; Ma et al., 2018), open-set noise (Wang et al., 2018b), and semantic noise.
Revised: We study random label noise (Goldberger & Ben-Reuven, 2016; Ma et al., 2018), open-set noise (Wang et al., 2018b), and semantic noise.

Original: To generate these noises, we use
Revised: To generate these noises, we use

Original: MNIST (LeCun et al., 1998), CIFAR-10/100 (Krizhevsky et al., 2009) that are commonly used to assess robustness.
Revised: MNIST (LeCun et al., 1998), CIFAR-10/100 (Krizhevsky et al., 2009) that are commonly used to assess robustness.

Original: For each benchmark dataset, we only corrupt its training set, while leaving its test set intact for testing.
Revised: For each benchmark dataset, we only corrupt its training set, while leaving its test set intact for testing.

Original: The details can be found in Section A.1.1.
Revised: The details can be found in Section A.1.1.

Original: • Random label noise.
Revised: • Random label noise.

Original: Annotation issues can occur in easy images as well as hard images (Wang et al., 2018a).
Revised: Annotation issues can occur in easy images as well as hard images (Wang et al., 2018a).

Original: This is simulated in two ways: sym- (cid:15) % and asym- (cid:15) % .
Revised: This is simulated in two ways: sym- (cid:15) % and asym- (cid:15) % .

Original: For sym- (cid:15) %, (cid:15) % of the entire set are randomly mislabeled to one of the other labels and for asym- (cid:15) %, each label i of (cid:15) % of the entire set is changed to i + 1 .
Revised: For sym- (cid:15) %, (cid:15) % of the entire set are randomly mislabeled to one of the other labels and for asym- (cid:15) %, each label i of (cid:15) % of the entire set is changed to i + 1 .

Original: We study four types: sym-20% and asym-20% for simulating a low level of noise, and sym-60% and asym-40% for simulating a high level of noise.
Revised: We study four types: sym-20% and asym-20% for simulating a low level of noise, and sym-60% and asym-40% for simulating a high level of noise.

Original: • Open-set noise.
Revised: • Open-set noise.

Original: In reality, annotated datasets may contain out-of-distribution (OOD) examples.
Revised: In reality, annotated datasets may contain out-of-distribution (OOD) examples.

Original: As in Yu et al.
Revised: As in Yu et al.

Original: , to make OOD training examples, images of (cid:15) % examples randomly sampled from the original dataset are replaced with images from another dataset, while labels are left intact.
Revised: , to make OOD training examples, images of (cid:15) % examples randomly sampled from the original dataset are replaced with images from another dataset, while labels are left intact.

Original: SVHN (Netzer et al., 2011) is used to generate open-set noise for CIFAR-100, and ImageNet-32 (Chrabaszcz et al., 2017) and CIFAR-100 are used to generate open-set noise for CIFAR-10.
Revised: SVHN (Netzer et al., 2011) is used to generate open-set noise for CIFAR-100, and ImageNet-32 (Chrabaszcz et al., 2017) and CIFAR-100 are used to generate open-set noise for CIFAR-10.

Original: Here, the similarity between original and open-set datasets is considered.
Revised: Here, the similarity between original and open-set datasets is considered.

Original: We study two types: 20% and 40% open-set noise.
Revised: We study two types: 20% and 40% open-set noise.

Original: • Semantic noise.
Revised: • Semantic noise.

Original: In general, images with easy patterns are correctly labeled, while images with ambiguous patterns are obscurely mislabeled.
Revised: In general, images with easy patterns are correctly labeled, while images with ambiguous patterns are obscurely mislabeled.

Original: To simulate this, we select the top (cid:15) % most uncertain images and then ﬂip their labels to the confusing ones.
Revised: To simulate this, we select the top (cid:15) % most uncertain images and then ﬂip their labels to the confusing ones.

Original: Here, the uncertainty of each image is computed by the amount of disagreement between the predictions of ﬁve networks trained with clean dataset as in Lakshminarayanan et al.
Revised: Here, the uncertainty of each image is computed by the amount of disagreement between the predictions of ﬁve networks trained with clean dataset as in Lakshminarayanan et al.

Original: Then, the label of each image is assigned to the label with the highest value of averaged softmax outputs of the ﬁve networks trained with a clean dataset except for its groundtruth label.
Revised: Then, the label of each image is assigned to the label with the highest value of averaged softmax outputs of the ﬁve networks trained with a clean dataset except for its groundtruth label.

Original: We study two types: 20% and 40% semantic noise.
Revised: We study two types: 20% and 40% semantic noise.

Original: Architecture and optimization.
Revised: Architecture and optimization.

Original: Unless otherwise speciﬁed, we use a variant of 9-convolutional layer architecture (Laine & Aila, 2016; Han et al., 2018).
Revised: Unless otherwise speciﬁed, we use a variant of 9-convolutional layer architecture (Laine & Aila, 2016; Han et al., 2018).

Original: All parameters are trained for 200 epochs with Adam (Kingma & Ba, 2014) with a batch size of 128.
Revised: All parameters are trained for 200 epochs with Adam (Kingma & Ba, 2014) with a batch size of 128.

Original: The details can be found in Section A.1.2.
Revised: The details can be found in Section A.1.2.

Original: Hyperparameter.
Revised: Hyperparameter.

Original: The proposed LEC involves three hyperparameters: duration of warming-up
Revised: The proposed LEC involves three hyperparameters: duration of warming-up

Original: T w , noise ratio (cid:15) %, and the number of predictions M .
Revised: T w , noise ratio (cid:15) %, and the number of predictions M .

Original: Unless otherwise speciﬁed, T w is set to 10, and M is set to 5 for random label noise and open-set noise, and 10 for semantic noise.
Revised: Unless otherwise speciﬁed, T w is set to 10, and M is set to 5 for random label noise and open-set noise, and 10 for semantic noise.

Original: We assume that a noise ratio of (cid:15) % is given.
Revised: We assume that a noise ratio of (cid:15) % is given.

Original: Further study on hyperparameters can be found in Section 5.2.
Revised: Further study on hyperparameters can be found in Section 5.2.

Original: Evaluation.
Revised: Evaluation.

Original: We use two metrics: test accuracy and label precision (Han et al., 2018).
Revised: We use two metrics: test accuracy and label precision (Han et al., 2018).

Original: At the end of each epoch, test accuracy is measured as the ratio of correctly predicted test examples to all test examples, and label precision is measured as the ratio of clean examples used for training to examples used for training.
Revised: At the end of each epoch, test accuracy is measured as the ratio of correctly predicted test examples to all test examples, and label precision is measured as the ratio of clean examples used for training to examples used for training.

Original: Thus, for both metrics, higher is better.
Revised: Thus, for both metrics, higher is better.

Original: For methods with multiple networks, the averaged values are reported.
Revised: For methods with multiple networks, the averaged values are reported.

Original: We report peak as well as ﬁnal accuracy because a small validation set may be available in reality.
Revised: We report peak as well as ﬁnal accuracy because a small validation set may be available in reality.

Original: For each noise type, every method is run four times with four random seeds, e.g., four runs of
Revised: For each noise type, every method is run four times with four random seeds, e.g., four runs of

Original: Standard on CIFAR-10 with sym-20%.
Revised: Standard on CIFAR-10 with sym-20%.

Original: For each run of both random label noise and open-set noise, both a noisy dataset and network initialization are randomized.
Revised: For each run of both random label noise and open-set noise, both a noisy dataset and network initialization are randomized.

Original: Here, four noisy datasets generated in four runs are the same for all methods.
Revised: Here, four noisy datasets generated in four runs are the same for all methods.

Original: On the other hand, semantic noise is generated in a deterministic way.
Revised: On the other hand, semantic noise is generated in a deterministic way.

Original: Thus, only network initialization is randomized for each run of semantic noise.
Revised: Thus, only network initialization is randomized for each run of semantic noise.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: v
Revised: v

Original: [Figure]
Revised: [Figure]

Original: c i s i
Revised: c i s i

Original: [Figure]
Revised: [Figure]

Original: 4. E FFECTIVENESS OF LEC S AT IDENTIFYING NOISY EXAMPLES
Revised: 4. E FFECTIVENESS OF LEC S AT IDENTIFYING NOISY EXAMPLES

Original: Comparison with Self-training.
Revised: Comparison with Self-training.

Original: In Section 3.1, we argue that (100- (cid:15) )% small-loss examples may be corrupted.
Revised: In Section 3.1, we argue that (100- (cid:15) )% small-loss examples may be corrupted.

Original: To show this, we run LEC with M = 1 that is a method of training on (100- (cid:15) )
Revised: To show this, we run LEC with M = 1 that is a method of training on (100- (cid:15) )

Original: % smallloss examples.
Revised: % smallloss examples.

Original: Note that this method is similar to the idea of Jiang et al.
Revised: Note that this method is similar to the idea of Jiang et al.

Original: ; Shen & Sanghavi (2019).
Revised: ; Shen & Sanghavi (2019).

Original: We call it Self-training for simplicity.
Revised: We call it Self-training for simplicity.

Original: Figure 1 shows the label precision of Self-training is low especially on the high level of noise, i.e., sym-60%.
Revised: Figure 1 shows the label precision of Self-training is low especially on the high level of noise, i.e., sym-60%.

Original: Compared to Self-training, three LECs are trained on a higher label precision of examples through the ﬁltering process, achieving higher test accuracy as shown in Table 1.
Revised: Compared to Self-training, three LECs are trained on a higher label precision of examples through the ﬁltering process, achieving higher test accuracy as shown in Table 1.

Original: Out of these three, LTEC performs the best in both label precision and test accuracy.
Revised: Out of these three, LTEC performs the best in both label precision and test accuracy.

Original: Noisy examples are removed through ensemble consensus.
Revised: Noisy examples are removed through ensemble consensus.

Original: In LTEC, we ﬁrst obtain (100- (cid:15) )
Revised: In LTEC, we ﬁrst obtain (100- (cid:15) )

Original: % small-loss examples of the current network and then train only on the intersection of small-loss examples of the current and preceding networks.
Revised: % small-loss examples of the current network and then train only on the intersection of small-loss examples of the current and preceding networks.

Original: We plot label precisions of small-loss examples of the current network (in green) and the intersection (in red) at each epoch during running LTEC on CIFAR-10 with random noise in Figure 2.
Revised: We plot label precisions of small-loss examples of the current network (in green) and the intersection (in red) at each epoch during running LTEC on CIFAR-10 with random noise in Figure 2.

Original: We observe that label precision of the intersection is always higher, indicating that noisy examples are removed through ensemble consensus ﬁltering.
Revised: We observe that label precision of the intersection is always higher, indicating that noisy examples are removed through ensemble consensus ﬁltering.

Original: 4. C OMPARISON WITH STATE - OF - THE - ART METHODS
Revised: 4. C OMPARISON WITH STATE - OF - THE - ART METHODS

Original: Competing methods.
Revised: Competing methods.

Original: The competing methods include a regular training method: Standard , a method of training with corrected labels: D2L (Ma et al., 2018), a method of training with modiﬁed loss function based on the noise distribution: Forward (Patrini et al., 2017), and a method of exploiting small-loss examples: Co-teaching (Han et al., 2018).
Revised: The competing methods include a regular training method: Standard , a method of training with corrected labels: D2L (Ma et al., 2018), a method of training with modiﬁed loss function based on the noise distribution: Forward (Patrini et al., 2017), and a method of exploiting small-loss examples: Co-teaching (Han et al., 2018).

Original: We tune all the methods individually as described in Section A.1.4.
Revised: We tune all the methods individually as described in Section A.1.4.

Original: [Figure]
Revised: [Figure]

Original: Standard
Revised: Standard

Original: [Figure]
Revised: [Figure]

Original: Forward
Revised: Forward

Original: Co
Revised: Co

Original: [Figure]
Revised: [Figure]

Original: teaching
Revised: teaching

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Results on MNIST/CIFAR with random label noise.
Revised: Results on MNIST/CIFAR with random label noise.

Original: The overall results can be found in Figures 3 and 4, and Table 2.
Revised: The overall results can be found in Figures 3 and 4, and Table 2.

Original: We plot the average as a solid line and the standard deviation as a shadow around the line.
Revised: We plot the average as a solid line and the standard deviation as a shadow around the line.

Original: Figure 3 states that the test accuracy of D2L increases at the low level of label noise as training progresses, but it does not increase at the high level of label noise.
Revised: Figure 3 states that the test accuracy of D2L increases at the low level of label noise as training progresses, but it does not increase at the high level of label noise.

Original: This is because D2L puts large weights on given labels in the early stage of training even under the high level of noise.
Revised: This is because D2L puts large weights on given labels in the early stage of training even under the high level of noise.

Original: Forward shows its strength only in limited scenarios such as MNIST.
Revised: Forward shows its strength only in limited scenarios such as MNIST.

Original: Co-teaching does not work well on CIFAR-100 with asym-40%, indicating that its cross-training scheme is vulnerable to smallloss examples of a low label precision (see Figure 4).
Revised: Co-teaching does not work well on CIFAR-100 with asym-40%, indicating that its cross-training scheme is vulnerable to smallloss examples of a low label precision (see Figure 4).

Original: Unlike Co-teaching, our methods attempt to remove noisy examples in small-loss examples.
Revised: Unlike Co-teaching, our methods attempt to remove noisy examples in small-loss examples.

Original: Thus, on CIFAR-100 with asym-40% noise, both
Revised: Thus, on CIFAR-100 with asym-40% noise, both

Original: LTEC and LTEC-full surpass Co-teaching by a wide margin of about 6% and 5%, respectively.
Revised: LTEC and LTEC-full surpass Co-teaching by a wide margin of about 6% and 5%, respectively.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Results on CIFAR with open-set noise.
Revised: Results on CIFAR with open-set noise.

Original: The overall results can be found in Table 3.
Revised: The overall results can be found in Table 3.

Original: All the methods including LTEC and LTEC-full perform well under open-set noise.
Revised: All the methods including LTEC and LTEC-full perform well under open-set noise.

Original: We speculate that this is due to a low correlation between open-set noisy examples.
Revised: We speculate that this is due to a low correlation between open-set noisy examples.

Original: This is supported by the results on
Revised: This is supported by the results on

Original: CIFAR-10 where all the methods perform better on ImageNet-32 noise than on CIFAR-100 noise, as ImageNet-32 has more classes than CIFAR-100.
Revised: CIFAR-10 where all the methods perform better on ImageNet-32 noise than on CIFAR-100 noise, as ImageNet-32 has more classes than CIFAR-100.

Original: Similar to poorly annotated examples, out-ofdistribution examples are difﬁcult to be generalized during the warming-up process.
Revised: Similar to poorly annotated examples, out-ofdistribution examples are difﬁcult to be generalized during the warming-up process.

Original: Therefore, they can be removed from training batches through ensemble consensus ﬁltering.
Revised: Therefore, they can be removed from training batches through ensemble consensus ﬁltering.

Original: Results on CIFAR with semantic noise.
Revised: Results on CIFAR with semantic noise.

Original: The overall results can be found in Table 4.
Revised: The overall results can be found in Table 4.

Original: The semantically generated noisy examples are highly correlated with each other, making it difﬁcult to ﬁlter out those examples through ensemble consensus.
Revised: The semantically generated noisy examples are highly correlated with each other, making it difﬁcult to ﬁlter out those examples through ensemble consensus.

Original: Intuitively, using a bigger M makes ensemble consensus more conservative, ﬁltering out more examples.
Revised: Intuitively, using a bigger M makes ensemble consensus more conservative, ﬁltering out more examples.

Original: Thus, we use 10 as the value of M for semantic noise.
Revised: Thus, we use 10 as the value of M for semantic noise.

Original: On CIFAR with semantic noise, LTEC and LTEC-full perform comparably or best, compared to the other methods.
Revised: On CIFAR with semantic noise, LTEC and LTEC-full perform comparably or best, compared to the other methods.

Original: Between these two, LTEC-full performs better on 40% semantic noise due to its training stability.
Revised: Between these two, LTEC-full performs better on 40% semantic noise due to its training stability.

Original:  D ISCUSSION 5. H ARD - TO - CLASSIFY BUT CLEAN EXAMPLES
Revised:  D ISCUSSION 5. H ARD - TO - CLASSIFY BUT CLEAN EXAMPLES

Original: It is hard to learn all clean examples during the warming-up process.
Revised: It is hard to learn all clean examples during the warming-up process.

Original: Therefore, clean examples with high losses may be excluded from training batches during the ﬁltering process.
Revised: Therefore, clean examples with high losses may be excluded from training batches during the ﬁltering process.

Original: However, we expect that the number of clean examples used for training would increase gradually as training proceeds
Revised: However, we expect that the number of clean examples used for training would increase gradually as training proceeds

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: since LEC allows the network to generalize clean examples without overﬁtting.
Revised: since LEC allows the network to generalize clean examples without overﬁtting.

Original: To conﬁrm this
Revised: To conﬁrm this

Original: [Table]
Revised: [Table]

Original: we measure recall deﬁned by the ratio of clean examples used for training to all clean examples at the end of each epoch during running LTEC and LTEC-full.
Revised: we measure recall deﬁned by the ratio of clean examples used for training to all clean examples at the end of each epoch during running LTEC and LTEC-full.

Original: As expected, recalls of both LTEC and
Revised: As expected, recalls of both LTEC and

Original: LTEC-full sharply increase in the ﬁrst 50 epochs as described in Figure 5.
Revised: LTEC-full sharply increase in the ﬁrst 50 epochs as described in Figure 5.

Original: Pre-training (Hendrycks et al., 2019) prior to the ﬁltering process may help in reducing the number of clean examples removed from training batches.
Revised: Pre-training (Hendrycks et al., 2019) prior to the ﬁltering process may help in reducing the number of clean examples removed from training batches.

Original: 5. A BLATION STUDY
Revised: 5. A BLATION STUDY

Original: The number of networks used for ﬁltering.
Revised: The number of networks used for ﬁltering.

Original: During the ﬁltering process of LEC, we use only the intersection of small-loss examples of M perturbed networks for training.
Revised: During the ﬁltering process of LEC, we use only the intersection of small-loss examples of M perturbed networks for training.

Original: Thus, the number of examples used for training highly depends on M .
Revised: Thus, the number of examples used for training highly depends on M .

Original: To understand the effect of M , we run LTEC with varying M on CIFAR-10 with random label noise.
Revised: To understand the effect of M , we run LTEC with varying M on CIFAR-10 with random label noise.

Original: In particular, the range of M is {1, 3, 5, 7, 9, ∞ }.
Revised: In particular, the range of M is {1, 3, 5, 7, 9, ∞ }.

Original: Table 5 shows that a larger M is not always lead to better performance.
Revised: Table 5 shows that a larger M is not always lead to better performance.

Original: This is because too many examples may be removed from training batches as M increases.
Revised: This is because too many examples may be removed from training batches as M increases.

Original: Indeed, the total number of examples used for training is critical for the robustness as claimed in Rolnick et al.
Revised: Indeed, the total number of examples used for training is critical for the robustness as claimed in Rolnick et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: Noise ratio.
Revised: Noise ratio.

Original: In reality, only a poorly estimated noise ratio may be accessible.
Revised: In reality, only a poorly estimated noise ratio may be accessible.

Original: To study the effect of poor noise estimates, we run LTEC on CIFAR-10 with random label noise using a bit lower and higher values than the actual noise ratio as in Han et al.
Revised: To study the effect of poor noise estimates, we run LTEC on CIFAR-10 with random label noise using a bit lower and higher values than the actual noise ratio as in Han et al.

Original: We also run Co-teaching that requires the noise ratio for comparison.
Revised: We also run Co-teaching that requires the noise ratio for comparison.

Original: The overall results can be found in Table 6.
Revised: The overall results can be found in Table 6.

Original: Since it is generally difﬁcult to learn all clean examples, training on small-loss examples selected using the over-estimated ratio ( i.e. , 1.1 (cid:15) ) is often helpful in both Co-teaching and LTEC.
Revised: Since it is generally difﬁcult to learn all clean examples, training on small-loss examples selected using the over-estimated ratio ( i.e. , 1.1 (cid:15) ) is often helpful in both Co-teaching and LTEC.

Original: In contrast, smallloss examples selected using the under-estimated ratio may be highly corrupted.
Revised: In contrast, smallloss examples selected using the under-estimated ratio may be highly corrupted.

Original: In this case, LTEC is robust to the estimation error of noise ratio, while Co-teaching is not.
Revised: In this case, LTEC is robust to the estimation error of noise ratio, while Co-teaching is not.

Original: Such robustness of LTEC against noise estimation error comes from ensemble consensus ﬁltering.
Revised: Such robustness of LTEC against noise estimation error comes from ensemble consensus ﬁltering.

Original: [Table]
Revised: [Table]

Original: Applicability to different architecture.
Revised: Applicability to different architecture.

Original: The key idea of LEC is rooted in the difference between generalization and memorization.
Revised: The key idea of LEC is rooted in the difference between generalization and memorization.

Original: Since this is an artifact of SGD optimization (Arpit et al., 2017), we expect that LEC can be applied to any architecture optimized with SGD.
Revised: Since this is an artifact of SGD optimization (Arpit et al., 2017), we expect that LEC can be applied to any architecture optimized with SGD.

Original: To support this, we run Standard and LTEC with ResNet-20 (He et al., 2016).
Revised: To support this, we run Standard and LTEC with ResNet-20 (He et al., 2016).

Original: The architecture is optimized based on Chollet et al.
Revised: The architecture is optimized based on Chollet et al.

Original: , achieving the ﬁnal test accuracy of 90.67% on clean CIFAR-10.
Revised: , achieving the ﬁnal test accuracy of 90.67% on clean CIFAR-10.

Original: Here, T w is set to 30.
Revised: Here, T w is set to 30.

Original: Table 7 shows LTEC (ResNet) beats Standard (ResNet) in both peak and ﬁnal accuracies, as expected.
Revised: Table 7 shows LTEC (ResNet) beats Standard (ResNet) in both peak and ﬁnal accuracies, as expected.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This work presents the method of generating and using the ensemble for robust training.
Revised: This work presents the method of generating and using the ensemble for robust training.

Original: We explore three simple perturbation methods to generate the ensemble and then develop the way of identifying noisy examples using the ensemble consensus on small-loss examples.
Revised: We explore three simple perturbation methods to generate the ensemble and then develop the way of identifying noisy examples using the ensemble consensus on small-loss examples.

Original: Along with growing attention to the use of small-loss examples for robust training, we expect that our ensemble method will be useful for such training methods.
Revised: Along with growing attention to the use of small-loss examples for robust training, we expect that our ensemble method will be useful for such training methods.

Original: A A PPENDIX A. I MPLEMENTATION DETAILS A.1. A NNOTATION NOISES
Revised: A A PPENDIX A. I MPLEMENTATION DETAILS A.1. A NNOTATION NOISES

Original: • Random label noise : The corruption matrices of sym- (cid:15) % and asym- (cid:15) % are described in Figures A1a and A1b, respectively.
Revised: • Random label noise : The corruption matrices of sym- (cid:15) % and asym- (cid:15) % are described in Figures A1a and A1b, respectively.

Original: • Open-set noise : To generate CIFAR-10 with open-set noise, we sample images fromclasses of CIFAR-100 (Abbasi et al., 2018) and 748 classes of ImageNet (Oliver et al., 2018) to avoid sampling similar images with CIFAR-10.
Revised: • Open-set noise : To generate CIFAR-10 with open-set noise, we sample images fromclasses of CIFAR-100 (Abbasi et al., 2018) and 748 classes of ImageNet (Oliver et al., 2018) to avoid sampling similar images with CIFAR-10.

Original: • Semantic noise : To generate semantic noise, we choose uncertain images and then mislabel them ambiguously.
Revised: • Semantic noise : To generate semantic noise, we choose uncertain images and then mislabel them ambiguously.

Original: In Figure A2, we see that clean examples have simple and easy images, while noisy examples have not.
Revised: In Figure A2, we see that clean examples have simple and easy images, while noisy examples have not.

Original: Also, its corruption matrix (see Figure A1c) describes the similarity between classes, e.g., cat and dog, car and truck, and so forth.
Revised: Also, its corruption matrix (see Figure A1c) describes the similarity between classes, e.g., cat and dog, car and truck, and so forth.

Original: planecar bird catdeerdogfroghorseshiptruckLabel plane car bird cat deer dog frog horse ship truck
Revised: planecar bird catdeerdogfroghorseshiptruckLabel plane car bird cat deer dog frog horse ship truck

Original: T r u e l a b e l symmetric random noise 0.0 0.
Revised: T r u e l a b e l symmetric random noise 0.0 0.

Original: (a) Random-sym planecar bird catdeerdogfroghorseshiptruckLabel plane car bird cat deer dog frog horse ship truck
Revised: (a) Random-sym planecar bird catdeerdogfroghorseshiptruckLabel plane car bird cat deer dog frog horse ship truck

Original: T r u e l a b e l asymmetric random noise 0.0 0.
Revised: T r u e l a b e l asymmetric random noise 0.0 0.

Original: (b
Revised: (b

Original: ) Random
Revised: ) Random

Original: -asym planecar bird catdeerdogfroghorseshiptruckLabel plane car bird cat deer dog frog horse ship truck
Revised: -asym planecar bird catdeerdogfroghorseshiptruckLabel plane car bird cat deer dog frog horse ship truck

Original: T r u e l a b e l semantic noise 0.0 0.
Revised: T r u e l a b e l semantic noise 0.0 0.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: cat
Revised: cat

Original: [Figure]
Revised: [Figure]

Original: frog
Revised: frog

Original: A.1. A RCHITECTURE AND OPTIMIZATION DETAILS
Revised: A.1. A RCHITECTURE AND OPTIMIZATION DETAILS

Original: The 9-convolutional layer architecture used in this study can be found in Table A1.
Revised: The 9-convolutional layer architecture used in this study can be found in Table A1.

Original: The network is optimized with Adam (Kingma & Ba, 2014) with a batchsize of 128 for 200 epochs.
Revised: The network is optimized with Adam (Kingma & Ba, 2014) with a batchsize of 128 for 200 epochs.

Original: The learning rate α is initially set to 0.1.
Revised: The learning rate α is initially set to 0.1.

Original: The learning rate is linearly annealed to zero during the lastepochs for MNIST and CIFAR-10, and during the last 100 epochs for CIFAR-100.
Revised: The learning rate is linearly annealed to zero during the lastepochs for MNIST and CIFAR-10, and during the last 100 epochs for CIFAR-100.

Original: The momentum parameters β 1 and β 2 are set to 0.9 and 0.999, respectively.
Revised: The momentum parameters β 1 and β 2 are set to 0.9 and 0.999, respectively.

Original: β 1 is linearly annealed to 0.1 during the last 120 epochs for MNIST and CIFAR-10, and during the last 100 epochs for CIFAR-100.
Revised: β 1 is linearly annealed to 0.1 during the last 120 epochs for MNIST and CIFAR-10, and during the last 100 epochs for CIFAR-100.

Original: The images of CIFAR are divided by 255 and are whitened with ZCA.
Revised: The images of CIFAR are divided by 255 and are whitened with ZCA.

Original: Additional regularizations such as data augmentation are not applied.
Revised: Additional regularizations such as data augmentation are not applied.

Original: The results on clean MNIST, CIFAR-10, and CIFAR-100 can be found in Table A2.
Revised: The results on clean MNIST, CIFAR-10, and CIFAR-100 can be found in Table A2.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A.1. P SEUDOCODES FOR LEC S
Revised: A.1. P SEUDOCODES FOR LEC S

Original: We present three LECs with different perturbations in Section 3.3.
Revised: We present three LECs with different perturbations in Section 3.3.

Original: The pseudocodes for LNEC,
Revised: The pseudocodes for LNEC,

Original: LSEC, LTEC, and LTEC-full are described in the following.
Revised: LSEC, LTEC, and LTEC-full are described in the following.

Original: In LTEC-full, we obtain small-loss examples utilized for ﬁltering from the second epoch to encourage its stability.
Revised: In LTEC-full, we obtain small-loss examples utilized for ﬁltering from the second epoch to encourage its stability.

Original: Algorithm A1 LNEC
Revised: Algorithm A1 LNEC

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Require
Revised: Require

Original: [Table] [Table]
Revised: [Table] [Table]

Original: randomly
Revised: randomly

Original: [Table] [Table]
Revised: [Table] [Table]

Original: do
Revised: do

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4: for
Revised: 4: for

Original: [Table]
Revised: [Table]

Original: do
Revised: do

Original: [Table]
Revised: [Table]

Original: Sample a
Revised: Sample a

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 8: if
Revised: 8: if

Original: [Table]
Revised: [Table]

Original: <
Revised: <

Original: [Table]
Revised: [Table]

Original: then (cid:73)
Revised: then (cid:73)

Original: [Table]
Revised: [Table]

Original: : else (cid:73)
Revised: : else (cid:73)

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: : else if
Revised: : else if

Original: [Table]
Revised: [Table]

Original: then
Revised: then

Original: [Table] [Table]
Revised: [Table] [Table]

Original: : else
Revised: : else

Original: [Table] [Table]
Revised: [Table] [Table]

Original: : end if
Revised: : end if

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 19: end if 20: end for 21: end for
Revised: 19: end if 20: end for 21: end for

Original: Require
Revised: Require

Original: [Table]
Revised: [Table]

Original: A.1. C OMPETING METHODS
Revised: A.1. C OMPETING METHODS

Original: The competing methods include a regular training method: Standard , a method of training with corrected labels: D2L (Ma et al., 2018), a method of training with modiﬁed loss function based on the noise distribution: Forward (Patrini et al., 2017), and a method of exploiting small-loss examples: Co-teaching (Han et al., 2018).
Revised: The competing methods include a regular training method: Standard , a method of training with corrected labels: D2L (Ma et al., 2018), a method of training with modiﬁed loss function based on the noise distribution: Forward (Patrini et al., 2017), and a method of exploiting small-loss examples: Co-teaching (Han et al., 2018).

Original: We tune all the methods individually as follows:
Revised: We tune all the methods individually as follows:

Original: • Standard : The network is trained using the cross-entropy loss.
Revised: • Standard : The network is trained using the cross-entropy loss.

Original: • D2L :
Revised: • D2L :

Original: The input vector of a fully connected layer in the architecture is used to measure the LID estimates.
Revised: The input vector of a fully connected layer in the architecture is used to measure the LID estimates.

Original: The parameter involved with identifying the turning point, window size W is set to 12.
Revised: The parameter involved with identifying the turning point, window size W is set to 12.

Original: The network is trained using original labels until the turning point is found and then trained using the bootstrapping target with adaptively tunable mixing coefﬁcient.
Revised: The network is trained using original labels until the turning point is found and then trained using the bootstrapping target with adaptively tunable mixing coefﬁcient.

Original: • Forward :
Revised: • Forward :

Original: Prior to training, the corruption matrix C where C ji = P
Revised: Prior to training, the corruption matrix C where C ji = P

Original: ( y = i | y true =
Revised: ( y = i | y true =

Original: j ) is estimated based on the 97 th percentile of probabilities for each class on MNIST and CIFAR-
Revised: j ) is estimated based on the 97 th percentile of probabilities for each class on MNIST and CIFAR-

Original: 10, and the 100 th percentile of probabilities for each class on CIFAR-100 as in Hendrycks et al.
Revised: 10, and the 100 th percentile of probabilities for each class on CIFAR-100 as in Hendrycks et al.

Original: The network is then trained using the corrected labels for 200 epochs.
Revised: The network is then trained using the corrected labels for 200 epochs.

Original: Co-teaching : Two networks are employed.
Revised: Co-teaching : Two networks are employed.

Original: At every update, they select their small-loss examples within a minibatch and then provide them to each other.
Revised: At every update, they select their small-loss examples within a minibatch and then provide them to each other.

Original: The ratio of selected examples based on training losses is linearly annealed from 100% to (100- (cid:15) )% over the ﬁrst 10 epochs.
Revised: The ratio of selected examples based on training losses is linearly annealed from 100% to (100- (cid:15) )% over the ﬁrst 10 epochs.

Original: A. C OMPLEXITY ANALYSIS
Revised: A. C OMPLEXITY ANALYSIS

Original: We compute space complexity as the number of network parameters and computational complexity as the number of forward and backward passes.
Revised: We compute space complexity as the number of network parameters and computational complexity as the number of forward and backward passes.

Original: Here we assume that early stopping is not used and the noise ratio of (cid:15) % is given.
Revised: Here we assume that early stopping is not used and the noise ratio of (cid:15) % is given.

Original: Note that the computational complexity of each method depends on its hyperparameter values, e.g., the duration of the warming-up process T w and the noise ratio (cid:15) %.
Revised: Note that the computational complexity of each method depends on its hyperparameter values, e.g., the duration of the warming-up process T w and the noise ratio (cid:15) %.

Original: The analysis is reported in Table A3.
Revised: The analysis is reported in Table A3.

Original: Our proposed LTEC is the most efﬁcient because it can be implemented with a single network based on Section A.1.3 and it updates only a subset of the entire set after the warming-up process.
Revised: Our proposed LTEC is the most efﬁcient because it can be implemented with a single network based on Section A.1.3 and it updates only a subset of the entire set after the warming-up process.

Original: [Table]
Revised: [Table]

Original: A. A DDITIONAL RESULTS A.3. R ESULTS OF LTEC WITH M = ∞
Revised: A. A DDITIONAL RESULTS A.3. R ESULTS OF LTEC WITH M = ∞

Original: Figure A3 shows that ensemble consensus ﬁltering with too large M removes even clean examples from training batches in the early stage of the ﬁltering process.
Revised: Figure A3 shows that ensemble consensus ﬁltering with too large M removes even clean examples from training batches in the early stage of the ﬁltering process.

Original: Unlike to LTEC with M = 5 , the recall of LTEC with M = ∞ does not increase as training proceeds, suggesting that its generalization ability is not improved.
Revised: Unlike to LTEC with M = 5 , the recall of LTEC with M = ∞ does not increase as training proceeds, suggesting that its generalization ability is not improved.

Original: This shows that a larger M does not always lead to better performance.
Revised: This shows that a larger M does not always lead to better performance.

Original: We expect that pre-training (Hendrycks et al., 2019) prior to the ﬁltering process helps in reducing the number of clean examples removed by ensemble consensus ﬁltering regardless of M
Revised: We expect that pre-training (Hendrycks et al., 2019) prior to the ﬁltering process helps in reducing the number of clean examples removed by ensemble consensus ﬁltering regardless of M

Original: [Figure]
Revised: [Figure]