Original: Robustness to Label Noise Depends on the Shape of the Noise Distributionin Feature Space
Revised: Robustness to Label Noise Depends on the Shape of the Noise Distribution

Original: Submission number:

Original: Anonymous Author(s)

Original: Affilliation email

Original: Abstract
Revised: Abstract

Original: Machine learning classifiers have been demonstrated, both empirically and theoretically, to be robust to label noise under certain conditions — notably the typicalassumption is that label noise is independent of the features given the class label.
Revised: Machine learning classifiers have been demonstrated, both empirically and theoretically, to be robust to label noise under certain conditions — notably the typical assumption is that label noise is independent of the features given the class label.

Original: We provide a theoretical framework that generalizes beyond this typical assumptionby modeling label noise as a distribution over feature space.
Revised: We provide a theoretical framework that generalizes beyond this typical assumption by modeling label noise as a distribution over feature space.

Original: We show that both thescale and the shape of the noise distribution influence the posterior likelihood; andthe shape of the noise distribution has a stronger impact on classification performance if the noise is concentrated in feature space where the decision boundarycan be moved.
Revised: We show that both the scale and the shape of the noise distribution influence the posterior likelihood; and the shape of the noise distribution has a stronger impact on classification performance if the noise is concentrated in feature space where the decision boundary can be moved.

Original: For the special case of uniform label noise (independent of featuresand the class label), we show that the Bayes optimal classifier for c classes isrobust to label noise until the ratio of noisy samples goes above c − 1 c (e.g. 90%for 10 classes), which we call the tipping point .
Revised: For the special case of uniform label noise (independent of features and the class label), we show that the Bayes optimal classifier for c classes is robust to label noise until the ratio of noisy samples goes above c − 1 c (e.g. 90% for 10 classes), which we call the tipping point .

Original: However, for the special caseof class-dependent label noise (independent of features given the class label), thetipping point can be as low as 50%.
Revised: However, for the special case of class-dependent label noise (independent of features given the class label), the tipping point can be as low as 50%.

Original: Most importantly, we show that when thenoise distribution targets decision boundaries (label noise is directly dependenton feature space), classification robustness can drop off even at a small scale ofnoise.
Revised: Most importantly, we show that when the noise distribution targets decision boundaries (label noise is directly dependent on feature space), classification robustness can drop off even at a small scale of noise.

Original: Even when evaluating recent label-noise mitigation methods we see reducedaccuracy when label noise is dependent on features.
Revised: Even when evaluating recent label-noise mitigation methods we see reduced accuracy when label noise is dependent on features.

Original: These findings explain whymachine learning often handles label noise well if the noise distribution is uniformin feature-space; yet it also points to the difficulty of overcoming label noise whenit is concentrated in a region of feature space where a decision boundary can move.
Revised: These findings explain why machine learning often handles label noise well if the noise distribution is uniform in feature-space; yet it also points to the difficulty of overcoming label noise when it is concentrated in a region of feature space where a decision boundary can move.

Original:  Introduction
Revised:  Introduction

Original: An open question in machine learning is how the quality of training data, especially the labels, affectsthe learned model [2, 4, 13, 21, 23].
Revised: An open question in machine learning is how the quality of training data, especially the labels, affects the learned model [2, 4, 13, 22, 24].

Original: As an example shown in Figure 1, we imagine a classificationtask for detecting cancer based on observed biomarkers.
Revised: As an example shown in Figure 1, we imagine a classification task for detecting cancer based on observed biomarkers.

Original: The generative model assumes that thereis a hidden variable, called Y ∗ , denoting whether a person actually has cancer or not.
Revised: The generative model assumes that there is a hidden variable, called Y ∗ , denoting whether a person actually has cancer or not.

Original: Dependingon the state of Y ∗ , biomarkers can be observed, denoted as a feature vector x , drawn from somedistribution x ∼ P [ X | Y ∗ ] .
Revised: Depending on the state of Y ∗ , biomarkers can be observed, denoted as a feature vector x , drawn from some distribution x ∼ P [ X | Y ∗ ] .

Original: The goal of machine learning (ML) is to predict the presence of cancergiven the biomarkers — yet to train and evaluate the ML model, we have diagnoses, { y } , which maynot always be correct (
Revised: The goal of machine learning (ML) is to predict the presence of cancer given the biomarkers — yet to train and evaluate the ML model, we have diagnoses, { y } , which may not always be correct (

Original: y ̸
Revised: y ̸

Original: = y ∗ for some samples).
Revised: = y ∗ for some samples).

Original: The question of label uncertainty is thus: Whathappens when the observed labels, y , do not match the hidden true labels y ∗ ?
Revised: The question of label uncertainty is thus: What happens when the observed labels, y , do not match the hidden true labels y ∗ ?

Original: Our theory concludesthat the answer depends on the shape of the noisy distribution P
Revised: Our theory concludes that the answer depends on the shape of the noisy distribution P

Original: [ Y | X ] with respect to feature space.
Revised: [ Y | X ] with respect to feature space.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ExampleDisease ( cancer or not_cancer )
Revised: ExampleDisease ( cancer or not_cancer )

Original: Observed biomarkers
Revised: Observed biomarkers

Original: Diagnosis ( cancer or not_cancer )
Revised: Diagnosis ( cancer or not_cancer )

Original: [Table]
Revised: [Table]

Original: p ( Y =| x ﬂip ) p ( Y =| x ﬂip )
Revised: p ( Y =| x ﬂip ) p ( Y =| x ﬂip )

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: distributions: b) uniform , class -dependent, and d) g feature -dependent.

Original: Empirical research suggests that ML is often able to generalize beyond label noise injected in trainingdata to make accurate predictions, particularly when the noisy label is sampled independently ofthe features and the true label, i.e. P [ Y | X, Y ∗ ]
Revised: Empirical research suggests that ML is often able to generalize beyond label noise injected in training data to make accurate predictions, particularly when the noisy label is sampled independently of the features and the true label, i.e. P [ Y | X, Y ∗ ]

Original: = P [ Y ] [2–5, 21, 23, 24].
Revised: = P [ Y ] [2–5, 22, 24, 25].

Original: This is a remarkablefinding considering how messy such data appears, as shown in Fig.
Revised: This is a remarkable finding considering how messy such data appears, as shown in Fig.

Original: 2b.
Revised: 2b.

Original: However, this robustnessto label noise drops significantly when these approaches test their findings by synthesizing labelnoise with the assumption that the noise is independent of the features given the true class label, i.e.
Revised: However, this robustness to label noise drops significantly when these approaches test their findings by synthesizing label noise with the assumption that the noise is independent of the features given the true class label, i.e.

Original: P [ Y | X, Y ∗ ] =
Revised: P [ Y | X, Y ∗ ] =

Original: P [ Y | Y ∗ ] .
Revised: P [ Y | Y ∗ ] .

Original: In other words, for a given class, all samples in that class are equally likelyto have the same erroneous label, as demonstrated in Fig.
Revised: In other words, for a given class, all samples in that class are equally likely to have the same erroneous label, as demonstrated in Fig.

Original: 2c.
Revised: 2c.

Original: We theoretically analyze the posteriorlikelihood of the true class given noisy training labels to demonstrate that both of these types of labelnoise (uniform and class-dependent) behave similarly but with very different tipping points of noiselevel beyond which the true labels are not recoverable.
Revised: We theoretically analyze the posterior likelihood of the true class given noisy training labels to demonstrate that both of these types of label noise (uniform and class-dependent) behave similarly but with very different tipping points of noise level beyond which the true labels are not recoverable.

Original: While the assumption of label noise being independent of features is convenient for generatingnoisy samples [3, 5, 13, 27], it is not clear that it is an appropriate assumption [4, 23].
Revised: While the assumption of label noise being independent of features is convenient for generating noisy samples [3, 5, 13, 28], it is not clear that it is an appropriate assumption [4, 24].

Original: Morerealistically, we expect that label noise occurs when the features are ambiguous; such as whencontradictory biomarkers are observed in an oncology patient, or when an image to be classified isblurry.
Revised: More realistically, we expect that label noise occurs when the features are ambiguous; such as when contradictory biomarkers are observed in an oncology patient, or when an image to be classified is blurry.

Original: In other words, P [ Y | X, Y ∗ ] ̸ = P [ Y | Y ∗ ] .
Revised: In other words, P [ Y | X, Y ∗ ] ̸ = P [ Y | Y ∗ ] .

Original: Figs. 2d and 2f show data generated with two noisedistributions such that label noise is most likely to occur in regions of feature-space that are nearclass boundaries; while Figs.
Revised: Figs. 2d and 2f show data generated with two noise distributions such that label noise is most likely to occur in regions of feature-space that are near class boundaries; while Figs.

Original: 2e and 2g show data where label flips are most likely to occur far fromtrue class boundaries.
Revised: 2e and 2g show data where label flips are most likely to occur far from true class boundaries.

Original: Our theory and empirical work demonstrate that feature-dependent label flipsthat create noisy class boundaries far from the true class boundaries have the most negative impacton classification accuracy, even at levels of label noise much lower than the tipping points seen inuniform and class-dependent label noise.
Revised: Our theory and empirical work demonstrate that feature-dependent label flips that create noisy class boundaries far from the true class boundaries have the most negative impact on classification accuracy, even at levels of label noise much lower than the tipping points seen in uniform and class-dependent label noise.

Original: We demonstrate theoretically and empirically that classification is generally robust to uniform andclass-dependent label noise until the scale of the noise exceeds a threshold that depends on the“spread" of the noise distribution; but that beyond this tipping point, classification accuracy declinesrapidly.
Revised: We demonstrate theoretically and empirically that classification is generally robust to uniform and class-dependent label noise until the scale of the noise exceeds a threshold that depends on the “spread" of the noise distribution; but that beyond this tipping point, classification accuracy declines rapidly.

Original: Yet, we also demonstrate that such robustness to label noise is misleading; because ourintroduction of feature-dependent label noise shows that classification accuracy can be loweredsignificantly even for small amounts of label noise.
Revised: Yet, we also demonstrate that such robustness to label noise is misleading; because our introduction of feature-dependent label noise shows that classification accuracy can be lowered significantly even for small amounts of label noise.

Original: We evaluate, for the first time, the damagingeffect of feature-dependent label noise on recent strategies for mitigating label noise.
Revised: We evaluate, for the first time, the damaging effect of feature-dependent label noise on recent strategies for mitigating label noise.

Original: We concludethat the shape of the label noise distribution with respect to feature space is a significant factor inlabel noise robustness.
Revised: We conclude that the shape of the label noise distribution with respect to feature space is a significant factor in label noise robustness.

Original: Our theoretical findings should guide future theoretical and empirical work onlabel noise to focus studies on the most informative combinations of noise scale and shape; savingsignificant computational effort while improving the research community’s ability to gain insightabout how label noise affects learned models.
Revised: Our theoretical findings should guide future theoretical and empirical work on label noise to focus studies on the most informative combinations of noise scale and shape; saving significant computational effort while improving the research community’s ability to gain insight about how label noise affects learned models.

Original:  Preliminaries2.1 Problem Statement
Revised:  Preliminaries 2.1 Problem Statement

Original: We consider the problem of label noise in which, for given data sample, ( X, Y ∗ ) ; for some fraction ofsamples, the true labels Y ∗ ∈ { 1 , . .
Revised: We consider the problem of label noise in which, for given data sample, ( X, Y ∗ ) ; for some fraction of samples, the true labels Y ∗ ∈ { 1 , . .

Original: . , c } are replaced with incorrect (noisy) labels Y .
Revised: . , c } are replaced with incorrect (noisy) labels Y .

Original: The fraction ofnoisy samples is called the noise level .
Revised: The fraction of noisy samples is called the noise level .

Original: The typical paradigm for modelling such noise is using labelflips [16, 21], where for each sample, there is some probability that the true label will be flipped to anincorrect label.
Revised: The typical paradigm for modelling such noise is using label flips [16, 22], where for each sample, there is some probability that the true label will be flipped to an incorrect label.

Original: Which label it flips to is typically modelled in one of two ways: 1) uniform noise(also known as symmetric label noise): each incorrect class is equally likely; or 2) class-dependentnoise (also known as asymmetric label noise) in which some classes are more likely than others for agiven true class.
Revised: Which label it flips to is typically modelled in one of two ways: 1) uniform noise (also known as symmetric label noise): each incorrect class is equally likely; or 2) class-dependent noise (also known as asymmetric label noise) in which some classes are more likely than others for a given true class.

Original: In the case of class-dependent noise, the noise function is typically represented as ac × c label transition matrix describing the probably of a transition from Y ∗ to Y .
Revised: In the case of class-dependent noise, the noise function is typically represented as a c × c label transition matrix describing the probably of a transition from Y ∗ to Y .

Original: The number ofnon-zero off-diagonal elements in each row of this matrix (or inversely, the sparsity) characterizes thespread of the noise distribution [13].
Revised: The number of non-zero off-diagonal elements in each row of this matrix (or inversely, the sparsity) characterizes the spread of the noise distribution [13].

Original: In other words, when spread is c − 1 , the noise distribution isuniform, but the lower the sparsity, the further from uniform the distribution is.
Revised: In other words, when spread is c − 1 , the noise distribution is uniform, but the lower the sparsity, the further from uniform the distribution is.

Original: 2.2 Related Work
Revised: 2.2 Related Work

Original: Several methods have been developed to improve classifier robustness to noisy labels during training.
Revised: Several methods have been developed to improve classifier robustness to noisy labels during training.

Original: CleanLab removes noisy samples from training [13].
Revised: CleanLab removes noisy samples from training [13].

Original: Methods like CoTeaching weight the samples[5, 20].
Revised: Methods like CoTeaching weight the samples [5, 21].

Original: Approximate expectation-maximization infers the true label [4].
Revised: Approximate expectation-maximization infers the true label [4].

Original: MentorNet uses a teacherstudent learning approach [7].
Revised: MentorNet uses a teacherstudent learning approach [7].

Original: MixUp interpolates pairs of examples and their labels [27].
Revised: MixUp interpolates pairs of examples and their labels [28].

Original: SoftLabelstreats the labels as learnable parameters [23].
Revised: SoftLabels treats the labels as learnable parameters [24].

Original: Symmetric cross-entropy (SCE) uses a robust lossfunction [24].
Revised: Symmetric cross-entropy (SCE) uses a robust loss function [25].

Original: Similar to our paper, [3] evaluates robustness of existing approaches to label noise.
Revised: Similar to our paper, [3] evaluates robustness of existing approaches to label noise.

Original: Chen et al.
Revised: Chen et al.

Original: [2] shows that the noisy test accuracy is a quadratic function of the noise level for the caseof uniform label noise.
Revised: [2] shows that the noisy test accuracy is a quadratic function of the noise level for the case of uniform label noise.

Original: Yet, none of these approaches are yet evaluated on feature-dependent noise.
Revised: Yet, none of these approaches are yet evaluated on feature-dependent noise.

Original: A few papers argue for the need to consider feature-dependent label noise, but they do not evaluatetheir approaches in this setting [4, 23].
Revised: A few papers argue for the need to consider feature-dependent label noise, but they do not evaluate their approaches in this setting [4, 24].

Original: Notably, two papers replace the label-flip paradigm withreal-world webly-labeled noise (which is likely feature-dependent) [8, 10], but simultaneously inducea domain shift which makes analysis challenging; and the dependence on the features is unknown.
Revised: Notably, two papers replace the label-flip paradigm with real-world webly-labeled noise (which is likely feature-dependent) [8, 10], but simultaneously induce a domain shift which makes analysis challenging; and the dependence on the features is unknown.

Original:  Theory
Revised:  Theory

Original: We show that the estimation of the true conditional distribution P [ Y ∗ | X ] depends on the level ofnoise, as well as the shape of the noise.
Revised: We show that the estimation of the true conditional distribution P [ Y ∗ | X ] depends on the level of noise, as well as the shape of the noise.

Original: We define the classification problem of c classes with labelflips; and examine different types of label noise: uniform, class-dependent, and feature-dependent.
Revised: We define the classification problem of c classes with label flips; and examine different types of label noise: uniform, class-dependent, and feature-dependent.

Original: Proofs are given in the Supplement.
Revised: Proofs are given in the Supplement.

Original: 3.1 Problem Definition: Classification with Label Flips
Revised: 3.1 Problem Definition: Classification with Label Flips

Original: In a classification problem with c classes, we are given an n-sample of pairs ( X, Y ∗ )
Revised: In a classification problem with c classes, we are given an n-sample of pairs ( X, Y ∗ )

Original: ∈ R d ×{ 1 , . . .
Revised: ∈ R d × { 1 , . . .

Original: , c } , with joint distribution determined by
Revised: , c } , with joint distribution determined by

Original: [Equation]
Revised: [Equation]

Original: and P [ Y

Original: [Equation] = k ] = [Equation] 1) for each k ∈ { 1 , . . .
Revised: for each k ∈ { 1 , . . .

Original: , c } .
Revised: , c } .

Original: We want to estimate the clean posterior :
Revised: We want to estimate the clean posterior :

Original: [Equation]
Revised: [Equation]

Original: Rather than observing Y ∗ , we observe the noisy label Y with probability P
Revised: Rather than observing Y ∗ , we observe the noisy label Y with probability P

Original: [ Y = i | Y ∗
Revised: [ Y = i | Y ∗

Original: =
Revised: =

Original: k, X =x ]
Revised: k, X = x ]

Original: = η ki ( x ) where (cid:80) ci =
Revised: = η ki ( x ) where (cid:80) ci =

Original: 1 η ki ( x ) = 1 .
Revised: 1 η ki ( x ) = 1 .

Original: In other words, Y | Y ∗ , X is multinomial distributed such that1 − η kk ( x ) is the probability of a label flip and each η ki ( x ) influences which noisy label, i , may bechosen.
Revised: In other words, Y | Y ∗ , X is multinomial distributed such that 1 − η kk ( x ) is the probability of a label flip and each η ki ( x ) influences which noisy label, i , may be chosen.

Original: The impact of label noise is that the estimate of the noisy posterior is
Revised: The impact of label noise is that the estimate of the noisy posterior is

Original: [Equation]
Revised: [Equation]

Original: 3.2 Uniform Label Noise
Revised: 3.2 Uniform Label Noise

Original: Classifiers are generally robust to uniform label noise up to a tipping point, as has been welldemonstrated empirically.
Revised: Classifiers are generally robust to uniform label noise up to a tipping point, as has been welldemonstrated empirically.

Original: Definition 3.1 (Uniform noise) For a classification problem with c classes, given the scale of labelnoise, 0 ≤ ϵ ≤ 1 , the distribution of noisy labels is constant such that:
Revised: Definition 3.1 (Uniform noise) For a classification problem with c classes, given the scale of label noise, 0 ≤ ϵ ≤ 1 , the distribution of noisy labels is constant such that:

Original: [Equation]
Revised: [Equation]

Original: Plugging the Definition 3.1 into the noisy posterior Eq 3; the noisy posterior under uniform noise is:
Revised: Plugging the Definition 3.1 into the noisy posterior Eq (3); the noisy posterior under uniform noise is:

Original: [Equation]
Revised: [Equation]

Original: Lemma 3.2 (Noisy accuracy)
Revised: Lemma 3.2 (Noisy accuracy)

Original: Given noisy train and test samples according to Definition 3.1, thenoisy test accuracy of a Bayes optimal classifier is quadratic with respect to the noise level ϵ :
Revised: Given noisy train and test samples according to Definition 3.1, the noisy test accuracy of a Bayes optimal classifier is quadratic with respect to the noise level ϵ :

Original: [Equation]
Revised: [Equation]

Original: Corollary 3.2.1 Given noisy train and test samples according to Definition 3.1, the minimum noisyaccuracy of 1 c occurs at ϵ = c − 1 c
Revised: Corollary 3.2.1 Given noisy train and test samples according to Definition 3.1, the minimum noisy accuracy of 1 c occurs at ϵ = c − 1 c

Original: Fig 3a plots the quadratic of Eq 6 which shows that models trained on noisy data will at first decreasein accuracy as tested on noisy labels, but then begin to fit the label noise rather than the true labels asthe fraction of noisy labels increases beyond the minimizing point.
Revised: Fig 3a plots the quadratic of Eq (6) which shows that models trained on noisy data will at first decrease in accuracy as tested on noisy labels, but then begin to fit the label noise rather than the true labels as the fraction of noisy labels increases beyond the minimizing point.

Original: Theorem 3.3 (Clean accuracy)
Revised: Theorem 3.3 (Clean accuracy)

Original: Given noisy train samples according to Definition 3.1 and cleantest samples, the clean test accuracy of a Bayes optimal classifier is logistic with respect to the noiselevel ϵ :
Revised: Given noisy train samples according to Definition 3.1 and clean test samples, the clean test accuracy of a Bayes optimal classifier is logistic with respect to the noise level ϵ :

Original: [Equation]
Revised: [Equation]

Original: where b is an arbitrary base of the exponent used in the softmax approximation to the argmax.
Revised: where b is an arbitrary base of the exponent used in the softmax approximation to the argmax.

Original: Thelarger the value of b , the closer to the true maximum function.
Revised: The larger the value of b , the closer to the true maximum function.

Original: Corollary 3.3.

Revised: Corollary 3.3.

Original: Given noisy train samples according to Definition 3.1 and clean test samples, cleanaccuracy drops to ¯ m ∗ k 2 at ϵ = c − 1 c .
Revised: Given noisy train samples according to Definition 3.1 and clean test samples, clean accuracy drops to ¯ m ∗ k 2 at ϵ = c − 1 c .

Original: [Figure]

Original: See Fig 3.
Revised: See Fig 3.

Original: For a dataset like ImageNet with 1000 classes, the clean accuracy would only drop offwhen the noise level is above 99 .
Revised: For a dataset like ImageNet with 1000 classes, the clean accuracy would only drop off when the noise level is above 99 .

Original: This finding is consistent with empirical studies [17] showingthat deep learning models are robust to this type of label noise; particularly when there are largenumbers of classes. 
Revised: This finding is consistent with empirical studies [17] showing that deep learning models are robust to this type of label noise; particularly when there are large numbers of classes. [Figure]

Original: 3.3 Class-Dependent Flips
Revised: 3.3 Class-Dependent Flips

Original: We consider the special case of class-dependent label noise that is also well-demonstrated empirically.
Revised: We consider the special case of class-dependent label noise that is also well-demonstrated empirically.

Original: Often, this class-dependent noise is defined such that η ki = 0 for several i .
Revised: Often, this class-dependent noise is defined such that η ki = 0 for several i .

Original: We refer to the numberof non-zero elements as the spread of the class-dependent noise function.
Revised: We refer to the number of non-zero elements as the spread of the class-dependent noise function.

Original: In the least-spread case,η kj = ϵ for some j ∈ { 1 , . .
Revised: In the least-spread case, η kj = ϵ for some j ∈ { 1 , . .

Original: . , c } \ { k } while η ki = 0 for all other values of i ∈ { 1 , . .
Revised: . , c } \ { k } while η ki = 0 for all other values of i ∈ { 1 , . .

Original: . , c } \ { k, j } .
Revised: . , c } \ { k, j } .

Original: Definition 3.4 (Class-dependent noise) For a classification problem with c classes, given the scaleof label noise, 0 ≤ ϵ ≤ 1 , and spread of noisy label distribution, 1 ≤ s ≤ c − 1 ; the distribution ofnoisy labels is constant within each class k ∈ { 1 , . .  ,c } :
Revised: Definition 3.4 (Class-dependent noise) For a classification problem with c classes, given the scale of label noise, 0 ≤ ϵ ≤ 1 , and spread of noisy label distribution, 1 ≤ s ≤ c − 1 ; the distribution of noisy labels is constant within each class k ∈ { 1 , . . . ,

Original: [Equation]
Revised: [Equation]

Original: (cid:40)
Revised: : η

Original: [Equation]
Revised: [Equation]

Original: , for i = k ϵ · t ki , ∀ i ∈ { 1 , . . .
Revised: , for i = k ϵ · t ki , ∀ i ∈ { 1 , . . .

Original: , c } \ { k } and (cid:80) i ̸
Revised: , c } \ { k } and (cid:80) i ̸

Original: = k | t ki | 

Revised: = k | t ki |

Original: =
Revised: =

Original: s and (cid:80) i ̸
Revised: s and (cid:80) i ̸

Original: = k t ki =(8)
Revised: = k t ki =(8)

Original: Plugging Definition 3.4 into the noisy posterior Eq 3 we see that the noisy posterior under classdependent noise depends on the spread, s , rather than the total number of classes:
Revised: Plugging Definition 3.4 into the noisy posterior Eq (3) we see that the noisy posterior under classdependent noise depends on the spread, s , rather than the total number of classes:

Original: [Equation]
Revised: [Equation]

Original: Lemma 3.5 (Noisy accuracy)
Revised: Lemma 3.5 (Noisy accuracy)

Original: Given noisy train and test samples according to Definition 3.4, thenoisy test accuracy of a Bayes optimal classifier is quadratic with respect to the noise level ϵ andspread s :
Revised: Given noisy train and test samples according to Definition 3.4, the noisy test accuracy of a Bayes optimal classifier is quadratic with respect to the noise level ϵ and spread s :

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.6 (Clean accuracy)
Revised: Theorem 3.6 (Clean accuracy)

Original: Given noisy train samples according to Definition 3.4 and cleantest samples, the clean test accuracy of a Bayes optimal classifier is logistic with respect to the noiselevel ϵ :
Revised: Given noisy train samples according to Definition 3.4 and clean test samples, the clean test accuracy of a Bayes optimal classifier is logistic with respect to the noise level ϵ :

Original: [Equation]
Revised: [Equation]

Original: Class-dependent noise impacts classification accuracy in a similar way to uniform noise; however,the tipping-point in class-dependent noise is controlled by the spread rather than by the total numberof classes.
Revised: Class-dependent noise impacts classification accuracy in a similar way to uniform noise; however, the tipping-point in class-dependent noise is controlled by the spread rather than by the total number of classes.

Original: Thus, even for a large number of classes, the tipping point can be as low as ϵ = 1 / 2 .
Revised: Thus, even for a large number of classes, the tipping point can be as low as ϵ = 1 / 2 .

Original: 3.4 Feature-Dependent Label Noise
Revised: 3.4 Feature-Dependent Label Noise

Original: We consider the more general paradigm of feature-dependent label noise which has not receivedmuch attention either theoretically or empirically.
Revised: We consider the more general paradigm of feature-dependent label noise which has not received much attention either theoretically or empirically.

Original: Consider the simplest feature-dependent label-flipscenario where a single sample ( X = x, Y ∗ = k ) for some fixed x with true label k will have noiseintroduced as ( X = x, Y = j ) for some j ∈ { 1 , . .
Revised: Consider the simplest feature-dependent label-flip scenario where a single sample ( X = x, Y ∗ = k ) for some fixed x with true label k will have noise introduced as ( X = x, Y = j ) for some j ∈ { 1 , . .

Original: , c } .
Revised: , c } .

Original: No matter which noisy label j appears,the posterior likelihood of the noisy data will be reduced to m k ( x ) =
Revised: No matter which noisy label j appears, the posterior likelihood of the noisy data will be reduced to m k ( x ) =

Original: (1 − p ( x )) m ∗ k
Revised: (1 − p ( x )) m ∗ k

Original: ( x ) ; while theposterior likelihood of the incorrect label will increase to m j ( x ) = m ∗ j ( x ) + p ( x )
Revised: ( x ) ; while the posterior likelihood of the incorrect label will increase to m j ( x ) = m ∗ j ( x ) + p ( x )

Original: m ∗ k ( x ) .
Revised: m ∗ k ( x ) .

Original: In the
Revised: In the

Original: Bayes optimal classifier, classification errors occur when:
Revised: Bayes optimal classifier, classification errors occur when:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: for true label k for given point x .
Revised: for true label k for given point x .

Original: Therefore, for a fixed x , the label-flip Y =
Revised: Therefore, for a fixed x , the label-flip Y =

Original: j that would have themost impact on classification (the greatest increase of the right-hand side of the inequality) is
Revised: j that would have the most impact on classification (the greatest increase of the right-hand side of the inequality) is

Original: [Equation]
Revised: [Equation]

Original: An immediate consequence of this observation is that worst-case label noise is a function of x ;therefore any evaluation of label noise (empirical or theoretical) that does not depend on features isoptimistic.
Revised: An immediate consequence of this observation is that worst-case label noise is a function of x ; therefore any evaluation of label noise (empirical or theoretical) that does not depend on features is optimistic.

Original: Now, consider which samples, i.e. the shape of η kk ( x ) , would have the most impact on cleanclassification accuracy.
Revised: Now, consider which samples, i.e. the shape of η kk ( x ) , would have the most impact on clean classification accuracy.

Original: By re-arranging Eq 12, we see that the ratio m k ( x ) / max i ∈{ 1 ,...,c }\ k m i ( x )defines the optimal classification boundaries.
Revised: By re-arranging Eq (12), we see that the ratio m k ( x ) / max i ∈{ 1 ,...,c }\ k m i ( x ) defines the optimal classification boundaries.

Original: By targeting label flips on samples x where this ratio ishigh, a new (noisy) classification boundary can be created; as illustrated in Fig.
Revised: By targeting label flips on samples x where this ratio is high, a new (noisy) classification boundary can be created; as illustrated in Fig.

Original: 1b.
Revised: 1b.

Original: Claim 3.7 (Worst-case noise)
Revised: Claim 3.7 (Worst-case noise)

Original: For a classification problem with c classes, the samples, x , that havethe greatest impact on clean label prediction are those that maximize the ratio between the likelihoodof the true class to the likelihood of the most-likely incorrect class:[Equation] for
Revised: For a classification problem with c classes, the samples, x , that have the greatest impact on clean label prediction are those that maximize the ratio between the likelihood of the true class to the likelihood of the most-likely incorrect class:

Original: [Equation]
Revised: [Equation]

Original: k

Original: [Equation]
Revised: [Equation]

Original: 3.5 Discussion of theoretical results
Revised: 3.5 Discussion of theoretical results

Original: Relationship to existing theory It is known that label noise is not generally distinguishable fromother label uncertainty (such as class overlap) [18, 19].
Revised: Relationship to existing theory It is known that label noise is not generally distinguishable from other label uncertainty (such as class overlap) [19, 20].

Original: Various approaches provide guarantees of labelnoise identification through constraints on m ∗ k ( x ) including “purity" of classes [1, 11, 15, 18, 19].
Revised: Various approaches provide guarantees of label noise identification through constraints on m ∗ k ( x ) including “purity" of classes [1, 11, 15, 19, 20].

Original: Other methods provide guarantees on robust learning given constraints on the level of label noise[1, 4, 25], or level of noise in classification [2, 13], or only for binary classification [12], or restrictedto generalized linear models [20].
Revised: Other methods provide guarantees on robust learning given constraints on the level of label noise [1, 4, 26], or level of noise in classification [2, 13], or only for binary classification [12], or restricted to generalized linear models [21].

Original: In contrast, we consider the unconstrained problem to analyze theeffect of label flips for any noise level and feature-dependent noise.
Revised: In contrast, we consider the unconstrained problem to analyze the effect of label flips for any noise level and feature-dependent noise.

Original: Rather than proving guaranteesabout effectiveness of models under constraints; we show that when such constraints do not hold,classification degrades suddenly—not gracefully.
Revised: Rather than proving guarantees about effectiveness of models under constraints; we show that when such constraints do not hold, classification degrades suddenly—not gracefully.

Original: Setting ¯ m ∗ k = 1 in our Corollary 3.2.1 yields the special case given in Corollary 1.1 of [2].
Revised: Setting ¯ m ∗ k = 1 in our Corollary 3.2.1 yields the special case given in Corollary 1.1 of [2].

Original: Ourgeneralization beyond the perfect classifiers (defined in our notation as m ∗ k ( x ) =
Revised: Our generalization beyond the perfect classifiers (defined in our notation as m ∗ k ( x ) =

Original: ¯ m ∗ k = 1 ) assumedin [2] is important because if we apply our result to the problem of identifying noisy labels (aspresented in [2]); then we see that the accuracy of predicting noisy labels is a function of m ∗ k ( x ) —i.e. accurate prediction of label noise depends on the features — even when label noise is independentof the features.
Revised: ¯ m ∗ k = 1 ) assumed in [2] is important because if we apply our result to the problem of identifying noisy labels (as presented in [2]); then we see that the accuracy of predicting noisy labels is a function of m ∗ k ( x ) — i.e. accurate prediction of label noise depends on the features — even when label noise is independent of the features.

Original: Implications The standard assumption of label noise being independent of features can lead tooverly-optimistic findings on the robustness of classification algorithms.
Revised: Implications The standard assumption of label noise being independent of features can lead to overly-optimistic findings on the robustness of classification algorithms.

Original: We can see that if label noiseis applied uniformly randomly in feature space (even if it is non-uniform across classes), then evenas the noisy likelihood begins to decrease, classification accuracy remains high until the likelihoodestimate decreases below the tipping point.[Figure] [Equation] [Figure]
Revised: We can see that if label noise is applied uniformly randomly in feature space (even if it is non-uniform across classes), then even as the noisy likelihood begins to decrease, classification accuracy remains high until the likelihood estimate decreases below the tipping point.

Original: [Figure]

Original:  Synthetic Data Experiments
Revised:  Synthetic Data Experiments

Original: The theory is validated empirically on synthetic data for which P [ Y ∗ | X ] and P [ Y | X ] are known.
Revised: The theory is validated empirically on synthetic data for which P [ Y ∗ | X ] and P [ Y | X ] are known.

Original: Data is 2-d Gaussian with up to 10 classes, with some classes overlapping and others easily-separableas shown in Fig.
Revised: Data is 2-d Gaussian with up to 10 classes, with some classes overlapping and others easily-separable as shown in Fig.

Original: 2a.
Revised: 2a.

Original: We also empirically evaluate 5-d Gaussian data with up to 50 classes.
Revised: We also empirically evaluate 5-d Gaussian data with up to 50 classes.

Original: All codewill be made available as open-source.
Revised: All code will be made available as open-source.

Original: For uniform noise, we sample y ∼ η ki according to Eq.
Revised: For uniform noise, we sample y ∼ η ki according to Eq.

Original: 4 for varying numbers of classes, c ∈{ 2 , 4 , 6 , 8 , 9 , 10 } , and noise ratio, ϵ ∈ { 0 , 0 . 
Revised: 4 for varying numbers of classes, c ∈ { 2 , 4 , 6 , 8 , 9 , 10 } , and noise ratio, ϵ ∈ { 0 , 0 . For class-dependent noise, we sample [Figure] [Equation]

Revised: [Figure]

Revised: Resampling

Original: For class-dependent noise, we sampley ∼ η ki according to Eq. 8 for varying numbers of classes, c ∈ { 2 , 4 , 6 , 8 , 9 , 10 } ; noise ratio,ϵ ∈ { 0 , 0 . 1 , . . .
Revised: [Equation] [Figure] [Equation] [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Figure]

Revised: y ∼ η ki according to Eq. 8 for varying numbers of classes, c ∈ { 2 , 4 , 6 , 8 , 9 , 10 } ; noise ratio, ϵ ∈ { 0 , 0 .

Original:  , 0 . 9 , 1 } ; and spread ∈ { 1 , 2 , 5 } .
Revised: . , 0 . 9 , 1 } ; and spread ∈ { 1 , 2 , 5 } .

Original: For feature-dependent noise, we sample y in fivedifferent ways as summarized in Table 1.
Revised: For feature-dependent noise, we sample y in five different ways as summarized in Table 1.

Original: (1) For uniform-x , the probability of a given sample flippingis constant: η kk ( x ) = 1 − ϵ ; while the choice of label to flip to is proportional to the most-likelyincorrect label: η kj ( x ) ∝ P
Revised: (1) For uniform-x , the probability of a given sample flipping is constant: η kk ( x ) = 1 − ϵ ; while the choice of label to flip to is proportional to the most-likely incorrect label: η kj ( x ) ∝ P

Original: [ Y ∗ =
Revised: [ Y ∗ =

Original: j | x ] .
Revised: j | x ] .

Original: (2) For resampling noise, η kk ( x ) ∝ P [ Y ∗ =
Revised: (2) For resampling noise, η kk ( x ) ∝ P [ Y ∗ =

Original: k | x ] , in otherwords, samples that are least likely in a true class are the most likely to flip.
Revised: k | x ] , in other words, samples that are least likely in a true class are the most likely to flip.

Original: (3) For inverse resampling ,η kk ( x ) ∝ P [ Y ∗ =
Revised: (3) For inverse resampling , η kk ( x ) ∝ P [ Y ∗ =

Original: k | x ] − 1 , i.e., samples with the highest likelihood in the true class are most likelyto flip.
Revised: k | x ] − 1 , i.e., samples with the highest likelihood in the true class are most likely to flip.

Original: (4) For minimum-gap noise, η kk ( x ) ∝
Revised: (4) For minimum-gap noise, η kk ( x ) ∝

Original: P [ Y ∗ =
Revised: [Equation]

Original: k | x ] max j ̸ = k P [ Y ∗ =

Original: j | x ] , so that the samples nearest decisionboundaries are the most likely to flip.
Revised: , so that the samples nearest decision boundaries are the most likely to flip.

Original: (5) For maximum-gap noise, ηkk ( x ) ∝ max j ̸ = k P
Revised: (5) For maximum-gap noise, η

Original: [ Y ∗ =
Revised: [Equation]

Original: j | x ] P [ Y ∗ =

Original: k | x ] , sothat the samples farthest from decision boundaries are the most likely to flip.
Revised: , so that the samples farthest from decision boundaries are the most likely to flip.

Original: There are 100 samples per class in the training set and 100 samples per class in the test set.
Revised: There are 100 samples per class in the training set and 100 samples per class in the test set.

Original: Thefeature matrices X train and X test are the same across all trials.
Revised: The feature matrices X train and X test are the same across all trials.

Original: The corresponding noisy label vectors
Revised: The corresponding noisy label vectors

Original: Y train and Y test are sampled for each combination of noise settings.
Revised: Y train and Y test are sampled for each combination of noise settings.

Original: We also evaluate the accuracyof the held-out clean test labels Y ∗ test .
Revised: We also evaluate the accuracy of the held-out clean test labels Y ∗ test .

Original: The noise level ϵ varies from 0 to 1 in 0 .
Revised: The noise level ϵ varies from 0 to 1 in 0 .

Original: 1 increments.
Revised: 1 increments.

Original: Aneural network with 2 hidden layers is trained; and further details of the architecture is given in the
Revised: A neural network with 2 hidden layers is trained; and further details of the architecture is given in the

Original: Supplement.
Revised: Supplement.

Original: The model is trained 5 times starting from a different random seed; with the mean andstandard deviation of the accuracies reported.
Revised: The model is trained 5 times starting from a different random seed; with the mean and standard deviation of the accuracies reported.

Original: Results Affirming existing results [2, 17, 21], and our theoretical analysis, the similarity of Figure 4to the theoretical expectation of Fig.
Revised: Results Affirming existing results [2, 17, 22], and our theoretical analysis, the similarity of Figureto the theoretical expectation of Fig.

Original: 3 is striking.
Revised: 3 is striking.

Original: The neural network is robust to uniform label noiseup to a point, where that tipping point depends on the number of classes.
Revised: The neural network is robust to uniform label noise up to a point, where that tipping point depends on the number of classes.

Original: Figure 4 gives insight intowhat is happening during training.
Revised: Figure 4 gives insight into what is happening during training.

Original: We see by the very low accuracies on the noisy test labels thatthe model is not able to fit the noise from this uninformative noise distribution.
Revised: We see by the very low accuracies on the noisy test labels that the model is not able to fit the noise from this uninformative noise distribution.

Original: Yet, there is enoughsignal still in the data to learn the patterns that generalize to the clean test set.
Revised: Yet, there is enough signal still in the data to learn the patterns that generalize to the clean test set.

Original: Beyond the tipping
Revised: Beyond the tipping

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: point (the minimizing point in the quadratic function of the noisy accuracy), the neural networkbegins fitting the noise instead of the true signal [Figure] and the clean accuracy drops off steeply.
Revised: point (the minimizing point in the quadratic function of the noisy accuracy), the neural network begins fitting the noise instead of the true signal, and the clean accuracy drops off steeply.

Original: Results on class-dependent noise as shown in Figure 5, also affirm existing results [13, 24, 27] andour theoretical analysis.
Revised: Results on class-dependent noise as shown in Figure 5, also affirm existing results [13, 25, 28] and our theoretical analysis.

Original: We see that the tipping point depends on the spread rather than the totalnumber of classes in the classification task.
Revised: We see that the tipping point depends on the spread rather than the total number of classes in the classification task.

Original: The most harmful class-dependent noise occurs withspread=1, that is when labels from a given class flip to one specific erroneous label, the tipping pointoccurs when the noise level is 0 .5 .
Revised: The most harmful class-dependent noise occurs with spread=1, that is when labels from a given class flip to one specific erroneous label, the tipping point occurs when the noise level is 0 .

Original: With the introduction of feature-dependent label noise as seen in Fig. 6, we see that the neuralnetwork is not always robust even at low levels of label noise.
Revised: With the introduction of feature-dependent label noise as seen in Fig. [Table] we see that the neural network is not always robust even at low levels of label noise.

Original: In particular, the worst-case labelnoise, GapMax, of Claim 3.7 significantly lowers classification accuracy even at ϵ = 0 .
Revised: In particular, the worst-case label noise, GapMax, of Claim 3.7 significantly lowers classification accuracy even at ϵ = 0 .

Original: 1 and 0 . 2 .
Revised: 1 and 0 . 2 .

Original: Interestingly, the classification accuracy from uniform-x noise is statistically indistinguishable fromclass-dependent noise; indicating that the dependence of η kk ( x ) on x is important.
Revised: Interestingly, the classification accuracy from uniform-x noise is statistically indistinguishable from class-dependent noise; indicating that the dependence of η kk ( x ) on x is important.

Original: Fig.
Revised: Fig.

Original: 7 highlights the clean test accuracy of two label-noise mitigation methods: CleanLab [13] and
Revised: 7 highlights the clean test accuracy of two label-noise mitigation methods: CleanLab [13] and

Original: CoTeaching [5].
Revised: CoTeaching [5].

Original: More methods are detailed in the Supplement.
Revised: More methods are detailed in the Supplement.

Original: At ϵ = 0 .
Revised: At ϵ = 0 .

Original: 1 both strategies improveresults compard with the baseline of Fig.
Revised: 1 both strategies improve results compard with the baseline of Fig.

Original: 6c.
Revised: 6c.

Original: But for higher levels of ϵ , CleanLab has little effectwhile CoTeaching performs worse than baseline.
Revised: But for higher levels of ϵ , CleanLab has little effect while CoTeaching performs worse than baseline

Original: [Figure] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original:  Image Benchmark Experiments
Revised:  Image Benchmark Experiments

Original: We compare methods for learning with noisy labels, including CleanLab with the confident joint[13], MixUp [27], symmetric cross entropy (SCE) [3] [Table] and CoTeaching [5]; using the public-licenseopen-source code provided by the authors.
Revised: We compare methods for learning with noisy labels, including CleanLab with the confident joint [13], MixUp [28], symmetric cross entropy (SCE) [3], and CoTeaching [5]; using the public-license open-source code provided by the authors.

Original: For all methods the base architecture is ResNet-32 [6]with more details including computational costs, and extended empirical results, in the Supplement.
Revised: For all methods the base architecture is ResNet-32 [6] with more details including computational costs, and extended empirical results, in the Supplement.

Revised: We also compare the baseline performance which is the ResNet-32 architecture without any label noise mitigation strategy.

Original: We use the classification benchmarks CIFAR-10 and CIFAR-100 of 32x32-pixel color images in 10or 100 classes, with 60,000 images per dataset [9].
Revised: We use the classification benchmarks CIFAR-10 and CIFAR-100 of 32x32-pixel color images inor 100 classes, with 60,000 images per dataset [9].

Original: Class-dependent noisy labels Y | Y ∗ are generatedaccording to Definition 3.4, assuming that the labels given in the dataset are the true labels Y ∗ .
Revised: Class-dependent noisy labels Y | Y ∗ are generated according to Definition 3.4, assuming that the labels given in the dataset are the true labels Y ∗ .

Original: Togenerate label noise Y | X that is dependent on features but independent from the learned classificationfunction itself, we usenon-learned similarity-preserving image hashes [14, 26].
Revised: Generating feature-dependent label noise To generate label noise Y | X that is dependent on features but independent from the learned classification function itself, we use similarity-preserving image hashes [14, 27].

Original: For each image inthe dataset, we compute the similarity-preserving hash, h ( x ) ; which puts visually-similar (rather thansemantically-similar) images near each other.
Revised: For each image in the dataset, we compute the similarity-preserving hash, h ( x ) ; which puts visually-similar (rather than semantically-similar) images near each other.

Original: Then for each class, we calculate the average hash, ¯ h k ,which represents the “center" of the class.
Revised: Then for each class, we calculate the average hash, ¯ h k , which represents the "center" of the class.

Original: For class-conditional noise, we generate label flips betweenclasses that are s -nearest neighbors based on the hash centers and spread s .
Revised: For class-conditional noise, we generate label flips between classes that are s -nearest neighbors based on the hash centers and spread s .

Original: For feature-dependentnoise, we generate label flips using the GapMax function of Table 1 based on the distance from aparticular sample’s hash to the s -nearest neighbor hash centers, i.e. P [ Y ∗ =
Revised: For feature-dependent noise, we generate label flips using the GapMax function of Table 1 based on the distance from a particular sample’s hash to the s -nearest neighbor hash centers, i.e. P [ Y ∗ =

Original: k | x ] ∝ 1 /δ k
Revised: k | x ] ∝ 1 /δ k

Original: ( x ) where 
Revised: ( x ) where δ k ( x ) =

Original: [Equation]
Revised: ∥ h ( x ) − ¯ h k ∥ 2 . We investigate two different similarity-preserving hashes: 1) VisHash [14] which was designed to detect near-copy images ignoring changes in color and shading, so it will put images with similar shapes near each other; and 2) h ( x ) is a ResNet-50 [6] pre-trained on ImageNet-1000 [18] using the activations of the layer before the fully connected layers as the hash vector. The results for the VisHash-derived label flips are provided here with the ResNet-50 driven label flips provided in the Supplement, noting that the general trends are similar for these two types of feature-driven label flips.

Original: Results
Revised: Results

Original: The results for class-dependent label flips on CIFAR-10 in Table 2 show general patternsof what our theory predicts: accuracy drops severely between ϵ = 0 .
Revised: The results for class-dependent label flips on CIFAR-10 in Table 2 show general patterns of what our theory predicts: accuracy drops severely between ϵ = 0 .

Original: 4 and ϵ = 0 .
Revised: 4 and ϵ = 0 .

Original: 6 when s = 1 .
Revised: 6 when s = 1 .

Original: For s = 2 , the drop is less severe until ϵ = 0 . 8 .
Revised: For s = 2 , the drop is less severe until ϵ = 0 . 8 .

Original: Also supporting our theory, the results for featuredependent label flips show more gradual decline in accuracy so that on average the accuracy is lowerfor feature-dependent label flips than for class-dependent label flips when ϵ ∈ { 0 . 2 , 0 . 4 } , as wellas for higher values of ϵ when spread, s ∈ { 2 , 5 } .
Revised: Also supporting our theory, the results for featuredependent label flips show more gradual decline in accuracy so that on average the accuracy is lower for feature-dependent label flips than for class-dependent label flips when ϵ ∈ { 0 . 2 , 0 . 4 } , as well as for higher values of ϵ when spread, s ∈ { 2 , 5 } .

Original: Table 3 shows that similar trends hold for adataset with a much high number of classes.
Revised: Table 3 shows that similar trends hold for a dataset with a much high number of classes.

Original: It is interesting to note
Revised: It is interesting to note

Original: that when a mitigation strategyis performing poorly (e.g. SCE on CIFAR10 at ϵ = 0 . 2 or CleanLab on CIFAR100 at ϵ = 0 . 2 )compared with the other methods; then it does not seem to matter whether the noise is featuredependent or class-dependent.
Revised: that when a mitigation strategy is performing poorly (e.g. SCE on CIFAR10 at ϵ = 0 . 2 or CleanLab on CIFAR100 at ϵ = 0 . 2 ) compared with the other methods; then it does not seem to matter whether the noise is featuredependent or class-dependent.

Original: Therefore, we see that in general the robustness of classifiers with successful label noise mitigation strategies behave differently whether that noise is feature-dependent.
Revised: Therefore, we see that in general the robustness of classifiers with successful label noise mitigation strategies behave differently whether that noise is feature-dependent.

Original: Feature-dependent noise reduces classification accuracy at lower levels of noise.
Revised: Feature-dependent noise reduces classification accuracy at lower levels of noise.

Original: CleanLab and MixUp show the strongest results, but there is no clear winner among mitigationstrategies for label noise; as all methods display weaknesses under various noise levels or noise types.
Revised: CleanLab and MixUp show the strongest results, but there is no clear winner among mitigation strategies for label noise; as all methods display weaknesses under various noise levels or noise types.

Original: One promising result is that the mitigation strategies tend to narrow the gap in performance betweenhandling class noise and feature noise; which is consistent with our findings on the synthetic data in
Revised: One promising result is that the mitigation strategies tend to narrow the gap in performance between handling class noise and feature noise; which is consistent with our findings on the synthetic data in

Original: Figure 7 .
Revised: Figure 7 .

Original: The strong performance of MixUp on CIFAR-100 is probably due to it being particularlywell-suited to the hierarchical nature of the classes [22, 27].
Revised: The strong performance of MixUp on CIFAR-100 is probably due to it being particularly well-suited to the hierarchical nature of the classes [23, 28].

Original: Whereas, CleanLab struggles with
Revised: Whereas, CleanLab struggles with

Original: CIFAR-100 probably because the low baseline accuracy even at low levels of noise makes it difficultto confidently predict clean versus noisy samples.
Revised: CIFAR-100 probably because the low baseline accuracy even at low levels of noise makes it difficult to confidently predict clean versus noisy samples.

Original:  Conclusions
Revised:  Conclusions

Original: Limitations Our theoretical results are limited to the Bayes optimal classifier.
Revised: Limitations Our theoretical results are limited to the Bayes optimal classifier.

Original: As with most theory,the degree to which these results apply in practice must be explored empirically.
Revised: As with most theory, the degree to which these results apply in practice must be explored empirically.

Original: Future Directions
Revised: Future Directions

Original: The theory could be extended to explain why learning from massive, noisydatasets can be so successful, especially when the noisy dataset is used as a pre-training task: majorpatterns in data can be learned despite label noise as long as there is some signal within the noise.
Revised: The theory could be extended to explain why learning from massive, noisy datasets can be so successful, especially when the noisy dataset is used as a pre-training task: major patterns in data can be learned despite label noise as long as there is some signal within the noise.

Original: Our theory could also be used to guide the curation of datasets as they are being collected.
Revised: Our theory could also be used to guide the curation of datasets as they are being collected.

Original: Not alllabels need to be clean, but enough labels in every class across the relevant feature space do needto be clean.
Revised: Not all labels need to be clean, but enough labels in every class across the relevant feature space do need to be clean.

Original: Data-specific approaches may look for regions of feature space where labels tend to benoisy and focus more effort on cleaning labels for those regions.
Revised: Data-specific approaches may look for regions of feature space where labels tend to be noisy and focus more effort on cleaning labels for those regions.

Original: Conclusion In both the uniform and class-dependent label flip scenarios, classification is robust tolabel noise up to a tipping point; and beyond that tipping point, classification fails catastrophically.
Revised: Conclusion In both the uniform and class-dependent label flip scenarios, classification is robust to label noise up to a tipping point; and beyond that tipping point, classification fails catastrophically.

Original: These theoretical findings are helpful in designing evaluation approaches for label noise detectionand mitigation.
Revised: These theoretical findings are helpful in designing evaluation approaches for label noise detection and mitigation.

Original: Considering that label noise mitigation strategies can require training 2 models (suchas for CoTeaching) or even 6 models (such as for CleanLab); we can see why empirical studies arecomputationally limited to investigating just a few variations on noise level and shape.
Revised: Considering that label noise mitigation strategies can require training 2 models (such as for CoTeaching) or even 6 models (such as for CleanLab); we can see why empirical studies are computationally limited to investigating just a few variations on noise level and shape.

Original: We nowknow that levels of noise near where the tipping-point occurs will be most informative; that thereis little reason to evaluate various levels of spread of class-dependent noise, and that incorporatingfeature-dependent noise is most informative for practical applications.
Revised: We now know that levels of noise near where the tipping-point occurs will be most informative; that there is little reason to evaluate various levels of spread of class-dependent noise, and that incorporating feature-dependent noise is most informative for practical applications.

Original: Classification accuracy is significantly lower at small levels of noise when the noise distributiondepends on features and is targeted at samples with the highest gap between the likelihood of thetrue label and the next-most-likely label.
Revised: Classification accuracy is significantly lower at small levels of noise when the noise distribution depends on features and is targeted at samples with the highest gap between the likelihood of the true label and the next-most-likely label.

Original: While various label noise mitigation strategies provideguarantees about conditions under which they work; we see that the mitigation strategies on real datagenerally fail for similar scale and shape of label noise distributions.
Revised: While various label noise mitigation strategies provide guarantees about conditions under which they work; we see that the mitigation strategies on real data generally fail for similar scale and shape of label noise distributions.

Original: Realistically, we expect thatlabel noise does depend on the features; and this paper demonstrates theoretically and empirically thatthe old assumption of label noise—being independent of features given the class—leads to an overlyoptimistic expectation of classifier robustness in the face of label noise.
Revised: Realistically, we expect that label noise does depend on the features; and this paper demonstrates theoretically and empirically that the old assumption of label noise—being independent of features given the class—leads to an overly optimistic expectation of classifier robustness in the face of label noise.

Original: While it is challenging tomodel feature-dependent label noise; it is clearly necessary as we see significantly lower classificationaccuracy when label noise is feature-dependent. 
Revised: While it is challenging to model feature-dependent label noise; it is clearly necessary as we see significantly lower classification accuracy when label noise is feature-dependent. Acknowledgments and Disclosure of Funding

Revised: Research presented in this paper was supported by the Laboratory Directed Research and Development program of Los Alamos National Laboratory under project number 20210043DR.