Original: I S THE PERFORMANCE OF MY DEEP NETWORK TOO GOOD TO BE TRUE ? A DIRECT APPROACH TO ESTIMATING THE B AYES ERROR IN BINARY CLASSIFICATION
Revised: I S THE PERFORMANCE OF MY DEEP NETWORK TOO GOOD TO BE TRUE ? A DIRECT APPROACH TO ESTIMATING THE B AYES ERROR IN BINARY CLASSIFICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: There is a fundamental limitation in the prediction performance that a machine learning model can achieve due to the inevitable uncertainty of the prediction target.
Revised: There is a fundamental limitation in the prediction performance that a machine learning model can achieve due to the inevitable uncertainty of the prediction target.

Original: In classification problems, this can be characterized by the Bayes error, which is the best achievable error with any classifier.
Revised: In classification problems, this can be characterized by the Bayes error, which is the best achievable error with any classifier.

Original: The Bayes error can be used as a criterion to evaluate classifiers with state-of-the-art performance and can be used to detect test set overfitting.
Revised: The Bayes error can be used as a criterion to evaluate classifiers with state-of-the-art performance and can be used to detect test set overfitting.

Original: We propose a simple and direct Bayes error estimator where we just take the mean of the labels that show uncertainty of the classes.
Revised: We propose a simple and direct Bayes error estimator, where we just take the mean of the labels that show uncertainty of the classes.

Original: Our flexible approach enables us to perform Bayes error estimation even for weakly supervised data.
Revised: Our flexible approach enables us to perform Bayes error estimation even for weakly supervised data.

Original: In contrast to others, our method is model-free and even instance-free.
Revised: In contrast to others, our method is model-free and even instance-free.

Original: Moreover, it has no hyperparameters and gives a more accurate estimate of the Bayes error than several baselines empirically.
Revised: Moreover, it has no hyperparameters and gives a more accurate estimate of the Bayes error than several baselines empirically.

Original: Experiments using our method suggest that a recently proposed classifier, the Vision Transformer, may have reached (or is about to reach) the Bayes error for the CIFAR-10H dataset
Revised: Experiments using our method suggest that a recently proposed classifier, the Vision Transformer, may have reached (or is about to reach) the Bayes error for the CIFAR-10H dataset.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Comparing the prediction performance of a deep neural network with the state-of-the-art (SOTA) performance is a common approach to validating advances brought by a new proposal in machine learning research.
Revised: Comparing the prediction performance of a deep neural network with the state-of-the-art (SOTA) performance is a common approach to validating advances brought by a new proposal in machine learning research.

Original: By definition, SOTA performance such as error or 1 − AUC monotonically decreases over time, but there is a fundamental limitation in the prediction performance that a machine learning model can achieve.
Revised: By definition, SOTA performance such as error or 1 − AUC monotonically decreases over time, but there is a fundamental limitation in the prediction performance that a machine learning model can achieve.

Original: Hence, it is important to figure out how close the current SOTA performance is to the underlying best performance achievable (Theisen et al., 2021).
Revised: Hence, it is important to figure out how close the current SOTA performance is to the underlying best performance achievable (Theisen et al., 2021).

Original: For example, Henighan et al.
Revised: For example, Henighan et al.

Original: Henighan et al.
Revised: Henighan et al.

Original: (2020) studied the scaling law (Kaplan et al., 2020) of Transformers (Vaswani et al., 2017) by distinguishing the reducible and irreducible errors.
Revised: (2020) studied the scaling law (Kaplan et al., 2020) of Transformers (Vaswani et al., 2017) by distinguishing the reducible and irreducible errors.

Original: In classification problems, one way to characterize the irreducible part is the Bayes error (Cover &
Revised: In classification problems, one way to characterize the irreducible part is the Bayes error (Cover &

Original: Hart, 1967; Fukunaga, 1990; Mohri et al., 2018), which is the best achievable expected error with any measurable function.
Revised: Hart, 1967; Fukunaga, 1990; Mohri et al., 2018), which is the best achievable expected error with any measurable function.

Original: The Bayes error will become zero in the special case when class distributions are completely separable, but in practice, we usually deal with complex distributions with some class overlap.
Revised: The Bayes error will become zero in the special case when class distributions are completely separable, but in practice, we usually deal with complex distributions with some class overlap.

Original: Natural images tend to have a lower Bayes error (e.g., 0.21% test error on MNIST (LeCun et al., 1998) in Wan et al.
Revised: Natural images tend to have a lower Bayes error (e.g., 0.21% test error on MNIST (LeCun et al., 1998) in Wan et al.

Original: (Wan et al., 2013)), while medical images tend to have a higher Bayes error, since even medical technologists can disagree on the ground truth label (Sasada et al., 2018).
Revised: (Wan et al., 2013)), while medical images tend to have a higher Bayes error, since even medical technologists can disagree on the ground truth label (Sasada et al., 2018).

Original: If the current model’s performance has already achieved the Bayes error, it is meaningless to aim for further error improvement.
Revised: If the current model’s performance has already achieved the Bayes error, it is meaningless to aim for further error improvement.

Original: We may even find out that the model’s error has exceeded the Bayes error.
Revised: We may even find out that the model’s error has exceeded the Bayes error.

Original: This may imply test set overfitting (Recht et al., 2018) is taking place.
Revised: This may imply test set overfitting (Recht et al., 2018) is taking place.

Original: Knowing the Bayes error will be helpful to avoid such common pitfalls.
Revised: Knowing the Bayes error will be helpful to avoid such common pitfalls.

Original: Estimating the Bayes error has been a topic of interest in the research community (Fukunaga &
Revised: Estimating the Bayes error has been a topic of interest in the research community (Fukunaga &

Original: Hostetler, 1975; Fukunaga, 1990; Devijver, 1985; Berisha et al., 2016).
Revised: Hostetler, 1975; Fukunaga, 1990; Devijver, 1985; Berisha et al., 2016; Michelucci et al., 2021).

Original: To the best of our knowledge, all previous papers have proposed ways to estimate the Bayes error from a dataset consisting of pairs of instances and their hard labels.
Revised: To the best of our knowledge, all previous papers have proposed ways to estimate the Bayes error from a dataset consisting of pairs of instances and their hard labels.

Original: When instances and hard labels are available, one can also train a supervised classifier, which is known to approach the Bayes classifier (that achieves the Bayes error) with sufficient training data provided that the model is correctly specified.
Revised: When instances and hard labels are available, one can also train a supervised classifier, which is known to approach the Bayes classifier (that achieves the Bayes error) with sufficient training data provided that the model is correctly specified.

Original: This is an interesting research problem from the point of view of Vapnik’s principle 1 (Vapnik, 2000) since we can derive the Bayes error from the Bayes classifier (and the underlying distribution) while we cannot recover the Bayes classifier from the knowledge of the Bayes error, which is just a scalar.
Revised: This is an interesting research problem from the point of view of Vapnik’s principle 1 (Vapnik, 2000) since we can derive the Bayes error from the Bayes classifier (and the underlying distribution) while we cannot recover the Bayes classifier from the knowledge of the Bayes error, which is just a scalar.

Original: How can we take full advantage of this property?
Revised: How can we take full advantage of this property?

Original: While the Bayes error is usually defined as the best achievable expected error with any measurable function, it is known to be equivalent to the expectation of the minimum of class-posteriors with respect to classes for binary classification.
Revised: While the Bayes error is usually defined as the best achievable expected error with any measurable function, it is known to be equivalent to the expectation of the minimum of class-posteriors with respect to classes for binary classification.

Original: Inspired by
Revised: Inspired by

Original: Vapnik’s principle, our main idea is to skip the intermediate step of learning a function model, and we directly approximate the minimum of the class-posteriors by using soft labels (corresponding to the class probability) or uncertainty labels (corresponding to the class uncertainty 2 ).
Revised: Vapnik’s principle, our main idea is to skip the intermediate step of learning a function model, and we directly approximate the minimum of the class-posteriors by using soft labels (corresponding to the class probability) or uncertainty labels (corresponding to the class uncertainty).

Original: Our proposed method has two benefits.
Revised: Our proposed method has two benefits.

Original: Firstly, our method is model-free .
Revised: Firstly, our method is model-free .

Original: Since we do not learn a model, we can escape the curse of dimensionality, while dealing with high-dimensional instance would cause issues such as overfitting if we were to train a model.
Revised: Since we do not learn a model, we can escape the curse of dimensionality, while dealing with high-dimensional instance would cause issues such as overfitting if we were to train a model.

Original: High dimensionality may cause performance deterioration for other Bayes error estimation methods due to divergence estimation.
Revised: High dimensionality may cause performance deterioration for other Bayes error estimation methods due to divergence estimation.

Original: We experimentally show how our method can more accurately estimate the Bayes error than baselines that utilize instances and soft labels.
Revised: We experimentally show how our method can more accurately estimate the Bayes error than baselines that utilize instances and soft labels.

Original: Our model-free method is also extremely fast since we do not have any hyperparameters to tune nor a function model to train.
Revised: Our model-free method is also extremely fast since we do not have any hyperparameters to tune nor a function model to train.

Original: The second benefit is a more practical one: our method is completely instance-free .
Revised: The second benefit is a more practical one: our method is completely instance-free .

Original: Suppose our final goal is to estimate the Bayes error instead of training a classifier.
Revised: Suppose our final goal is to estimate the Bayes error instead of training a classifier.

Original: In that case, we do not need to collect instance-label pairs, and it may be less costly to collect soft/uncertainty labels without instances.
Revised: In that case, we do not need to collect instance-label pairs, and it may be less costly to collect soft/uncertainty labels without instances.

Original: Dealing with instances can cause privacy issues, and it can be expensive due to data storage costs especially when they are high-dimensional or can come in large quantities.
Revised: Dealing with instances can cause privacy issues, and it can be expensive due to data storage costs especially when they are high-dimensional or can come in large quantities.

Original: It may lead to security costs to protect instances from a data breach.
Revised: It may lead to security costs to protect instances from a data breach.

Original: As an example of an instance-free scenario, we can consider doctors who are diagnosing patients by inspecting symptoms and asking questions, without explicitly collecting or storing the patients’ data in the database.
Revised: As an example of an instance-free scenario, we can consider doctors who are diagnosing patients by inspecting symptoms and asking questions, without explicitly collecting or storing the patients’ data in the database.

Original: In this scenario, the hospital will only have the decisions and confidence of doctors, which can be used as soft labels.
Revised: In this scenario, the hospital will only have the decisions and confidence of doctors, which can be used as soft labels.

Original: The contributions of the paper is as follows.
Revised: The contributions of the paper is as follows.

Original: We first propose a direct way to estimate the Bayes error from soft (or uncertainty) labels without a model nor instances.
Revised: We first propose a direct way to estimate the Bayes error from soft (or uncertainty) labels without a model nor instances.

Original: We show that our estimator is unbiased and consistent.
Revised: We show that our estimator is unbiased and consistent.

Original: In practice, collecting soft/uncertainty labels can be difficult since the labelling process can become noisy.
Revised: In practice, collecting soft/uncertainty labels can be difficult since the labelling process can become noisy.

Original: We propose a modified estimator that is still unbiased and consistent even when the soft labels are contaminated with zero-mean noise.
Revised: We propose a modified estimator that is still unbiased and consistent even when the soft labels are contaminated with zero-mean noise.

Original: We also show that our approach can be applied to other classification problems, such as weakly supervised learning.
Revised: We also show that our approach can be applied to other classification problems, such as weakly supervised learning.

Original: Finally, we show the proposed methods’ behavior through various experiments.
Revised: Finally, we show the proposed methods’ behavior through various experiments.

Original: Our results suggest that Vision Transformer (Dosovitskiy et al., 2021) has reached or is about to reach the Bayes error for the CIFAR-10H dataset (Peterson et al., 2019).
Revised: Our results suggest that Vision Transformer (Dosovitskiy et al., 2021) has reached or is about to reach the Bayes error for the CIFAR-10H dataset (Peterson et al., 2019).

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Before we discuss our setup, we will review ordinary binary classification from positive and negative data and binary classification from positive-confidence data in this section.
Revised: Before we discuss our setup, we will review ordinary binary classification from positive and negative data and binary classification from positive-confidence data in this section.

Original: Learning from positive-negative data Suppose a pair of d -dimensional instance x ∈ R d and its class label y ∈ { +1 , − 1 } follows an unknown probability distribution with probability density function p ( x , y ) .
Revised: Learning from positive-negative data Suppose a pair of d -dimensional instance x ∈ R d and its class label y ∈ { +1 , − 1 } follows an unknown probability distribution with probability density function p ( x , y ) .

Original: The goal of binary classification is to train a binary classifier g : R d → R so that the following classification risk R ( g ) will be minimized: R ( g ) =
Revised: The goal of binary classification is to train a binary classifier g : R d → R so that the following classification risk R ( g ) will be minimized: R ( g ) =

Original: E p ( x ,y )
Revised: E p ( x ,y )

Original: [ ℓ ( yg ( x ))] , where ℓ is a margin-based surrogate loss function such as the logistic loss ℓ L ( z )
Revised: [ ℓ ( yg ( x ))] , where ℓ is a margin-based surrogate loss function such as the logistic loss ℓ L ( z )

Original: =
Revised: =

Original: log(1 + exp( − z )) or the squared loss ℓ S ( z )
Revised: log(1 + exp( − z )) or the squared loss ℓ S ( z )

Original: =
Revised: =

Original: ( z − 1) 2 .
Revised: ( z − 1) 2 .

Original: The classification error is R 01 ( g ) =
Revised: The classification error is R 01 ( g ) =

Original: E p ( x ,y ) [ ℓ 01 ( yg ( x ))] , where we use the zero-one loss ℓ 01 , which takes 0 when y · g ( x ) > 0 and 1 otherwise.
Revised: E p ( x ,y ) [ ℓ 01 ( yg ( x ))] , where we use the zero-one loss ℓ 01 , which takes 0 when y · g ( x ) > 0 and 1 otherwise.

Original: In empirical risk minimization, the following estimator of R ( g ) is often used:
Revised: In empirical risk minimization, the following estimator of R ( g ) is often used:

Original: [Equation]
Revised: [Equation]

Original: (1) where { ( x i , y i ) } ni =1 i .
Revised: where { ( x i , y i ) } ni =1 i .

Original: i .
Revised: i .

Original: d . ∼ p ( x , y ) .
Revised: d . ∼ p ( x , y ) .

Original: In this paper, our focus is not on the classifier itself but on the value of R 01 ( g ) when a good classifier is learned.
Revised: In this paper, our focus is not on the classifier itself but on the value of R 01 ( g ) when a good classifier is learned.

Original: This can be nicely captured by the notion of the Bayes error, defined as the following.
Revised: This can be nicely captured by the notion of the Bayes error, defined as the following.

Original: Definition 2.1.
Revised: Definition 2.1.

Original: (Mohri et al., 2018)
Revised: (Mohri et al., 2018)

Original: The Bayes error is the infimum of the classification risk with respect to all measurable functions: β = inf g : R d → R , measurable E p ( x ,y )
Revised: The Bayes error is the infimum of the classification risk with respect to all measurable functions: β = inf g : R d → R , measurable E p ( x ,y )

Original: [ ℓ 01 ( yg ( x ))] .
Revised: [ ℓ 01 ( yg ( x ))] .

Original: Learning from positive-confidence data In positive-confidence classification (Ishida et al., 2018;
Revised: Learning from positive-confidence data In positive-confidence classification (Ishida et al., 2018;

Original: Shinoda et al., 2020), we only have access to positive data equipped with confidence { ( x i , r i ) } n + i =1 , where x i is a positive instance drawn independently from p ( x | y = +1) and r i is the positive confidence given by r i
Revised: Shinoda et al., 2020), we only have access to positive data equipped with confidence { ( x i , r i ) } n + i =1 , where x i is a positive instance drawn independently from p ( x | y = +1) and r i is the positive confidence given by r i

Original: = p ( y = +

Revised: = p ( y = +

Original: | x i ) .
Revised: | x i ) .

Original: Since we only have positive instances, we cannot directly minimizethe empirical risk in Eq.
Revised: Since we only have positive instances, we cannot directly minimize Eq.

Original: However, Ishida et al.
Revised: However, Ishida et al.

Original: (2018) showed that the following theorem holds.
Revised: (2018) showed that the following theorem holds.

Original: Theorem 2.2.
Revised: Theorem 2.2.

Original: (Ishida et al., 2018)
Revised: (Ishida et al., 2018)

Original: The classification risk can be expressed as
Revised: The classification risk can be expressed as

Original: [Equation]
Revised: [Equation]

Original: (2) if we have r ( x ) = p ( y = +

Revised: if we have r ( x ) = p ( y = +

Original: | x ) ̸ = 0 for all x sampled from p ( x ) .
Revised: | x ) ̸ = 0 for all x sampled from p ( x ) .

Original: E
Revised: E

Original: + is the expectation over the positive distribution p ( x | y
Revised: + is the expectation over the positive distribution p ( x | y

Original: = +1) and π + = p
Revised: = +1) and π + = p

Original: ( y = +1) .
Revised: ( y = +1) .

Original: Since the expectation is over p ( x | y = +1) , we do not have to prepare negative data.
Revised: Since the expectation is over p ( x | y = +1) , we do not have to prepare negative data.

Original: Although π + is usually unknown, we do not need this knowledge since π + can be regarded as a constant when minimizing Eq. 
Revised: Although π + is usually unknown, we do not need this knowledge since π + can be regarded as a constant when minimizing Eq. 2.

Original: 2 to learn a classifier.
Revised: Hence, we can see how learning a classifier is possible in this weakly-supervised setup, but is estimating the Bayes error also possible? We will study this problem further in Sec. 4.

Original:  E STIMATION WITH ORDINARY DATA
Revised:  E STIMATION WITH ORDINARY DATA

Original: In this section, we propose our direct approach to Bayes error estimation.
Revised: In this section, we propose our direct approach to Bayes error estimation.

Original: Clean soft labels First of all, it is known that the Bayes error can be rewritten as follows (Mohri et al., 2018):
Revised: Clean soft labels First of all, it is known that the Bayes error can be rewritten as follows (Mohri et al., 2018):

Original: [Equation]
Revised: [Equation]

Original: This means that we can express the Bayes error β without explicitly involving the classifier g .
Revised: This means that we can express the Bayes error β without explicitly involving the classifier g .

Original: If we assume that we have soft labels, which are { c i } ni =1 , where c
Revised: If we assume that we have soft labels, which are { c i } ni =1 , where c

Original: i = p ( y = +

Revised: i = p ( y = +

Original: | x i ) and { x i } ni =

Revised: | x i ) and { x i } ni =

Original: i .
Revised: i .

Original: i .
Revised: i .

Original: d . ∼ p ( x ) , we can construct an extremely straightforward estimator for Bayes error estimation:
Revised: d . ∼ p ( x ) , we can construct an extremely straightforward estimator for Bayes error estimation:

Original: [Equation]
Revised: [Equation]

Original: In fact, Eq. 4 is just the empirical version of Eq. 3.
Revised: In fact, Eq. 4 is just the empirical version of Eq. 3.

Original: However, the significance lies in the observation that this can be performed in a model-free and instance-free fashion, since taking the averages of the smaller one out of c i and 1 − c i in Eq.
Revised: However, the significance lies in the observation that this can be performed in a model-free and instance-free fashion, since taking the averages of the smaller one out of c i & 1 − c i in Eq.

Original: 4 does not require model training and does not involve x i at all.
Revised: 4 does not require model training and does not involve x i at all.

Original: Note that we can further weaken the supervision by sampling uncertainty labels , i.e., c ′ i =
Revised: Note that we can further weaken the supervision by sampling uncertainty labels , i.e., c ′ i =

Original: min
Revised: min

Original: y ∈{± 1 } p ( y | x i ) .
Revised: y ∈{± 1 } p ( y | x i ) .

Original: Practically, this would mean the label provider does not need to reveal the soft label.
Revised: Practically, this would mean the label provider does not need to reveal the soft label.

Original: Although the two are similar, we can see how uncertainty labels are weaker by the fact that we cannot recover p ( y = +

Revised: Although the two are similar, we can see how uncertainty labels are weaker by the fact that we cannot recover p ( y = +

Original: | x ) from min y ∈{± 1 } p ( y | x )  .
Revised: | x ) from min y ∈{± 1 } p ( y | x ) 3 .

Original: Furthermore, we can recover π + = p ( y = +1) by the expectation of soft labels E p ( x ) [ p ( y = +1 | x )]
Revised: Furthermore, we can recover π + = p ( y = +1) by the expectation of soft labels E p ( x ) [ p ( y = +1 | x )]

Original: = π + , but π + cannot be recovered by averaging uncertainty labels.
Revised: = π + , but π + cannot be recovered by averaging uncertainty labels.

Original: We can also enjoy theoretical properties of our estimator, which we show in the following propositions.
Revised: We can also enjoy theoretical properties of our estimator, which we show in the following propositions.

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: The estimator ˆ β is an unbiased estimator of the Bayes error.
Revised: The estimator ˆ β is an unbiased estimator of the Bayes error.

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: For any δ > 0 , with probability at least 1 − δ , we have, | ˆ β − β | ≤ (cid:113) 18 n log 2 δ .
Revised: For any δ > 0 , with probability at least 1 − δ , we have, | ˆ β − β | ≤ (cid:113) 18 n log 2 δ .

Original: The proofs are shown in App.
Revised: The proofs are shown in App.

Original: B.1 and App.
Revised: B.1 and App.

Original: B.2, respectively.
Revised: B.2, respectively.

Original: This implies that ˆ β is a consistent estimator to the Bayes error, i.e., ∀ ϵ > 0 , lim n →∞ P ( | ˆ β − β | ≥ ϵ ) = 0 .
Revised: This implies that ˆ β is a consistent estimator to the Bayes error, i.e., ∀ ϵ > 0 , lim n →∞ P ( | ˆ β − β | ≥ ϵ ) = 0 .

Original: The rate of convergence is | ˆ β − β | =
Revised: The rate of convergence is | ˆ β − β | =

Original: O P (1 / √ n ) , where O P is the stochastic big-O notation, cf.
Revised: O P (1 / √ n ) , where O P is the stochastic big-O notation, cf.

Original: App.
Revised: App.

Original: C.
Revised: C.

Original: This rate is the same as classifier training and is the optimal parametric rate (Mendelson, 2008) although our method can be regarded as non-parametric.
Revised: This rate is the same as classifier training and is the optimal parametric rate (Mendelson, 2008) although our method can be regarded as non-parametric.

Original: We are also free from the curse of dimensionality since the instance dimension d does not appear in the bound.
Revised: We are also free from the curse of dimensionality since the instance dimension d does not appear in the bound.

Original: Noisy soft labels So far, our discussions were based on an assumption that we can collect soft (or uncertainty) labels that is precisely the class-posterior c i (or c ′ i ).
Revised: Noisy soft labels So far, our discussions were based on an assumption that we can collect soft (or uncertainty) labels that is precisely the class-posterior c i (or c ′ i ).

Original: This may not be satisfied in practice, and we will present a way to weaken this assumption.
Revised: This may not be satisfied in practice, and we will present a way to weaken this assumption.

Original: We consider a situation where noise ξ i is added to the class-posterior values.
Revised: We consider a situation where noise ξ i is added to the class-posterior values.

Original: We observe noisy soft labels { u i } ni =1 where u i
Revised: We observe noisy soft labels { u i } ni =1 where u i

Original: =
Revised: =

Original: c i + ξ i .
Revised: c i + ξ i .

Original: Since we ask labellers to give us a probability, a constraint is 0 ≤ u i ≤ 1 , and we assume E [ ξ i | c i ] = 0 .
Revised: Since we ask labellers to give us a probability, a constraint is 0 ≤ u i ≤ 1 , and we assume E [ ξ i | c i ] = 0 .

Original: This implies that ξ i becomes smaller as c i approaches 1 or 0 .
Revised: This implies that ξ i becomes smaller as c i approaches 1 or 0 .

Original: If we naively use these noisy versions, our estimator becomes
Revised: If we naively use these noisy versions, our estimator becomes

Original: [Equation]
Revised: [Equation]

Original: However, we can show the following negative result.
Revised: However, we can show the following negative result.

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: The estimator ˜ β is generally a biased estimator of the Bayes error:
Revised: The estimator ˜ β is generally a biased estimator of the Bayes error:

Original: E
Revised: E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This is basically due to the noise inside the nonlinear min operation.
Revised: This is basically due to the noise inside the nonlinear min operation.

Original: We show the proof in App.
Revised: We show the proof in App.

Original: B.3.
Revised: B.3.

Original: This issue can be fixed if we have an additional binary signal that shows if it is likely to be more positive or negative.
Revised: This issue can be fixed if we have an additional binary signal that shows if it is likely to be more positive or negative.

Original: Formally, if we have access to sign labels, i.e., s i = sign [ p ( y = +

Revised: Formally, if we have access to sign labels, i.e., s i = sign [ p ( y = +

Original: | x i ) − 0 .
Revised: | x i ) − 0 .

Original: 5] ∀ i ∈ { 1 , 2 , . . . , n } , we can divide { u i } ni =1 into { u + i } n + s i
Revised: 5] ∀ i ∈ { 1 , 2 , . . . , n } , we can divide { u i } ni =1 into { u + i } n + s i

Original: =1 and { u − i } n − s i
Revised: =1 and { u − i } n − s i

Original: = depending on the sign label, with n = n + s + n − s .
Revised: = depending on the sign label, with n = n + s + n − s .

Original: Then we can construct the following estimator:
Revised: Then we can construct the following estimator:

Original: [Equation]
Revised: [Equation]

Original: With this modified estimator ˆ β Noise , we can show the following propositions.
Revised: With this modified estimator ˆ β Noise , we can show the following propositions.

Original: Proposition 3.4.
Revised: Proposition 3.4.

Original: The estimator ˆ β Noise is an unbiased estimator of the Bayes error.
Revised: The estimator ˆ β Noise is an unbiased estimator of the Bayes error.

Original: Proof is shown in App.
Revised: Proof is shown in App.

Original: B.5.
Revised: B.5.

Original: The proof itself is simple, but the main idea lies in where we modified the estimator from Eq.
Revised: The proof itself is simple, but the main idea lies in where we modified the estimator from Eq.

Original: 4 to avoid dealing with the noise inside the min operator, by utilizing the additional information of s i .
Revised: 4 to avoid dealing with the noise inside the min operator, by utilizing the additional information of s i .

Original: Proposition 3.5.
Revised: Proposition 3.5.

Original: For any δ > 0 , with probability at least 1 − δ , we have, | ˆ βNoise −
Revised: For any δ > 0 , with probability at least 1 − δ , we have, | ˆ β

Original: [Equation]
Revised: [Equation]

Original: | ≤ (cid:113) 12 n log 2 δ .

Original: Proof is shown in App.
Revised: Proof is shown in App.

Original: B.6.
Revised: B.6.

Original: This implies that ˆ β Noise is a consistent estimator to the Bayes error, i.e., for ∀ ϵ > 0 , lim n →∞ P ( | ˆ β Noise − β | ≥ ϵ ) = 0 .
Revised: This implies that ˆ β Noise is a consistent estimator to the Bayes error, i.e., for ∀ ϵ > 0 , lim n →∞ P ( | ˆ β Noise − β | ≥ ϵ ) = 0 .

Original: The rate of convergence is O P (1 / √ n ) .
Revised: The rate of convergence is O P (1 / √ n ) .

Original: In practice, one idea is to ask many labellers for each instance and then use the histogram of the hard labels as a noisy soft label.
Revised: In practice, one idea is to ask many labellers for each instance and then use the histogram of the hard labels as a noisy soft label.

Original: This is the main idea which we use later on in Sec.??
Revised: This is the main idea which we use later on in Sec.

Original:  with CIFAR-10H (Peterson et al., 2019).
Revised: 5.2 with CIFAR-10H (Peterson et al., 2019).

Original: For simplicity, suppose that we have m labels for every instance i , and u i is the sample average of those labels: u i = 1 m (cid:80) mj
Revised: For simplicity, suppose that we have m labels for every instance i , and u i is the sample average of those labels: u i = 1 m (cid:80) mj

Original: =1 1[ y i = 1] .
Revised: =1 1[ y i = 1] .

Original: This satisfies our assumptions for noisy soft labels since 0 ≤ u i ≤ 1 and E [ ξ i |
Revised: This satisfies our assumptions for noisy soft labels since 0 ≤ u i ≤ 1 and E [ ξ i |

Original: c i ] = 0 for any m ∈ N .
Revised: c i ] = 0 for any m ∈ N .

Original: If in practice we do not have access to sign labels, we cannot use the unbiased estimator ˆ β Noise and we have no choice but to use the biased estimator ˜ β .
Revised: If in practice we do not have access to sign labels, we cannot use the unbiased estimator ˆ β Noise and we have no choice but to use the biased estimator ˜ β .

Original: However, in the following proposition, we show the bias can be bounded with respect to m .
Revised: However, in the following proposition, we show the bias can be bounded with respect to m .

Original: Proposition 3.6.
Revised: Proposition 3.6.

Original: The estimator ˜ β is asymptotically unbiased, i.e., β − E [ (cid:101) β ] ≤ 1 2 √ m + (cid:112) log(2 n √ m ) / (2 m ) , where m is the number of labellers for each of the n instances.
Revised: The estimator ˜ β is asymptotically unbiased, i.e., β − E [ (cid:101) β ] ≤ 1 2 √ m + (cid:112) log(2 n √ m ) / (2 m ) , where m is the number of labellers for each of the n instances.

Original: Proof is shown in App.
Revised: Proof is shown in App.

Original: B.4.
Revised: B.4.

Original: The rate of convergence is O ( (cid:112) log(2 nm ) /m ) .
Revised: The rate of convergence is O ( (cid:112) log(2 nm ) /m ) .

Original: Hard labels Surprisingly, we can regard the usual hard labels as a special case of noisy soft labels.
Revised: Hard labels Surprisingly, we can regard the usual hard labels as a special case of noisy soft labels.

Original: This enables us to estimate the Bayes error with hard labels and sign labels.
Revised: This enables us to estimate the Bayes error with hard labels and sign labels.

Original: To see why this is the case, we consider a specific type of noise:
Revised: To see why this is the case, we consider a specific type of noise:

Original: [Equation]
Revised: [Equation]

Original: Since u i =
Revised: Since u i =

Original: c i + ξ i , this implies u
Revised: c i + ξ i , this implies u

Original: i
Revised: i

Original: =
Revised: =

Original: 1 w.p. p ( y = +

Revised: 1 w.p. p ( y = +

Original: | x i ) and u i = 

Revised: | x i ) and u i =

Original: w.p. p ( y = − 1 | x i ) .
Revised: w.p. p ( y = − 1 | x i ) .

Original: We can easily confirm that our assumptions for noisy soft labels are satisfied: 0 ≤ ξ i ≤ 1 and E [ ξ i | x i ] = 0 .
Revised: We can easily confirm that our assumptions for noisy soft labels are satisfied: 0 ≤ ξ i ≤ 1 and E [ ξ i | x i ] = 0 .

Original: If we assign y =
Revised: If we assign y =

Original: +1 for u
Revised: +1 for u

Original: i = 1 and y = − 1 for u
Revised: i = 1 and y = − 1 for u

Original: i
Revised: i

Original: = 0 , this will be equivalent to providing a hard label based on a Bernoulli distribution Be ( p ( y = +

Revised: = 0 , this will be equivalent to providing a hard label based on a Bernoulli distribution Be ( p ( y = +

Original: | x i )) for an instance x i sampled from p ( x ) .
Revised: | x i )) for an instance x i sampled from p ( x ) .

Original: This corresponds to the usual data generating process in supervised learning.
Revised: This corresponds to the usual data generating process in supervised learning.

Original:  E STIMATION WITH POSITIVE - CONFIDENCE DATA
Revised:  E STIMATION WITH POSITIVE - CONFIDENCE DATA

Original: In this section, we move our focus to the problem of estimating the Bayes error with positiveconfidence (Pconf) data (Sec. 2).
Revised: In this section, we move our focus to the problem of estimating the Bayes error with positiveconfidence (Pconf) data (Sec. 2).

Original: First, we show the following theorem.
Revised: First, we show the following theorem.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: The Bayes error can be expressed with positive-confidence data and class prior:
Revised: The Bayes error can be expressed with positive-confidence data and class prior:

Original: [Equation]
Revised: [Equation]

Original: Recall that E + is E p ( x | y =+1) and r ( x ) = p ( y = +

Revised: Recall that E + is E p ( x | y =+1) and r ( x ) = p ( y = +

Original: | x ) .
Revised: | x ) .

Original: The proof is shown in App.
Revised: The proof is shown in App.

Original: B.7.
Revised: B.7.

Original: Based on
Revised: Based on

Original: Theorem 4.1, we can derive the following estimator:
Revised: Theorem 4.1, we can derive the following estimator:

Original: [Equation]
Revised: [Equation]

Original: where r i = p ( y = +1 | x i ) for x i ∼ p ( x | y = +1) , and n + is the number of Pconf data.
Revised: where r i = p ( y = +1 | x i ) for x i ∼ p ( x | y = +1) , and n + is the number of Pconf data.

Original: We can estimate the Bayes error in an instance-free fashion, even if we do not have any r i from the negative distribution.
Revised: We can estimate the Bayes error in an instance-free fashion, even if we do not have any r i from the negative distribution.

Original: All we require is instance-free positive data and π + .
Revised: All we require is instance-free positive data and π + .

Original: A counter-intuitive point of Eq.
Revised: A counter-intuitive point of Eq.

Original: 9 is that we can throw away r i when r i < 0 .
Revised: 9 is that we can throw away r i when r i < 0 .

Original: 5 , due to the max operation.
Revised: 5 , due to the max operation.

Original: If the labeler is confident that r i < 0 . 5 , we do not need the specific value of r i .
Revised: If the labeler is confident that r i < 0 . 5 , we do not need the specific value of r i .

Original: Note that in practice, r i = 0 may occur for positive samples.
Revised: Note that in practice, r i = 0 may occur for positive samples.

Original: This will cause an issue for classifier training (since Eq.
Revised: This will cause an issue for classifier training (since Eq.

Original: 2 will diverge), but such samples can be ignored in Bayes error estimation (due to the max in Eq. 9).
Revised: 2 will diverge), but such samples can be ignored in Bayes error estimation (due to the max in Eq. 9).

Original: As discussed in Sec. 2, π + was unnecessary when learning a Pconf classifier.
Revised: As discussed in Sec. 2, π + was unnecessary when learning a Pconf classifier.

Original: Fortunately, this is available from other sources depending on the task, e.g., market share, government statistics, or other domain knowledge (Quadrianto et al., 2009; Patrini et al., 2014).
Revised: Fortunately, this is available from other sources depending on the task, e.g., market share, government statistics, or other domain knowledge (Quadrianto et al., 2009; Patrini et al., 2014).

Original: Instead of the Bayes error itself, we may be interested in the following: 1) the relative comparison between the Bayes error and the classifier’s error, 2) whether the classifier’s error has exceeded the Bayes error or not.
Revised: Instead of the Bayes error itself, we may be interested in the following: 1) the relative comparison between the Bayes error and the classifier’s error, 2) whether the classifier’s error has exceeded the Bayes error or not.

Original: In these cases, π + is unnecessary since we can divide both errors (Eq. 9 and empirical version of Eq. 2) by π + without changing the order of the two.
Revised: In these cases, π + is unnecessary since we can divide both errors (Eq. 9 and empirical version of Eq. 2) by π + without changing the order of the two.

Original: We can derive the following two propositions.
Revised: We can derive the following two propositions.

Original: Proposition 4.2.
Revised: Proposition 4.2.

Original: The estimator ˆ β Pconf is an unbiased estimator of the Bayes error.
Revised: The estimator ˆ β Pconf is an unbiased estimator of the Bayes error.

Original: Proposition 4.3.
Revised: Proposition 4.3.

Original: For any δ > 0 , with probability at least 1 − δ , we have, | ˆ β Pconf − β | ≤
Revised: For any δ > 0 , with probability at least 1 − δ , we have, | ˆ β Pconf − β | ≤

Original: [Equation]
Revised: [Equation]

Original: Proofs are shown in App.
Revised: Proofs are shown in App.

Original: B.8 and App.
Revised: B.8 and App.

Original: B.9, respectively.
Revised: B.9, respectively.

Original: This implies that the estimator ˆ β Pconf is consistent to the Bayes error, i.e., for ∀ ϵ > 0 , lim n →∞ P ( | ˆ β Pconf − β | ≥ ϵ )
Revised: This implies that the estimator ˆ β Pconf is consistent to the Bayes error, i.e., for ∀ ϵ > 0 , lim n →∞ P ( | ˆ β Pconf − β | ≥ ϵ )

Original: = 0 .
Revised: = 0 .

Original: The rate of convergence is O P (1 / √ n + ) .
Revised: The rate of convergence is O P (1 / √ n + ) .

Original: We can also see that the upper bound is equivalent to the upper bound in Prop. 3.2 when the data is balanced, i.e., π + = 0 .
Revised: We can also see that the upper bound is equivalent to the upper bound in Prop. 3.2 when the data is balanced, i.e., π + = 0 .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we perform experiments with synthetic data to investigate the behavior of our methods.
Revised: In this section, we perform experiments with synthetic data to investigate the behavior of our methods.

Original: Then, we use benchmark datasets to compare the estimated Bayes error with SOTA classifiers.
Revised: Then, we use benchmark datasets to compare the estimated Bayes error with SOTA classifiers.

Original: 5. I LLUSTRATION
Revised: 5. I LLUSTRATION

Original: Positive-negative data Setup: We use two Gaussian distributions to construct a binary classification dataset.
Revised: Positive-negative data Setup: We use two Gaussian distributions to construct a binary classification dataset.

Original: We then derive the positive-class posterior analytically for each data and use it as our label.
Revised: We then derive the positive-class posterior analytically for each data and use it as our label.

Original: We use Eq.
Revised: We use Eq.

Original: 4 to estimate the Bayes error.
Revised: 4 to estimate the Bayes error.

Original: We change the number of training samples per class ( 2 , 4 , 8 , . . . , 1028 ) to observe how quickly we can approach the true Bayes error.
Revised: We change the number of training samples per class ( 2 , 4 , 8 , . . . , 1028 ) to observe how quickly we can approach the true Bayes error.

Original: We perform experiments with four different setups (A, B, C, and D) and the details are explained in the supplementary material.
Revised: We perform experiments with four different setups (A, B, C, and D) and the details are explained in the supplementary material.

Original: To compare with the true Bayes error of the setup, we take the
Revised: To compare with the true Bayes error of the setup, we take the

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: average of c ( x ) for 10 , 000 samples.
Revised: average of c ( x ) for 10 , 000 samples.

Original: We expect this to be a good substitute because we are using the analytically derived class posteriors for the soft labels.
Revised: We expect this to be a good substitute because we are using the analytically derived class posteriors for the soft labels.

Original: Although our problem setting does not require instances, we also compare with a recent Bayes error estimator that uses instances and hard labels (Noshad et al., 2019) and classifiers trained from instances and hard labels.
Revised: Although our problem setting does not require instances, we also compare with a recent Bayes error estimator that uses instances and hard labels (Noshad et al., 2019) and classifiers trained from instances and hard labels.

Original: We train a multi-layer perceptron model with one hidden layer (500 units) with the following cross-entropy loss: − 1 [ y i =
Revised: We train a multi-layer perceptron model with one hidden layer (500 units) with the following cross-entropy loss: − 1 [ y i =

Original: +1] log ˆ p θ ( y
Revised: +1] log ˆ p θ ( y

Original: = +

Revised: = +

Original: | x i ) − 

Revised: | x i ) −

Original: [ y i = − 1] log ˆ p θ ( y = − 1 | x i ) , where ˆ p θ ( y | x ) is the probabilistic output of the trained classifier and θ represents the model parameters.
Revised: [ y i = − 1] log ˆ p θ ( y = − 1 | x i ) , where ˆ p θ ( y | x ) is the probabilistic output of the trained classifier and θ represents the model parameters.

Original: We chose the cross entropy loss because it is a proper scoring rule (Kull & Flach, 2015).
Revised: We chose the cross entropy loss because it is a proper scoring rule (Kull & Flach, 2015).

Original: We use stochastic gradient descent for 150 epochs with a learning rate of 0.005 and momentum of 0.9.
Revised: We use stochastic gradient descent for 150 epochs with a learning rate of 0.005 and momentum of 0.9.

Original: We also compare with another baseline (“classifier-BER”), where we plug in ˆ p θ ( y | x ) to Eq.
Revised: We also compare with another baseline (“classifier-BER”), where we plug in ˆ p θ ( y | x ) to Eq.

Original: Finally, we add a stronger baseline “soft classifier” which utilizes soft labels.
Revised: Finally, we add a stronger baseline “soft classifier” which utilizes soft labels.

Original: If the soft label is r i for sample x i , then the cross-entropy loss is modified as follows:
Revised: If the soft label is r i for sample x i , then the cross-entropy loss is modified as follows:

Original: [Equation]
Revised: [Equation]

Original: y = +

Revised: y = +

Original: | x i ) − (1 − r i ) log ˆ p θ ( y = − 1 | x i ) .
Revised: | x i ) − (1 − r i ) log ˆ p θ ( y = − 1 | x i ) .

Original: Results: We construct a dataset with the same underlying distribution ten times and report the means and standard error for the final epoch in Fig.
Revised: Results: We construct a dataset with the same underlying distribution ten times and report the means and standard error for the final epoch in Fig.

Original: We can see how the estimated Bayes error is relatively accurate even when we have few samples.
Revised: We can see how the estimated Bayes error is relatively accurate even when we have few samples.

Original: The benefit against classifier training is clear: the MLP model requires much more samples before the test error approaches the Bayes error.
Revised: The benefit against classifier training is clear: the MLP model requires much more samples before the test error approaches the Bayes error.

Original: The classifier is slightly better when it can use soft labels but is not as good as our proposed method.
Revised: The classifier is slightly better when it can use soft labels but is not as good as our proposed method.

Original: It is interesting how the “hard classifier-BER” tends to underestimate the Bayes error (except Fig. 1b).
Revised: It is interesting how the “hard classifier-BER” tends to underestimate the Bayes error (except Fig. 1b).

Original: This can be explained by the overconfidence of neural networks (Guo et al., 2017), which will lead to a small min y ∈{± 1 } ˆ p ( y | x ) .
Revised: This can be explained by the overconfidence of neural networks (Guo et al., 2017), which will lead to a small min y ∈{± 1 } ˆ p ( y | x ) .

Original: This underestimation does not occur with “soft classifier-BER,” implying that there are less room to be overconfident when we learn with the modified cross-entropy loss.
Revised: This underestimation does not occur with “soft classifier-BER,” implying that there are less room to be overconfident when we learn with the modified cross-entropy loss.

Original: Positive-negative data with noise Setup: Next, we perform experiments when we have noise on the labels.
Revised: Positive-negative data with noise Setup: Next, we perform experiments when we have noise on the labels.

Original: We follow the assumption in Sec. 3, and we design the noise to have mean 0 and
Revised: We follow the assumption in Sec. 3, and we design the noise to have mean 0 and

Original: [Figure]
Revised: [Figure]

Original: the label to be between 0 and 1 after adding this noise, to be a valid probability.
Revised: the label to be between 0 and 1 after adding this noise, to be a valid probability.

Original: For this purpose, we add noise to the underlying class posterior and use a truncated Gaussian distribution p ( z ) with standard deviation 0.4.
Revised: For this purpose, we add noise to the underlying class posterior and use a truncated Gaussian distribution p ( z ) with standard deviation 0.4.

Original: We truncate the left tail { z ∈ R | z ≤ µ − min(1 − µ, µ − 0) } and the right tail { z ∈ R | z ≥ µ + min(1 − µ, µ − 0) } where µ ∈ [0 , 1] is the mean of the distribution, i.e., the true class posterior.
Revised: We truncate the left tail { z ∈ R | z ≤ µ − min(1 − µ, µ − 0) } and the right tail { z ∈ R | z ≥ µ + min(1 − µ, µ − 0) } where µ ∈ [0 , 1] is the mean of the distribution, i.e., the true class posterior.

Original: We compare three estimators: 1) Original estimator ˆ β in Eq.
Revised: We compare three estimators: 1) Original estimator ˆ β in Eq.

Original: 4 with clean labels, 2) original estimator ˜ β in Eq.
Revised: 4 with clean labels, 2) original estimator ˜ β in Eq.

Original: 4 with noisy labels, and 3) proposed estimator ˆ β Noise in Eq.
Revised: 4 with noisy labels, and 3) proposed estimator ˆ β Noise in Eq.

Original: 6 with noisy labels.
Revised: 6 with noisy labels.

Original: We repeat the experiment 10 times with different data realizations.
Revised: We repeat the experiment 10 times with different data realizations.

Original: Results:
Revised: Results:

Original: The mean and standard error for 10 trials are shown in Fig.
Revised: The mean and standard error for 10 trials are shown in Fig.

Original: 2, for varying number of training samples.
Revised: 2, for varying number of training samples.

Original: As expected, using noisy soft labels lead to worse results for ˜ β , as we can see with the larger standard errors.
Revised: As expected, using noisy soft labels lead to worse results for ˜ β , as we can see with the larger standard errors.

Original: Increasing the sample size is effective to some degree, but the bias does not vanish.
Revised: Increasing the sample size is effective to some degree, but the bias does not vanish.

Original: However, if we use a modified estimator ˆ β Noise , the bias vanishes completely and works well even with noise.
Revised: However, if we use a modified estimator ˆ β Noise , the bias vanishes completely and works well even with noise.

Original: Positive-confidence data Setup: Similarly, we perform experiments for positive-confidence (Pconf) data.
Revised: Positive-confidence data Setup: Similarly, we perform experiments for positive-confidence (Pconf) data.

Original: The data is generated so that the class prior π + is 0 . 5 , and we assume this information is available.
Revised: The data is generated so that the class prior π + is 0 . 5 , and we assume this information is available.

Original: We only use the confidence of data from the positive distribution and do not use the confidence of data from the negative distribution.
Revised: We only use the confidence of data from the positive distribution and do not use the confidence of data from the negative distribution.

Original: We use this data to estimate ˆ β Pconf and compare it with the ˆ β derived earlier.
Revised: We use this data to estimate ˆ β Pconf and compare it with the ˆ β derived earlier.

Original: Results:
Revised: Results:

Original: We report the mean and standard error for 10 trials in Fig.
Revised: We report the mean and standard error for 10 trials in Fig.

Original: Although Bayes error estimation tends to be inaccurate with Pconf data for fewer samples, the difference between the two disappears with more samples.
Revised: Although Bayes error estimation tends to be inaccurate with Pconf data for fewer samples, the difference between the two disappears with more samples.

Original: 5. E XPERIMENTS WITH BENCHMARK DATASETS : CIFAR- 
Revised: 5. E XPERIMENTS WITH BENCHMARK DATASETS : CIFAR-

Original: Setup: We use CIFAR-10 (C10) (Krizhevsky, 2009), ciFAIR-10 (C10F) (Barz & Denzler, 2020),
Revised: Setup: We use CIFAR-10 (C10) (Krizhevsky, 2009), ciFAIR-10 (C10F) (Barz & Denzler, 2020),

Original: CIFAR-10.1 (C10.1) (Recht et al., 2018), and CIFAR-10H (C10H) (Peterson et al., 2019).
Revised: CIFAR-10.1 (C10.1) (Recht et al., 2018), and CIFAR-10H (C10H) (Peterson et al., 2019).

Original: C10 is an image dataset of 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.
Revised: C10 is an image dataset of 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.

Original: However, the test set in C10 is known to include complete duplicates and near-duplicates from the training dataset.
Revised: However, the test set in C10 is known to include complete duplicates and near-duplicates from the training dataset.

Original: C10F replaces the duplicates in the test set with a different image.
Revised: C10F replaces the duplicates in the test set with a different image.

Original: C10.1 is a new test set to evaluate C10 classifiers.
Revised: C10.1 is a new test set to evaluate C10 classifiers.

Original: It ensures the underlying distribution to be as close to the original distribution as possible, but it does not include any images from the original test set.
Revised: It ensures the underlying distribution to be as close to the original distribution as possible, but it does not include any images from the original test set.

Original: C10H is a dataset with additional human annotations for the C10 test set.
Revised: C10H is a dataset with additional human annotations for the C10 test set.

Original: Each image in the test set has class labels annotated by around 50 different human labelers.
Revised: Each image in the test set has class labels annotated by around 50 different human labelers.

Original: We can take the proportion to construct soft labels that we will use for Bayes error estimation.
Revised: We can take the proportion to construct soft labels that we will use for Bayes error estimation.

Original: Although these datasets are originally multi-class setups with 10 classes, we re-construct binary datasets in the following four ways: animals vs. artifacts, land vs. other (water/sky), odd vs. even, and first five vs. last five.
Revised: Although these datasets are originally multi-class setups with 10 classes, we re-construct binary datasets in the following four ways: animals vs. artifacts, land vs. other (water/sky), odd vs. even, and first five vs. last five.

Original: See the details after Sec. 6.
Revised: See the details after Sec. 6.

Original: We compare the estimated Bayes error ˆ β with the base model of Vision Transformer (Dosovitskiy et al., 2021) with 16 × 16 input patch size (ViT B/16), DenseNet169 (Huang et al., 2017), ResNet (He et al., 2016), and VGG11 (Simonyan & Zisserman, 2015).
Revised: We compare the estimated Bayes error ˆ β with the base model of Vision Transformer (Dosovitskiy et al., 2021) with 16 × 16 input patch size (ViT B/16), DenseNet169 (Huang et al., 2017), ResNet (He et al., 2016), and VGG11 (Simonyan & Zisserman, 2015).

Original: For ViT B/16, we downloaded the official pretrained model (Dosovitskiy et al., 2021) and used the implementation by Jeon (Jeon, 2020) to fine-tune on C10.
Revised: For ViT B/16, we downloaded the official pretrained model (Dosovitskiy et al., 2021) and used the implementation by Jeon (Jeon, 2020) to fine-tune on C10.

Original: For others, we use the models provided in Phan (Phan, 2021), which are trained on C10.
Revised: For others, we use the models provided in Phan (Phan, 2021), which are trained on C10.

Original: Although these are ten-class classifiers, we transform them into binary classifiers by
Revised: Although these are ten-class classifiers, we transform them into binary classifiers by

Original: [Figure]
Revised: [Figure]

Original: Vision Transformer B/16 has surpassed the Bayes error for all four setups for CIFAR-10 (C10), ciFAIR-10 (C10F), and CIFAR-10.1 (C10.1), while it is still comparable or larger than the Bayes error for CIFAR-10H (C10H).
Revised: Vision Transformer B/16 has surpassed the Bayes error for all four setups for CIFAR-10 (C10), ciFAIR-10 (C10F), and CIFAR-10.1 (C10.1), while it is still comparable or larger than the Bayes error for CIFAR-10H (C10H).

Original: In Sec. 5.2, we discuss how this has to be investigated carefully.
Revised: In Sec. 5.2, we discuss how this has to be investigated carefully.

Original: The black error bar for the Bayes error is the 95% confidence interval.
Revised: The black error bar for the Bayes error is the 95% confidence interval.

Original: The black error bar for the C10H is the standard error for sampling test labels 20 times.
Revised: The black error bar for the C10H is the standard error for sampling test labels 20 times.

Original: swapping the predictions into positive and negative based on the setups (App. ?? ).
Revised: swapping the predictions into positive and negative based on the setups (details shown after Sec. 6).

Original: For example, a prediction of “cat” would be swapped with a positive label in “animals vs. artifacts”.
Revised: For example, a prediction of “cat” would be swapped with a positive label in “animals vs. artifacts”.

Original: Results: We compare the estimated Bayes error ˆ β with classifier training in Fig.
Revised: Results: We compare the estimated Bayes error ˆ β with classifier training in Fig.

Original: As expected, the test error (shown in light blue) is lower for newer network architectures, with ViT achieving the lowest test error.
Revised: As expected, the test error (shown in light blue) is lower for newer network architectures, with ViT achieving the lowest test error.

Original: An interesting observation is that the test error of ViT has exceeded the estimated
Revised: An interesting observation is that the test error of ViT has exceeded the estimated

Original: Bayes error in all four setups, which theoretically should not take place.
Revised: Bayes error in all four setups, which theoretically should not take place.

Original: We first suspected that ViT’s error is too low (overfitting to the test set; Sec.
Revised: We first suspected that ViT’s error is too low (overfitting to the test set; Sec.

Original: A) but found out this is not necessarily the case.
Revised: A) but found out this is not necessarily the case.

Original: We investigated the degree of overfitting to the test set by performing experiments with C10F and C10.1.
Revised: We investigated the degree of overfitting to the test set by performing experiments with C10F and C10.1.

Original: The former dataset excludes the duplicated images between training and test set, and the latter dataset is a fresh test set that is designed to be close to the original distribution of C10.
Revised: The former dataset excludes the duplicated images between training and test set, and the latter dataset is a fresh test set that is designed to be close to the original distribution of C10.

Original: The test error for the C10F dataset becomes slightly worse and much worse for C10.1.
Revised: The test error for the C10F dataset becomes slightly worse and much worse for C10.1.

Original: This is what we expected and implies that overfitting to the test set may be happening to some degree.
Revised: This is what we expected and implies that overfitting to the test set may be happening to some degree.

Original: However, ViT’s error is still below the estimated Bayes error even for C10.1.
Revised: However, ViT’s error is still below the estimated Bayes error even for C10.1.

Original: To explain why ViT’s error is still below the estimated Bayes error, we discuss the discrepancy between the underlying p ( y | x ) for C10 and C10H. C10 is constructed with a three-step procedure: first, collect images by searching for specific terms in search engines, downscale them to 32 ×(Torralba et al., 2008), and finally ask labelers to filter out mislabelled images (Krizhevsky, 2009).
Revised: To explain why ViT’s error is still below the estimated Bayes error, we discuss the discrepancy between the underlying p ( y | x ) for C10 and C10H. C10 is constructed with a three-step procedure: first, collect images by searching for specific terms in search engines, downscale them to 32 ×(Torralba et al., 2008), and finally ask labelers to filter out mislabelled images (Krizhevsky, 2009).

Original: On the other hand, C10H constructs the labels by asking labelers to choose the correct class out of ten classes after seeing each downscaled image of C10.
Revised: On the other hand, C10H constructs the labels by asking labelers to choose the correct class out of ten classes after seeing each downscaled image of C10.

Original: This makes it visually harder for human labellers.
Revised: This makes it visually harder for human labellers.

Original: Hence, the labelling process will become less confident, and this may lead to a higher entropy of the distribution of the human labels.
Revised: Hence, the labelling process will become less confident, and this may lead to a higher entropy of the distribution of the human labels.

Original: According to Eq. 4, this will lead to an overestimation of ˆ β .
Revised: According to Eq. 4, this will lead to an overestimation of ˆ β .

Original: To adjust for the distribution discrepancy, we also plot the test errors when we sample a test label for each sample from the C10H dataset (repeated 20 times) and show this in Fig.
Revised: To adjust for the distribution discrepancy, we also plot the test errors when we sample a test label for each sample from the C10H dataset (repeated 20 times) and show this in Fig.

Original: 4 with pink.
Revised: 4 with pink.

Original: Note that this generally will not go below the estimated Bayes error.
Revised: Note that this generally will not go below the estimated Bayes error.

Original: For example, consider an extreme case with zero test error for the original test set.
Revised: For example, consider an extreme case with zero test error for the original test set.

Original: If a test sample has p ( y = +1 | x ) = 0 . 8 in C10H, then it will incur an error of 0.2 on average for the new test labels.
Revised: If a test sample has p ( y = +1 | x ) = 0 . 8 in C10H, then it will incur an error of 0.2 on average for the new test labels.

Original: After adjusting for the discrepancy, we can see how the test error of ViT is on par with or slightly higher than the estimated Bayes error.
Revised: After adjusting for the discrepancy, we can see how the test error of ViT is on par with or slightly higher than the estimated Bayes error.

Original: This implies that we have reached (or is about to reach) the Bayes error for C10H.
Revised: This implies that we have reached (or is about to reach) the Bayes error for C10H.

Original: 5. E XPERIMENTS WITH BENCHMARK DATASETS : F ASHION -MNIST
Revised: 5. E XPERIMENTS WITH BENCHMARK DATASETS : F ASHION -MNIST

Original: Although there are many research areas within machine learning that utilize soft labels or multiple hard labels, 3 to our knowledge there is only one large-scale dataset with over 10 annotations per sample (Peterson et al., 2019), which we used in Sec. 5.2.
Revised: Although there are many research areas within machine learning that utilize soft labels or multiple hard labels, 4 to our knowledge there is only one large-scale dataset with over 10 annotations per sample (Peterson et al., 2019), which we used in Sec. 5.2.

Original: Hence, it would be beneficial to construct another benchmark dataset with multiple annotations.
Revised: Hence, it would be beneficial to construct another benchmark dataset with multiple annotations.

Original: This will also demonstrate how our Bayes error estimator can be used for more difficult benchmark datasets compared with CIFAR-10 in Sec. 5.2.
Revised: This will also demonstrate how our Bayes error estimator can be used for more difficult benchmark datasets compared with CIFAR-10 in Sec. 5.2.

Original: Data collection:
Revised: Data collection:

Original: We chose the Fashion-MNIST dataset (Xiao et al., 2017) as a more challenging dataset.
Revised: We chose the Fashion-MNIST dataset (Xiao et al., 2017) as a more challenging dataset.

Original: This is a 10 class classification dataset with small fashion images with categories such as dress, bag, and sneaker.
Revised: This is a 10 class classification dataset with small fashion images with categories such as dress, bag, and sneaker.

Original: Other popular datasets such as CIFAR-100 (Krizhevsky, 2009) and ImageNet (Deng et al., 2009) are problematic because they are known to be essentially multi-label classification datasets rather than single-label (Shankar et al., 2020; Beyer et al., 2020;
Revised: Other popular datasets such as CIFAR-100 (Krizhevsky, 2009) and ImageNet (Deng et al., 2009) are problematic because they are known to be essentially multi-label classification datasets rather than single-label (Shankar et al., 2020; Beyer et al., 2020;

Original: Yun et al., 2021;
Revised: Yun et al., 2021;

Original: Wei et al., 2022).
Revised: Wei et al., 2022).

Original: Since the Bayes error is defined for the single-labeled case, this means we have to define a similar but different and novel concept that corresponds to the lowest error for multi-label classification if we want to estimate the best possible performance on datasets such as CIFARand ImageNet.
Revised: Since the Bayes error is defined for the single-labeled case, this means we have to define a similar but different and novel concept that corresponds to the lowest error for multi-label classification if we want to estimate the best possible performance on datasets such as CIFARand ImageNet.

Original: This is definitely an interesting direction to pursue in the future, but is out of the scope of the current paper since we are focusing on the Bayes error.
Revised: This is definitely an interesting direction to pursue in the future, but is out of the scope of the current paper since we are focusing on the Bayes error.

Original: We used Amazon Mechanical Turk (Buhrmester et al., 2016) to collect multiple annotations per each image in the test dataset with 10,000 images.
Revised: We used Amazon Mechanical Turk (Buhrmester et al., 2016) to collect multiple annotations per each image in the test dataset with 10,000 images.

Original: We designed the task so that the worker is instructed to annotate 10 randomly chosen fashion images, with a multiple choice of 10 labels.
Revised: We designed the task so that the worker is instructed to annotate 10 randomly chosen fashion images, with a multiple choice of 10 labels.

Original: We showed several correct images for each of categories in the beginning of the task, so that the worker can have a good understanding of each labels.
Revised: We showed several correct images for each of categories in the beginning of the task, so that the worker can have a good understanding of each labels.

Original: For each task with 10 image annotations, we paid $0.06 or $0.08 to the worker.
Revised: For each task with 10 image annotations, we paid $0.06 or $0.08 to the worker.

Original: We removed the duplicates when the same worker provide a label for the same image more than once.
Revised: We removed the duplicates when the same worker provide a label for the same image more than once.

Original: We further removed the annotation when the worker provided an answer withinseconds, which we considered as too short, e.g., when we performed the task, it took us 45 seconds when we skipped reading the instructions.
Revised: We further removed the annotation when the worker provided an answer withinseconds, which we considered as too short, e.g., when we performed the task, it took us 45 seconds when we skipped reading the instructions.

Original: After removing the shorter ones, the mean and median answering time was about 147 and 78 seconds, respectively.
Revised: After removing the shorter ones, the mean and median answering time was about 147 and 78 seconds, respectively.

Original: Finally, we ended up with aroundannotations per image.
Revised: Finally, we ended up with aroundannotations per image.

Original: The least number was 53 and the largest number was 79 annotations.
Revised: The least number was 53 and the largest number was 79 annotations.

Original: Setup and results: Since aiming for SOTA performance on Fashion-MNIST is less common in our community compared with CIFAR-10, we simply chose ResNet-18 (He et al., 2016) for our model architecture.
Revised: Setup and results: Since aiming for SOTA performance on Fashion-MNIST is less common in our community compared with CIFAR-10, we simply chose ResNet-18 (He et al., 2016) for our model architecture.

Original: We trained for 100 epochs with batch size 128, learning rate 0.001 on the original labels (no data augmentation) and reconstructed the 10-class predictions into binary predictions by grouping shirt, dress, coat, pullover, and t-shirt/top as the positive class, and the other 5 categories as the negative class.
Revised: We trained for 100 epochs with batch size 128, learning rate 0.001 on the original labels (no data augmentation) and reconstructed the 10-class predictions into binary predictions by grouping shirt, dress, coat, pullover, and t-shirt/top as the positive class, and the other 5 categories as the negative class.

Original: We compared the binary predictions with a test label sampled from the multiple annotations (repeated 10 times) and derived the error, which is basically the same procedure of the pink bar in Fig. 4.
Revised: We compared the binary predictions with a test label sampled from the multiple annotations (repeated 10 times) and derived the error, which is basically the same procedure of the pink bar in Fig. 4.

Original: The error was 3.852% ( ± 0.041%), where percentage in parenthesis is the standard error.
Revised: The error was 3.852% ( ± 0.041%), where percentage in parenthesis is the standard error.

Original: On the other hand, the estimated Bayes error was 3.478 % ( ± 0.079 % ), where percentage in parenthesis is the 95 % confidence interval.
Revised: On the other hand, the estimated Bayes error was 3.478 % ( ± 0.079 % ), where percentage in parenthesis is the 95 % confidence interval.

Original: We can see how the estimated Bayes error is higher than the results for CIFAR-10H in Fig. 4, implying that Fashion-MNIST is more difficult than CIFAR-10.
Revised: We can see how the estimated Bayes error is higher than the results for CIFAR-10H in Fig. 4, implying that Fashion-MNIST is more difficult than CIFAR-10.

Original: This is intuitive as the 28 × 28 grayscale images are ambiguous while the original 51 × 73 PNG image is much more clear, e.g., see Fig.
Revised: This is intuitive as the 28 × 28 grayscale images are ambiguous while the original 51 × 73 PNG image is much more clear, e.g., see Fig.

Original: 1 in Xiao et al.
Revised: 1 in Xiao et al.

Original: (2017), and also consistent with the results in Theisen et al.
Revised: (2017), and also consistent with the results in Theisen et al.

Original: The ResNet-18 error is approaching the estimated Bayes error, but may still have room for small improvement.
Revised: The ResNet-18 error is approaching the estimated Bayes error, but may still have room for small improvement.

Original: Since we made a simple choice of model architecture, we may be able to close this gap by other architectures that are more suited for Fashion-MNIST images, adding data augmentation, and trying extensive hyper-parameter tuning.
Revised: Since we made a simple choice of model architecture, we may be able to close this gap by other architectures that are more suited for Fashion-MNIST images, adding data augmentation, and trying extensive hyper-parameter tuning.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed a model-free and instance-free Bayes error estimation method that is unbiased and consistent.
Revised: We proposed a model-free and instance-free Bayes error estimation method that is unbiased and consistent.

Original: We showed how we can similarly estimate it with uncertainty labels, noisy soft labels, or positive-confidence.
Revised: We showed how we can similarly estimate it with uncertainty labels, noisy soft labels, or positive-confidence.

Original: In experiments, we demonstrated how the proposed method could be helpful when we want to compare the SOTA performance for benchmark datasets with the Bayes error.
Revised: In experiments, we demonstrated how the proposed method could be helpful when we want to compare the SOTA performance for benchmark datasets with the Bayes error.

Original: Although we did not detect strong evidence of test set overfitting in Sec.
Revised: Although we did not detect strong evidence of test set overfitting in Sec.

Original: 5.2 and Sec.
Revised: 5.2 and Sec.

Original: 5.3, we are interested in investigating other benchmark datasets in the future.
Revised: 5.3, we are interested in investigating other benchmark datasets in the future.

Original: A current limitation is that we focus on binary classification.
Revised: A current limitation is that we focus on binary classification.

Original: Since binary classification is popular in industry, e.g., click prediction, fraud detection, anomaly detection, churn prediction, we believe estimating the Bayes error for binary classification problems is interesting on its own.
Revised: Since binary classification is popular in industry, e.g., click prediction, fraud detection, anomaly detection, churn prediction, we believe estimating the Bayes error for binary classification problems is interesting on its own.

Original: However, in the future we are interested in working on the multi-class case (and the multi-label case we discussed in Sec. 5.3).
Revised: However, in the future we are interested in working on the multi-class case (and the multi-label case we discussed in Sec. 5.3).

Original: Ethics statement: Constructing soft labels may rely on collecting human annotations.
Revised: Ethics statement: Constructing soft labels may rely on collecting human annotations.

Original: This may potentially have a negative impact, e.g., cost of the annotations.
Revised: This may potentially have a negative impact, e.g., cost of the annotations.

Original: We are hoping to lower the cost of annotations in our future work.
Revised: We are hoping to lower the cost of annotations in our future work.

Original: We will only publish our annotations we collected in Sec.
Revised: We will only publish our annotations we collected in Sec.

Original: 5.3 in a way that does not contain any private information, e.g., only the category counts for each images and drop all other worker-related information.
Revised: 5.3 in a way that does not contain any private information, e.g., only the category counts for each images and drop all other worker-related information.

Original: Reproducibility statement: In the synthetic experiments, we used four setups: A, B, C, D. A is a 10 -dimensional isotropic setup with means µ + = [0 , . . .
Revised: Reproducibility statement: In the synthetic experiments, we used four setups: A, B, C, D. A is a 10 -dimensional isotropic setup with means µ + = [0 , . . .

Original: , 0] ⊤ and µ − =
Revised: , 0] ⊤ and µ − =

Original: , 1] ⊤ with identity matrices for the covariance matrices.
Revised: , 1] ⊤ with identity matrices for the covariance matrices.

Original: B is the same as A except that the dimension is doubled to 20 .
Revised: B is the same as A except that the dimension is doubled to 20 .

Original: C and D are 2 -dimensional anisotropic setups and is borrowed from the first two synthetic datasets in Ishida et al.
Revised: C and D are 2 -dimensional anisotropic setups and is borrowed from the first two synthetic datasets in Ishida et al.

Original: (Ishida et al., 2018), where the code is provided in https: //github.com/takashiishida/pconf .
Revised: (Ishida et al., 2018), where the code is provided in https: //github.com/takashiishida/pconf .

Original: We used data and code provided in https://cvjena.github.io/cifair/ for ciFAIR10 (Barz & Denzler, 2020).
Revised: We used data and code provided in https://cvjena.github.io/cifair/ for ciFAIR10 (Barz & Denzler, 2020).

Original: We used data and code provided in https://github.com/ jcpeterson/cifar-10h for CIFAR-10H (Peterson et al., 2019).
Revised: We used data and code provided in https://github.com/ jcpeterson/cifar-10h for CIFAR-10H (Peterson et al., 2019).

Original: We used data and code provided in https://github.com/modestyachts/CIFAR-10.1 for CIFAR-10.1 (Recht et al., 2019).
Revised: We used data and code provided in https://github.com/modestyachts/CIFAR-10.1 for CIFAR-10.1 (Recht et al., 2019).

Original: We used data provided in https://www.cs.toronto.edu/~kriz/cifar.
Revised: We used data provided in https://www.cs.toronto.edu/~kriz/cifar.

Original: html for CIFAR-10.
Revised: html for CIFAR-10.

Original: We are not aware of any personally identifiable information or offensive content in the datasets we used.
Revised: We are not aware of any personally identifiable information or offensive content in the datasets we used.

Original: We also collected annotations by using Amazon Mechanical Turk (Buhrmester et al., 2016).
Revised: We also collected annotations by using Amazon Mechanical Turk (Buhrmester et al., 2016).

Original: We explained the details of how we collected the annotations in Sec. 5.3, and we plan to remove all personally identifiable information when we publish the data after acceptance of the paper.
Revised: We explained the details of how we collected the annotations in Sec. 5.3, and we plan to remove all personally identifiable information when we publish the data after acceptance of the paper.

Original: We used https://github.com/mrtnoshad/Bayes_Error_Estimator for the ensemble method (Noshad et al., 2019).
Revised: We used https://github.com/mrtnoshad/Bayes_Error_Estimator for the ensemble method (Noshad et al., 2019).

Original: We partially used https://github.com/ takashiishida/flooding for training classifiers in synthetic experiments with Gaussian distributions.
Revised: We partially used https://github.com/ takashiishida/flooding for training classifiers in synthetic experiments with Gaussian distributions.

Original: We used https://github.com/huyvnphan/PyTorch_CIFAR10 for
Revised: We used https://github.com/huyvnphan/PyTorch_CIFAR10 for

Original: VGG11, ResNet50, and DenseNet169 models trained on CIFAR-10.
Revised: VGG11, ResNet50, and DenseNet169 models trained on CIFAR-10.

Original: We used https://github. com/jeonsworld/ViT-pytorch to fine-tune ViT and downloaded pretrained weights from https://console.cloud.google.com/storage/browser/vit_models;tab= objects?pli=1&prefix=&forceOnObjectsSortingFiltering=
Revised: We used https://github. com/jeonsworld/ViT-pytorch to fine-tune ViT and downloaded pretrained weights from https://console.cloud.google.com/storage/browser/vit_models;tab= objects?pli=1&prefix=&forceOnObjectsSortingFiltering=

Original: false .
Revised: false .

Original: All of our experiments were conducted with a single Mac Pro except when we trained ResNet-18 in Sec. 5.
Revised: All of our experiments were conducted with a single Mac Pro except when we trained ResNet-18 in Sec. 5.

Original: and fine-tuned ViT in Sec.
Revised: and fine-tuned ViT in Sec.

Original: 5.2, which was when we utilized Google Colab Pro+.
Revised: 5.2, which was when we utilized Google Colab Pro+.

Original: The details of the four binary datasets that are used in Sec.
Revised: The details of the four binary datasets that are used in Sec.

Original: 5.2 is as follows.
Revised: 5.2 is as follows.

Original: • Animals vs. artifacts: we divide the dataset into cat, deer, dog, frog, and horse for the positive class and plane, car, ship, truck, and bird for the negative class.
Revised: • Animals vs. artifacts: we divide the dataset into cat, deer, dog, frog, and horse for the positive class and plane, car, ship, truck, and bird for the negative class.

Original: • Land vs. other (water/sky): we divide the dataset into car, truck, cat, deer, dog, horse for the positive class and plane, ship, bird, and frog for the negative class.
Revised: • Land vs. other (water/sky): we divide the dataset into car, truck, cat, deer, dog, horse for the positive class and plane, ship, bird, and frog for the negative class.

Original: • Odd vs. even: we divide the dataset into odd and even class numbers, with plane, bird, deer, frog, and ship for the positive class and car, cat, dog, horse, and truck for the negative class.
Revised: • Odd vs. even: we divide the dataset into odd and even class numbers, with plane, bird, deer, frog, and ship for the positive class and car, cat, dog, horse, and truck for the negative class.

Original: • First five vs. last five: we divide the dataset into the first five and last five classes based on the class number, with plane, car, bird, cat, and deer for the positive class and dog, frog, horse, ship, and truck for the negative class.
Revised: • First five vs. last five: we divide the dataset into the first five and last five classes based on the class number, with plane, car, bird, cat, and deer for the positive class and dog, frog, horse, ship, and truck for the negative class.

Original: A R ELATED WORK
Revised: A R ELATED WORK

Original: Bayes error estimation has been a topic of interest for nearly half a century (Fukunaga & Hostetler, 1975).
Revised: Bayes error estimation has been a topic of interest for nearly half a century (Fukunaga & Hostetler, 1975).

Original: Several works have derived upper and lower bounds of the Bayes error and proposed ways to estimate those bounds.
Revised: Several works have derived upper and lower bounds of the Bayes error and proposed ways to estimate those bounds.

Original: Various f -divergences, such as the Bhattacharyya distance (Fukunaga, 1990) or the Henze-Penrose divergence (Berisha et al., 2016; Sekeh et al., 2020) were studied.
Revised: Various f -divergences, such as the Bhattacharyya distance (Fukunaga, 1990) or the Henze-Penrose divergence (Berisha et al., 2016; Sekeh et al., 2020), were studied.

Original: Other approaches include Noshad et al.
Revised: Other approaches include Noshad et al.

Original: (Noshad et al., 2019) which directly estimated the Bayes error with f -divergence representation instead of using a bound, and Theisen et al.
Revised: (2019), which directly estimated the Bayes error with f -divergence representation instead of using a bound, and Theisen et al.

Original: (Theisen et al., 2021), which computed the Bayes error of generative models learned using normalizing flows (Kingma & 
Revised: (2021), which computed the Bayes error of generative models learned using normalizing flows (Kingma & Dhariwal, 2018).

Original: Dhariwal, 2018).
Revised: Michelucci et al. (2021) proposed a method to estimate the Bayes error and the best possible AUC for categorical features.

Original: More recently, Renggli et al.
Revised: More recently, Renggli et al.

Original: (Renggli et al., 2021) proposed a method to evaluate Bayes error estimators on real-world datasets, for which we usually do not have access to the true Bayes error.
Revised: (2021) proposed a method to evaluate Bayes error estimators on real-world datasets, for which we usually do not have access to the true Bayes error.

Original: All of these works that estimate the Bayes error are based on instance-label pairs.
Revised: All of these works that estimate the Bayes error are based on instance-label pairs.

Original: Not only do they require instances, but they also usually have a model, which may need careful tuning of hyperparameters to avoid overfitting.
Revised: Not only do they require instances, but they also usually have a model, which may need careful tuning of hyperparameters to avoid overfitting.

Original: Li et al.
Revised: Li et al.

Original: (Li et al., 2021) proposed to estimate the maximal performance for regression problems, while we focus on the classification problem.
Revised: (2021) proposed to estimate the maximal performance for regression problems, while we focus on the classification problem.

Original: Weakly supervised learning (Zhou, 2018) tries to learn with a weaker type of supervision, e.g., semi-supervised learning (Chapelle et al., 2006;
Revised: Weakly supervised learning (Zhou, 2018) tries to learn with a weaker type of supervision, e.g., semi-supervised learning (Chapelle et al., 2006;

Original: Sakai et al., 2017), positive-unlabeled learning (du Plessis et al., 2014), similar-unlabeled learning (Bao et al., 2018), and unlabeled-unlabeled learning (Lu et al., 2019).
Revised: Sakai et al., 2017), positive-unlabeled learning (du Plessis et al., 2014), similar-unlabeled learning (Bao et al., 2018), and unlabeled-unlabeled learning (Lu et al., 2019).

Original: On the other hand, the focus of most previous works on estimating the Bayes error is on ordinary classification problems with full supervision.
Revised: On the other hand, the focus of most previous works on estimating the Bayes error is on ordinary classification problems with full supervision.

Original: To our knowledge, Bayes error estimation for weakly supervised data has not been studied.
Revised: To our knowledge, Bayes error estimation for weakly supervised data has not been studied.

Original: We discussed how our formulation introduced in this paper is flexible and can be extended to weakly supervised learning in Sec. 4.
Revised: We discussed how our formulation introduced in this paper is flexible and can be extended to weakly supervised learning in Sec. 4.

Original: Overfitting to the test dataset is one of the issues that concerns the machine learning community (Mania et al., 2019; Werpachowski et al., 2019; Kiela et al., 2021).
Revised: Overfitting to the test dataset is one of the issues that concerns the machine learning community (Mania et al., 2019; Werpachowski et al., 2019; Kiela et al., 2021).

Original: Models can overfit to the test dataset by training on the test set, explicitly or implicitly .
Revised: Models can overfit to the test dataset by training on the test set, explicitly or implicitly .

Original: Barzet al. (Barz & Denzler, 2020) reported that 3% of the test data in CIFAR-10 has duplicates and near-duplicates in the training set.
Revised: Barz & Denzler (2020) reported that 3% of the test data in CIFAR-10 has duplicates and near-duplicates in the training set.

Original: Lewis et al.
Revised: Lewis et al.

Original: (Lewis et al., 2020) studied popular open-domain question answering datasets and reported that 30% of test questions have a near-duplicate in the training data.
Revised: (2020) studied popular open-domain question answering datasets and reported that 30% of test questions have a near-duplicate in the training data.

Original: These are examples of when we may  showed how removing the duplicates will lead to substantially worse results.
Revised: These are examples of when we may explicitly train on the test set. Barz & Denzler (2020) and Lewis et al. (2020) showed how removing the duplicates will lead to substantially worse results.

Original: Overfitting to the test set can also happen implicitly because research is conducted on the same test data that is public for many years (often with a fixed train/test partition), in contrast to how engineers in industry can sample fresh unseen data for evaluation.
Revised: Overfitting to the test set can also happen implicitly because research is conducted on the same test data that is public for many years (often with a fixed train/test partition), in contrast to how engineers in industry can sample fresh unseen data for evaluation.

Original: Recht et al.
Revised: Recht et al.

Original: (Recht et al., 2018) constructed CIFAR-10.1, an alternative unseen test set for CIFAR-10 classifiers and showed how deep learning models became worse with respect to the test error, with a 4% to 10% increase.
Revised: (2018) constructed CIFAR-10.1, an alternative unseen test set for CIFAR-10 classifiers and showed how deep learning models became worse with respect to the test error, with a 4% to 10% increase.

Original: Yadavet al. (Yadav & Bottou, 2019) and Recht et al.
Revised: Yadav & Bottou (2019) and Recht et al.

Original: (Recht et al., 2019) showed similar results for MNIST (LeCun et al., 1998) and ImageNet (Deng et al., 2009), respectively.
Revised: (2019) showed similar results for MNIST (LeCun et al., 1998) and ImageNet (Deng et al., 2009), respectively.

Original: With closer inspection, however, Recht et al.
Revised: With closer inspection, however, Recht et al.

Original: (Recht et al., 2019) discussed how the accuracy improvement proportional to that in the original test set was observed in the new test set, for both CIFAR-10 and ImageNet.
Revised: (2019) discussed how the accuracy improvement proportional to that in the original test set was observed in the new test set, for both CIFAR-10 and ImageNet.

Original: Roelofs et al.
Revised: Roelofs et al.

Original: (Roelofs et al., 2019) studied Kaggle competition datasets by comparing accuracies based on a public test set (which is used repeatedly by practitioners) and a private test set (which is used to determine the final rankings), but found little evidence of overfitting.
Revised: (2019) studied Kaggle competition datasets by comparing accuracies based on a public test set (which is used repeatedly by practitioners) and a private test set (which is used to determine the final rankings), but found little evidence of overfitting.

Original: Constructing new test datasets is one way to examine the degree of overfitting to the test dataset.
Revised: Constructing new test datasets is one way to examine the degree of overfitting to the test dataset.

Original: In
Revised: In

Original: Sec. 5, we discussed how estimating the Bayes error can be another way to detect overfitting to the test dataset.
Revised: Sec. 5, we discussed how estimating the Bayes error can be another way to detect overfitting to the test dataset.

Original: If models have lower test error than the estimated Bayes error, that is a strong signal for this type of overfitting.
Revised: If models have lower test error than the estimated Bayes error, that is a strong signal for this type of overfitting.

Original: We compared the test error of recent classifiers with the estimated Bayes error in Sec. 5.
Revised: We compared the test error of recent classifiers with the estimated Bayes error in Sec. 5.

Original: B P ROOFS B. P ROOF OF P ROPOSITION 3. 
Revised: B P ROOFS B. P ROOF OF P ROPOSITION 3.

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: B. P ROOF OF P ROPOSITION 3. 
Revised: B. P ROOF OF P ROPOSITION 3.

Original: Proof.
Revised: Proof.

Original: Since 0 ≤ min[ p ( y = +1 | x ) , p ( y = − 1 | x )] ≤ 0 . 5 , we can use the Hoeffding’s inequality as follows:
Revised: Since 0 ≤ min[ p ( y = +1 | x ) , p ( y = − 1 | x )] ≤ 0 . 5 , we can use the Hoeffding’s inequality as follows:

Original: [Equation]
Revised: [Equation]

Original: Therefore, for any δ > 0 , with probability at least 1 − δ , we have,
Revised: Therefore, for any δ > 0 , with probability at least 1 − δ , we have,

Original: [Equation]
Revised: [Equation]

Original: B. P ROOF THAT ˜ β IS A BIASED ESTIMATOR
Revised: B. P ROOF THAT ˜ β IS A BIASED ESTIMATOR

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: B. P ROOF OF P ROPOSITION 3. 
Revised: B. P ROOF OF P ROPOSITION 3.

Original: For simplicity, suppose that we have m labels for every sample i , and u i is the sample average of those labels:
Revised: For simplicity, suppose that we have m labels for every sample i , and u i is the sample average of those labels:

Original: [Equation]
Revised: [Equation]

Original: Since 1[ y i = 1] ∈ [0 , 1] , using Hoeffding’s inequality, we get
Revised: Since 1[ y i = 1] ∈ [0 , 1] , using Hoeffding’s inequality, we get

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for any ϵ > 0 .
Revised: for any ϵ > 0 .

Original: In other words, for any δ ′ > 0 , with probability at least 1 − δ ′ , we have
Revised: In other words, for any δ ′ > 0 , with probability at least 1 − δ ′ , we have

Original: [Equation]
Revised: [Equation]

Original: 27) for all i = 1 , . .
Revised: 27) for all i = 1 , . .

Original: . , n .
Revised: . , n .

Original: Note that
Revised: Note that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The conditional bias can be bounded as
Revised: The conditional bias can be bounded as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the last line follows from Eq.
Revised: where the last line follows from Eq.

Original: Eq.
Revised: Eq.

Original: 37 holds because min { c i , 1 − c i } ∈ [0 , 1 / 2] , min { u i , 1 − u i } ∈ [0 , 1 / 2] , and | u i − c
Revised: 37 holds because min { c i , 1 − c i } ∈ [0 , 1 / 2] , min { u i , 1 − u i } ∈ [0 , 1 / 2] , and | u i − c

Original: i | < ϵ ′ ( δ ′ ) implies that u i >
Revised: i | < ϵ ′ ( δ ′ ) implies that u i >

Original: c
Revised: c

Original: i − ϵ ′
Revised: i − ϵ ′

Original: ( δ ′ ) and 1 − u i > 1 − c
Revised: ( δ ′ ) and 1 − u i > 1 − c

Original: i − ϵ ′
Revised: i − ϵ ′

Original: ( δ ′ ) .
Revised: ( δ ′ ) .

Original: Further taking the expectation on both sides, we get a bound on the bias:
Revised: Further taking the expectation on both sides, we get a bound on the bias:

Original: [Equation]
Revised: [Equation]

Original: Setting δ ′ to m − 1 / 2 yields
Revised: Setting δ ′ to m − 1 / 2 yields

Original: [Equation]
Revised: [Equation]

Original: B. P ROOF OF P ROPOSITION 3. 
Revised: B. P ROOF OF P ROPOSITION 3.

Original: Proof.
Revised: Proof.

Original: ˆ β Noise can be expressed as
Revised: ˆ β Noise can be expressed as

Original: [Equation]
Revised: [Equation]

Original: Recall that u i =
Revised: Recall that u i =

Original: c i + ξ i .
Revised: c i + ξ i .

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: The second term in Eq.
Revised: The second term in Eq.

Original: 45 is,
Revised: 45 is,

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: the fourth term in Eq.
Revised: the fourth term in Eq.

Original: 45 becomes zero
Revised: 45 becomes zero

Original: [Equation]
Revised: [Equation]

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: B. P ROOF OF P ROPOSITION 3. 
Revised: B. P ROOF OF P ROPOSITION 3.

Original: Proof.
Revised: Proof.

Original: ˆ β Noise can be expressed as
Revised: ˆ β Noise can be expressed as

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Since we can derive that 0 ≤ ˇ b i ≤ 1 , we can again use the Hoeffding’s inequality:
Revised: Since we can derive that 0 ≤ ˇ b i ≤ 1 , we can again use the Hoeffding’s inequality:

Original: [Equation]
Revised: [Equation]

Original: Therefore, for any δ > 0 , with probability at least 1 − δ , we have,
Revised: Therefore, for any δ > 0 , with probability at least 1 − δ , we have,

Original: [Equation]
Revised: [Equation]

Original: B. P ROOF OF T HEOREM 4. 
Revised: B. P ROOF OF T HEOREM 4.

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:27)(cid:21)(cid:19) (65)
Revised: (cid:27)(cid:21)(cid:19) (65)

Original: B. P ROOF OF P ROPOSITION 4. 
Revised: B. P ROOF OF P ROPOSITION 4.

Original: Proof.
Revised: Proof.

Original: E
Revised: E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: max
Revised: max

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: ( x i ) (cid:27)(cid:21) (66) =
Revised: ( x i ) (cid:27)(cid:21) (66) =

Original: [Equation]
Revised: [Equation]

Original: + (cid:18)
Revised: + (cid:18)

Original: [Equation]
Revised: [Equation]

Original: E + (cid:20) max (cid:26) 0 , 2 − r ( x ) (cid:27)(cid:21)(cid:19) (67) =
Revised: E + (cid:20) max (cid:26) 0 , 2 − r ( x ) (cid:27)(cid:21)(cid:19) (67) =

Original: [Equation]
Revised: [Equation]

Original: (cid:18)
Revised: (cid:18)

Original: [Equation]
Revised: [Equation]

Original: E + (cid:20) min (cid:26)
Revised: E + (cid:20) min (cid:26)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( x
Revised: ( x

Original: [Equation]
Revised: [Equation]

Original: (cid:27)(cid:21)(cid:19) (68) =
Revised: (cid:27)(cid:21)(cid:19) (68) =

Original: [Equation]
Revised: [Equation]

Original: (cid:18) 1 + (cid:18)
Revised: (cid:18) 1 + (cid:18)

Original: [Equation]
Revised: [Equation]

Original: (cid:19)(cid:19) (69) =
Revised: (cid:19)(cid:19) (69) =

Original: β (70)
Revised: β (70)

Original: In the third equality, we used max { a, b } = − min {− a, − b } .
Revised: In the third equality, we used max { a, b } = − min {− a, − b } .

Original: In the fourth equality, we used
Revised: In the fourth equality, we used

Original: E + (cid:104) min (cid:110) 0 , 1 r ( x ) − 2 (cid:111)(cid:105) =
Revised: E + (cid:104) min (cid:110) 0 , 1 r ( x ) − 2 (cid:111)(cid:105) =

Original: β π + − 1 which can be derived from Eq.
Revised: β π + − 1 which can be derived from Eq.

Original: B. P ROOF OF P ROPOSITION 4. 
Revised: B. P ROOF OF P ROPOSITION 4.

Original: We can rewrite ˆ β Pconf as
Revised: We can rewrite ˆ β Pconf as

Original: [Equation]
Revised: [Equation]

Original: Although 1 /r ( x i ) is not bounded, since 0 ≤ π + − π + max (cid:16) 0 , 2 − 1 r
Revised: Although 1 /r ( x i ) is not bounded, since 0 ≤ π + − π + max (cid:16) 0 , 2 − 1 r

Original: [Equation]
Revised: [Equation]

Original: ≤ π + , we can use the
Revised: ≤ π + , we can use the

Original: Hoeffding’s inequality to derive,
Revised: Hoeffding’s inequality to derive,

Original: [Equation]
Revised: [Equation]

Original: Therefore, for any δ > 0 , with probability at least 1 − δ we have,
Revised: Therefore, for any δ > 0 , with probability at least 1 − δ we have,

Original: [Equation]
Revised: [Equation]

Original: C S TOCHASTIC BIG -O NOTATION
Revised: C S TOCHASTIC BIG -O NOTATION

Original: Suppose { X n } ( n = 1 , 2 , . . . )
Revised: Suppose { X n } ( n = 1 , 2 , . . . )

Original: is a sequence of random variables and { a n } ( n = 1 , 2 , . . . )
Revised: is a sequence of random variables and { a n } ( n = 1 , 2 , . . . )

Original: is a sequence of positive numbers, respectively.
Revised: is a sequence of positive numbers, respectively.

Original: We say X n =
Revised: We say X n =

Original: O p ( a n ) when for all ϵ > 0 , there exists
Revised: O p ( a n ) when for all ϵ > 0 , there exists

Original: C ∈ R and finite N such that for all n > N , P ( | X n | a n > C )
Revised: C ∈ R and finite N such that for all n > N , P ( | X n | a n > C )

Original: <
Revised: <

Original: ϵ .
Revised: ϵ .