L ABEL C LEANING WITH A L IKELIHOOD R ATIO T EST
Anonymous authors
Paper under double-blind review
A BSTRACT
To collect large scale annotated data, it is inevitable to introduce label noise, i.e., incorrect class labels.
A major challenge is to develop robust deep learning models that achieve high test performance despite training set label noise.
We introduce a novel approach that directly cleans labels in order to train a high quality model.
Our method leverages statistical principles to correct data labels and has a theoretical guarantee of the correctness.
In particular, we use a likelihood ratio test (LRT) to ﬂip the labels of training data.
We prove that our LRT label correction algorithm is guaranteed to ﬂip the label so it is consistent with the true Bayesian optimal classiﬁer with high probability.
We incorporate our label correction algorithm into the training of deep neural networks and train models that achieve superior testing performance on multiple public datasets.
I NTRODUCTION
Label noise is ubiquitous in real world data.
It may be caused by unintentional mistakes of manual or automatic annotators (Yan et al., 2014; Veit et al., 2017).
It may also be introduced by malicious attackers (Steinhardt et al., 2017).
Noisy labels impair the performance of a model (Smyth et al., 1994; Brodley & Friedl, 1999), especially a deep neural network, which tends to have strong memorization power (Frnay & Verleysen, 2014; Zhang et al., 2017).
Improving the robustness of a model to label noise is a crucial yet challenging task in many applications (Mnih & Hinton, 2012; Wu et al., 2018).
Existing methods mainly follow two directions, probabilistic reasoning and data selecting.
Probabilistic methods explicitly model a noise transition matrix , namely, the probability of one label being corrupted into another (Goldberger & Ben-Reuven, 2017; Patrini et al., 2017).
The transition matrix is often estimated from the data, and is used to re-calibrate the training loss or to correct the prediction.
Explicit estimation of the transition matrix can be problematic due to the large variation of noise patterns, e.g., uniform noise, asymmetric noise, or mixtures.
Furthermore, the transition matrix size is quadratic to the number of classes, making the estimation task prohibitive when the data has hundreds or even thousands of classes.
Data-selecting methods are agnostic of the underlying noise pattern.
These methods gradually collect clean data whose labels are trustworthy (Malach & Shalev-Shwartz, 2017; Jiang et al., 2018;
Han et al., 2018).
As more clean data are collected, the quality of the trained models improves.
The major issue of these methods is the lack of a quantitative control of the quality of the collected clean data.
Without a principled guideline, it is hard to ﬁnd the correct data collection pace.
An aggressive selection can unknowingly accumulate irreversible errors.
On the other hand, an overly-conservative strategy can be very slow in training, or stops with insufﬁcient clean data and mediocre models.
We propose a novel method with the beneﬁt from both the probabilistic and the data-selecting approaches.
Similar to data-selecting methods, our method continuously improves the purity of the data labels by correcting the noise-corrupted ones.
Meanwhile, we improve the classiﬁer using the updated labels.
Our label correction algorithm is based on statistical principles and is theoretically guaranteed to deliver a high quality label set.
Instead of explicitly estimating the transition matrix, the correction algorithm only depends on the prediction of the current model, denoted as f .
Using an f -based likelihood ratio test, we determine whether the current label of each data should be corrected.
Our main theorem proves that the label correction algorithm will clean a majority of noisy labels with high probability.
In practice, we incorporate the label correction algorithm into the training of deep neural networks.
Our method iteratively updates the labels of the data while continuously training a deep neural network.
To ensure the deep neural network does not overﬁt with noise labels that are yet to be corrected, we introduce a new retroactive loss term that regulates the model by enforcing its consistency with models in previous epochs.
The rationale is that the model in an earlier training stage tends to ﬁt the true signal rather than noise, although its overall performance is sub-optimal.
Through experiments on various datasets with various noise patterns and levels, we show that our method produces robust neural network models with superior performance.
To the best of our knowledge, our method is the ﬁrst to correct labels with theoretical guarantees .
It is has advantages over both probabilistic methods and data-selecting methods.
Compared with other data-selecting methods, it has a better quantitative control of the label quality and thus is less brittle when generalizing to different datasets and different noise patterns.
Also note that we are not selecting clean data.
Instead, we correct labels and always use the whole training set to train.
This brings an additional advantage of fully leveraging the data.
Compared with other probabilistic methods, our correction algorithm assumes a rather general family of underlying noise patterns and avoids an explicit estimation of the transition matrix.
1. R ELATED W ORK
One representative strategy for handling label noise is to model and employ noise transition matrix to correct the loss.
For example, Patrini et al.
(2017) propose to correct the loss function with estimated noise pattern.
The resulting loss is an unbiased estimator of the ground truth loss, and enables the trained model to achieve better performance.
However, such an estimator relies on strong assumptions and could be inaccurate in certain scenarios.
Reed et al.
(2014) consider modeling the noise pattern with a hidden layer.
The learning of this hidden layer is regularized with a feature reconstruction loss, yet without a guarantee that the true label distribution is learned.
Another method mentioned in their work is to minimize the entropy of neural network output; however, this method tends to predict a single class.
To address this weakness, Hendrycks et al.
(2019) propose to utilize a small number of kosher data to pre-train a network and estimate the noise pattern.
However, such clean data may not always be available in practice.
Alternatively, another direction proposes to design models that are intrinsically robust to noisy data.
Crammer et al.
(2009) introduce a regularized conﬁdence weighting learning algorithm (AROW), where parameters of classiﬁers are assumed normally distributed and the mean and covariance of this distribution is updated during training.
The idea here is to preserve the weight distribution as much as possible while requiring the model to maintain predictive ability.
The follow-up work (Crammer &
Lee 2010) proposes to improve this algorithm by herding the updating direction via speciﬁc velocity ﬁeld (NHERD), achieving better performance.
Both of these works impose parametric constraint on parameters, which could prevent classiﬁers from adapting to complex dataset.
Arpit et al.
(2017) show that deep neural networks tend to learn meaningful patterns before they over-ﬁt to noisy ones.
Based on this observation, they propose to add Gaussian or adversarial noise to input when training with noisy labels, and empirically show that such data perturbation is able to make the resulting model more robust.
Other commonly adopted techniques, such as weight decay and dropout, are also shown to be effective in increasing the robustness of trained classiﬁer (Arpit et al. 2017; Zhang et al. 2017).
However, the intrinsic reasons for this phenomenon still remains unclear and overﬁtting to noisy label is still inevitable.
Apart from the above mentioned strategies, one recent work proposes to correct the corrupted labels during training.
In particular, Tanaka et al.
(2018) propose to jointly train the deep network and estimate the underlying true labels.
While achieving improved performance, their method largely relies on the prior distribution and is difﬁcult to deploy under cases where there is a large number of classes.
In addition, there are a few other works attempting to select clean data while eliminating noisy ones during training (Malach & Shalev-Shwartz, 2017; Jiang et al., 2018; Han et al., 2018).
These methods demonstrate promising results, but lack quantitative control of the quality of the collected clean data.
Finally, beyond deep learning framework, there are several theoretic works that demonstrate the robustness of a variety of losses to label noise (Long & Servedio 2010; Natarajan et al. 2013; Ghosh et al. 2015; van Rooyen et al. 2015).
Following the work of (Wang & Chaudhuri 2018), Gao et al.
(2016) propose an algorithm that can converge to the Bayesian optimal classiﬁer under different noisy settings.
Moreover, they also provide in-depth discussion regarding the performance of k- nearest neighbor (KNN) classiﬁers.
However, the problem with KNN is that it is computationally intensive and difﬁcult to be incorporated into a learning context.
Within the framework of deep learning, there are more efforts that need to be made to bridge theory and practice.
M ETHOD
Our method has two synchronized modules, the training module and the label correction module.
The training module continues to learn a classiﬁer based on the current labels.
Meanwhile, the label correction module uses the prediction of the classiﬁer to correct labels.
[Figure]
We start with some preliminaries necessary for the exposition (Section 2.1).
In Section 2.2, we explain our correction algorithm.
It uses the prediction of the classiﬁer (trained on noisy labels) for a likelihood ratio test.
Based on the test result, it decides whether to correct the label of a data.
Theoretically, we prove that under certain assumptions of the prediction, f , the algorithm will change the labels to the correct ones, i.e., the ones consistent with the Bayes optimal classiﬁer (Theorem 1).
In Section 2.3, we present the overall training method.
We incrementally train a deep neural network based on the corrected labels, (cid:101) y .
Meanwhile, the network’s prediction is used for label correction.
To improve the quality of the prediction, we introduce a new loss term, called the retroactive loss .
The goal is to regulate the model using models trained in earlier epoches, as they may be less overﬁtting with corrupted labels.
For the label correction module, we focus on a binary classiﬁer.
But the algorithm and the theoretical results can easily be generalized to the multiclass setting (Corollary 1)
2. P RELIMINARIES
Let X be the feature space, Y = { 0 , 1 } be the label space, and D be an unknown distribution on X × Y .
The joint probability can be factored as D ( x , y ) =
Pr ( y | x ) Pr ( x ) .
We denote by η ( x ) =
Pr ( y = 1 | x ) the true conditional probability .
The Bayes risk of a classiﬁer h : X → Y is
R ( D, h ) =
Pr ( x ,y ) ∼ D ( h ( x ) (cid:54) =
y ) .
A Bayes optimal classiﬁer is the minimizer of the Bayes risk, i.e., h ∗ = arg min h R
( D, h ) .
It can be calculated using the true conditional probability, η ,
[Equation]
We assume the true conditional probability, η , satisﬁes the Tsybakov condition (also called the TNC condition)
(Tsybakov et al., 2004).
This condition, also called margin assumption, stipulates that the uncertainty of η (region close to decision boundary at 1 / 2 ) and thus the Bayes optimal classiﬁer is bounded.
Deﬁnition 1 (Tsybakov Condition) .
There exist C > 0 , λ > 0 , and t 0 ∈ (0 , 12 ] , such that for all t ≤ t 0 ,
[Equation]
The noisy label setting.
Instead of samples from D , we are given a sample set with noisy labels
S = { ( x , (cid:101) y ) } where (cid:101) y is the possibly corrupted label based on the true label y .
We assume a transition probability τ i
→ j = Pr ( (cid:101) y =
j | y
= i ) , i.e., the chance a ground truth label y is ﬂipped from class i to class j .
For simplicity, we denote τ ij = τ i → j .
The transition probabilities τ 01 and τ 10 are independent of the true joint distribution D and the feature x .
We denote the conditional probability of the noisy labels as (cid:101) η
( x ) =
Pr ( (cid:101) y
= 1 | x ) .
In short, we call (cid:101)
η the noisy conditional probability .
It is related linearly to the true conditional probability, η :
[Equation]
[Figure]
2. T HE L ABEL C ORRECTION A LGORITHM AND ITS T HEORETICAL G UARANTEES
Our label correction algorithm takes in a current neural network prediction f : X → [0 , 1] (i.e., an estimation of η based on the noisy labels).
For all training data and their current noisy label ( x , (cid:101) y ) , the correction algorithm uses f to run a likelihood ratio test and to decide whether to ﬂip the label according to the result.
The goal of the likelihood test is to decide whether the null hypothesis,
H 0 : (cid:101) y
=
y , is true.
If yes, (cid:101)
y is accepted as it is.
Otherwise, we ﬂip (cid:101) y , so that hopefully it becomes y .
Formally, the likelihood ratio is deﬁned as
[Equation]
We compare this likelihood ratio with a predetermined value ∆ .
If LR( f, x , y ) ≤ ∆ , we reject the null hypothesis and ﬂip the label (cid:101)
y new = 1 − (cid:101) y .
Otherwise, the label remains unchanged, (cid:101)
y new
= (cid:101) y .
See Figure 2 for an illustration of the algorithm.
Note that the constant ∆ depends on the underlying noise pattern and f .
Below we show that if we choose ∆ carefully, the label correction algorithm is guaranteed to make proper correction and clean most of the corrupted labels.
However, in practice, ∆ is unknown and needs to be tuned.
Intuition.
In the likelihood ratio (Eq.
, the numerator is the likelihood that the prediction f is consistent with the noisy label (cid:101) y .
The denominator is the likelihood of the opposite case.
When this ratio is smaller than 1 , we know that the prediction of f is more likely to be inconsistent with (cid:101) y .
But whether f agrees with (cid:101) y is not the hypothesis to test.
To test the intended null hypothesis ( (cid:101)
y
= y ), we need to check whether (cid:101) y is consistent with the true conditional distribution η , namely, the Bayes optimal classiﬁer prediction h ∗ ( x ) .
To this end, we assume f is a close enough approximate of (cid:101)
η as it is trained on the noisy labels.
This way, testing whether f agrees with (cid:101) y is close to testing whether (cid:101) η agrees with (cid:101) y , except that the threshold ∆ needs to be carefully chosen.
Another issue we need to consider is that the ∆ is unknown.
Our main theorem will bound the chance of failed correction by how close f approximates (cid:101) η and how close we can set ∆ to the perfect one.
2.2. F ORMAL S TATEMENT OF THE A LGORITHM AND THE T HEOREM
Assume the knowledge of some prediction function f (perhaps from a neural network).
The label correction algorithm is given in Procedure 1.
It checks the likelihood ratio with regard to a parameter ∆ =
min( 1+ τ 01 − τ 10 1+ τ 10 − τ 01 , 1+ τ 10 − τ 01 1+ τ 01 − τ 10 ) .
If LR is no greater than ∆ , we ﬂip the label.
Otherwise, (cid:101)
y new is the same as (cid:101) y .
Note that one only needs to know the difference τ 01 − τ 10 in order to know ∆ .
In practice, ∆ may be unknown and needs to be decided empirically.
Procedure
Input: ( x , (cid:101) y ) , f ( x ) , ∆ .
Output: (cid:101)
y new 1: if LR( f, x , (cid:101) y ) ≤ ∆ then 2: (cid:101)
y new = 1 − (cid:101) y 3: else 4: (cid:101) y new
= (cid:101) y 5: end if
Input: ( x , (cid:101) y ) , f ( x ) , ∆ .
Output: (cid:101)
y new y ∗ := arg max i [ f ( x )]
i ; LR( f, x , (cid:101) y ) :=
[ f ( x )] (cid:101)
y
[ f ( x )] y ∗ ; 1: if LR( f, x , (cid:101) y ) ≤ ∆ then 2: (cid:101)
y new =
y ∗ 3: else 4: (cid:101)
y new
= (cid:101) y 5: end if
Our main theorem states that suppose the classiﬁer prediction, f , is a close approximation of the noisy conditional probability, (cid:101) η .
And suppose we can ﬁnd a good enough ∆ (cid:48) that is close enough to the ideal ∆ .
Then there is a very good chance that our algorithm corrects most labels to the correct ones, i.e., the same as the Bayes optimal classiﬁer prediction.
Please note that here “proper correction” means that the new label, (cid:101)
y new , is the same as the Bayes optimal classiﬁer prediction, h ∗ ( x ) , instead of y .
This is well justiﬁed as it means that the correction will give us a classiﬁer as good as the Bayes optimal one.
Theorem 1.
Assume η satisﬁes the Tsybakov condition with constants C > 0 and λ > 0 .
Let h ∗ denote the Bayes optimal classiﬁer, and (cid:101)
η the noisy conditional probability (cid:101) η
( x ) = (1 − τ 01 − τ 10 )
η ( x ) + τ 01 .
Let f be any classiﬁer such that || f − (cid:101) η || ∞ ≤ (cid:15) for some (cid:15) > 0 .
Let ∆ =
min
( 1+ τ 01 − τ 10 1+ τ 10 − τ 01 , 1+ τ 10 − τ 01 1+ τ 01 − τ 10 ) , and let ∆ (cid:48) > 0 be a constant such that ∆ (cid:48) ∈ [∆ − (cid:15), ∆ + (cid:15) ] .
If (cid:101) y new denotes the output of the LRT-Correction with (cid:101)
y , x , f , and the given ∆ (cid:48) , then
[Equation]
λ .
If τ 01 = τ 10 , then Pr ( x,y ) ∼ D ( (cid:101)
y new (cid:54) =
h ∗ ) =
C (2 (cid:15) ) λ .
Intuition of the proof.
We will prove two lemmas.
The ﬁrst lemma shows that when f is linear in η , i.e., f ( x ) = aη ( x ) + b , with known coefﬁcients a, b , we can ﬁnd a ∆ such that then the correction algorithm can be correct everywhere.
The second lemma proves a more relaxed version.
If f is exactly (cid:101) η and if we only know the difference between the transition probabilities, τ 01 − τ 10 , then we can bound the chance of mistakes of the correction algorithm.
Finally, based on these two lemmas, careful case analysis, and the Tsybakov condition of the true conditional probability, η , we can prove the theorem.
The complete proof can be found in Appendix A.
Multiclass setting.
So far, all the description and theoretical results are based on a binary classiﬁcation setting.
However, the results can be generalized to a multiclass setting without any technical difﬁculties.
Let (cid:101)
y be the observed (possibly) corrupted label, f ( x ) be a classiﬁer vector, where [ f ( x )]
i (the i th coordinate of f ( x ) ) speciﬁes the conﬁdence of assigning x to class i .
The algorithm is presented in Procedure 2.
Informally, we present the theoretical guarantee for multiclass LRT as
Corollary 1.
The proof is left for a journal version.
Corollary 1.
Theorem 1 can be generalized to the multiclass setting, when using algorithm
LRT-MultiCorrection .
2. T RAINING D EEP N EURAL N ETWORKS WITH LRT-L ABEL -C ORRECTION
Our training algorithm continuously trains a deep neural network while correcting the noisy labels.
Procedure 3 is the pseudocode of the training method, called AdaCorr .
It trains a neural network model iteratively. Each iteration includes both label correction and model training steps. In label correction step, the prediction of the current neural network, f , is used to run LRT test on all training data, and to correct their labels according to the test result. Since f is used to approximate the conditional probability (cid:101) η , we use the softmax layer output of the neural network as f .
After the labels of all training data are updated, we use them to train the neural network incrementally.
We continue this iterative procedure until the whole training converges.
Procedure
Input: S = { x , (cid:101) y } , ∆ , m , T 1: for epoch=1 to m do 2:
Train neural network with L CE 3: end for 4: f (cid:48)
= current model prediction 5: for epoch= m + 1 to T do 6: if epoch ≥ m + 10 then 7: f = current model prediction 8: for all ( x , (cid:101) y ) ∈ S do 9: (cid:101)
y new =
LRT-Correction ( f , ( x , (cid:101) y ) , ∆ ) 10: (cid:101)
y
= (cid:101)
y new 11: end for 12: end if 13:
Train using L retro + L CE , with f (cid:48) and (cid:101) y 14: end for
We also have a burn-in stage in which we train the network using the original noisy labels for m epochs.
During the burn-in stage, we use the original cross-entropy loss, L CE .
Afterwards, we add an additional retroactive loss which will be explained below.
Training with retroactive loss.
After the burn-in stage, we want to avoid overﬁtting of the neural network, so that its output better approximates (cid:101) η .
To achieve this goal, we introduce a retroactive loss term L retro ( f ( x ) , (cid:101) y ) .
The idea is to enforce the consistency between f and the prediction of the model at a previous epoch, f (cid:48) .
It has been observed that a neural network at earlier training stage tends to learn the true pattern rather than to overﬁt the noise (Arpit et al., 2017).
Formally, the loss can be written as (cid:80) N c c =1 f (cid:48) c
( x ) log f c ( x ) , in which N c is the number of possible label classes.
The training loss is the sum of the retroactive loss and the cross-entropy loss
[Equation]
In practice, we set f (cid:48) to be the prediction of the model at the m -th epoch.
In other words, once the burn-in stage is ﬁnished, the training switches from L CE to L CE + L retro .
And the model at the end of the burn-in stage is used for the retroactive loss.
We also set the label correction to start slightly after the burn-in stage, say m + 10 .
The key hyperparameter is the starting epoch m .
Another hyperparameter is ∆ .
We select both m and ∆ empirically.
Ablation study in Section 4 shows that our method is robust to these hyperparameters.
E XPERIMENTS
In this section we empirically evaluate our proposed method with several datasets, where noisy labels are injected according to speciﬁed noise transition matrices.
Datasets.
We use the following datasets: MNIST (LeCun & Cortes 2010), CIFAR10 (Krizhevsky et al.
a), CIFAR100 (Krizhevsky et al.
b) and ModelNet40 (Z. Wu & Xiao 2015).
MNIST consists of 28 × 28 grayscale images with 10 categories.
It contains 60,000 images, and we use 45,000 for training, 5,000 for validation and 10,000 for testing.
CIFAR10 and CIFAR100 share the same 60, 32 × 32 × 3 image data, with CIFAR10 having 10 categories while CIFAR100 having 100 categories.
Similar to MNIST, we split 90% and 10% data from the ofﬁcial training set for the training and validation respectively, and use the ofﬁcial test set for testing.
ModelNet40 contains 12,311 CAD models from 40 categories, where 8,859 are used for training, 984 for validation and the remaining 2,468 for testing.
We follow the protocol of (Qi et al.
(2017)) to convert the CAD models into point clouds by uniformly sampling 1,024 points from the triangular mesh and normalizing them within a unit ball.
In all experiments, we use early stopping on validation set to tune hyperparameters and to report the performance on test set.
Noise patterns.
Following (Reed et al. 2014; Patrini et al. 2017), we corrupt our labels artiﬁcially using a noise transition matrix T , where T ij = τ ij =
Pr ( (cid:101) y =
j | y
= i ) is the probability that category i is ﬂipped to category j .
In our work we focus on two types of T : (1) uniform, where the true label i is ﬂipped to other classes with equal probabilities, i.e., T ij =
p/ ( N c − 1) for i (cid:54) =
j and T ii = 1 − p , where p is the noise level and N c is the class number; (2) pair ﬂipping, where the true label i is ﬂipped to j with T ij = p for i (cid:54) =
j and T ii = 1 − p .
Examples of these two types of transition matrices are as follows:
[Equation]
[Equation]
[Equation]
[Equation] [Equation] [Equation]   in which T 1 is for uniform noise with level 0.3, and T 2 is for pair ﬂipping with noise level 0.3.
Baselines.
We compare the proposed method with the following methods: (1) Standard, which trains the network in a standard manner, without any label resistance technique; (2) Forward correction (Patrini et al. 2017), which explicitly estimates the noise transition matrix to correct the training loss; (3) Decoupling (Malach & Shalev-Shwartz 2017), which trains two networks simultaneously and updates the parameters on selected data whose labels are possibly clean; (4) Coteaching (Han et al.
2018), which also trains two networks but exchanges their error information for network update;
(5) MentorNet (Jiang et al. 2018), which learns a curriculum to ﬁlter out noisy data; (6) Forgetting (Arpit et al., 2017), which uses dropout to help deep models resist label noise.
Experimental Setup.
For the classiﬁcation of MNIST, CIFAR10 and CIFAR100, we use preactive resnet34 (He et al. 2016) as the backbone for all the methods.
On ModelNet40, we use PointNet (Qi et al. 2017).
We train the models for 180 epochs to ensure that all the methods have converged.
We utilize RAdam (Liu et al. 2019) for the network optimization, and adopt batch size 128 for all the datasets.
We use an initial learning rate of 0.001, which is decayed by 0.5 very 60 epochs.
The experimental results are listed in Table 1.
As is shown, our method outperforms the competing methods across the datasets under different noise settings
[Table]
[Table]
Clothing 1M.
We also evaluate our method on a large scale Clothing 1M dataset (Xiao et al., 2015).
We use pretrained resnet-50 and trained the model using SGD for 20 epochs.
Our method has 71. accuracy.
It outperforms Standard (68.94) and Forward (69.84).
Other baselines (Forget, Decouple,
MentorNet, Coteach) did not report result on this dataset.
A BLATION S TUDY AND D ISCUSSION
We conduct ablation study to see the signiﬁcance of our contributions.
We compare our method (LRT + L ce + L retro ) with two baselines: our method without the retroactive loss (LRT+ L ce ) and using cross-entropy loss only without LRT-Correction ( L ce Only).
We report the test accuracy on
CIFAR10 in Table 2.
We observe that adding LRT Label Correction to L ce alone helps improve the performance signiﬁcantly.
The numbers inside the parenthesis are the percentage of correct labels after the training and label correction.
We observe that the LRT Correction corrected a large portion of noise labels.
Also we observe adding the retroactive loss improves the method further in both test accuracy and label correction rates.
[Table]
We evaluate how different hyperparameters affect the performance of our method.
We compare our method with different m , the length of the burn-in stage.
We start introducing the retroactive loss after m epochs, and start label corrections after m + 10 epochs.
The ﬁnal testing accuracies are shown in Table 3.
We observe the performance of our method is rather robust w.r.t.
different m ’s.
We choose m = 20 in this data set (CIFAR10) and similarly in other datasets.
[Table]
We also evaluate the performance on different ∆ ’s. ∆ is the unknown value for our likelihood ratio test.
It controls how aggressive we would like to correct the labels.
From Table
[Table]
we observe a bigger ∆ tends to give better results (as it is less aggressive in correcting labels).
We observe 1/1.
is the best one for CIFAR10.
Similar values of optimal ∆ are found in other data sets.
[Table]
In general, we observe our hyperparameters are rather consistent across different datasets.
This reveals a better generalization power of our method over other datasets and noise patterns.
We believe this is due to the principled approach we take in label cleaning.
In Table 5 we show the performance of different methods trained on clean dataset.
We observe that when without label noise, all the methods achieve very similar performance.
However, note that on
CIFAR10, CIFAR100 and ModelNet40, the performances of MentorNet and Forward are slightly inferior to the standard approach.
The reason could be that MentorNet is designed speciﬁcally for handling noisy data, and would behave too conservatively under clean data setting; for Forward, although the data is clean, the estimated noise transition matrix is imperfect and still indicates a certain level of noise, leading to inferior performance. AdaCorr remains competitive in these experiments.
[Table]
In Figure 3, we draw convergence curves on CIFAR10 with Uniform 0.4 noise level.
On the left, we show the curves of our proposed AdaCorr method. We show the accuracy on the test set (compared with clean label).
On the training set, we show the accuracy evaluated against the corrected label y new and against the original noisy label (cid:101) y . We notice a clear bifurcation of the two training accuracy curves at the epoch 25, when LRT-based label ﬂipping starts. Afterwards, the training accuracy against noisy label ( (cid:101) y ) continues increasing but then decreased.
Meanwhile, the training accuracy against corrected label ( y new ) continues to increase. This shows that the model and the label correction are improving in a harmonic fashion and do not collapse.
On the right, we show the curves of the Standard model. Without label correction, the model overﬁts with the noisy label and loses on test data catastrophically.  C ONCLUSION
We propose a label correction algorithm to combat label noise.
The correction algorithm performs a likelihood ratio test for each input label to decide whether to correct the label.
Theoretically, we prove that our method corrects noisy labels with high probability.
By combining with deep neural network training with a novel retroactive loss, our method can produce models robust to different noise patterns.
Experiments on various datasets show that our method outperforms state-of-the-arts and is robust to hyperparameters.
A P ROOF OF T HEOREM
We restate the theorem below for completeness.
Theorem 1 Assume η satisﬁes the Tsybakov condition with constants C > 0 and λ > 0 .
Let h ∗ denote the Bayes optimal classiﬁer and (cid:101)
η the noisy conditional probability (cid:101) η
( x ) = (1 − τ 01 − τ 10 )
η ( x ) + τ 01 .
Let f be any classiﬁer such that || f − (cid:101) η || ∞ ≤ (cid:15) for some (cid:15) > 0 .
Let ∆ =
min
( 1+ τ 01 − τ 10 1+ τ 10 − τ 01 , 1+ τ 10 − τ 01 1+ τ 01 − τ 10 ) , and let ∆ (cid:48) > 0 be a constant such that ∆ (cid:48) ∈ [∆ − (cid:15), ∆ + (cid:15) ] .
If (cid:101) y new denotes the output of the LRT-Correction with (cid:101)
y , x , f , and the given ∆ (cid:48) , then
[Equation] ∼ D ( (cid:101)
y new (cid:54) =
h ∗ ) ≤ C (cid:18)(cid:12)(cid:12)(cid:12)(cid:12) τ 10 − τ2(1 − τ 10 − τ 01 ) (cid:12)(cid:12)(cid:12)(cid:12) + 2 (cid:15) (cid:19) λ .
If τ 01 = τ 10 , then Pr ( x,y ) ∼ D ( (cid:101)
y new (cid:54) =
h ∗ ) =
C (2 (cid:15) ) λ .
To prove this theorem, we ﬁrst prove two lemmas.
In the following we will state and prove the lemmas, and then continue to ﬁnish the proof of Theorem 1.
Lemma 2.
Assume η satisﬁes the Tsybakov condition with constants C > 0 and λ > 0 .
Let h ∗ denote the Bayes classiﬁer.
If f depends linearly on η , i.e., f ( x ) = aη ( x ) + b , a, b > 0 , and a, b and u are known, and ∆ is chosen as in Table 6, then
• If LRT-Correction ( f , ( x , (cid:101) y ) , a , b ,) ﬂips the label of (cid:101)
y , then (cid:101)
y new =
h ∗ ( x ) , and • Pr ( x ,y ) ∼ D ( (cid:101)
y new ( x, (cid:101) y )
(cid:54) =
h ∗ ( x ))
= 0 .
[Table]
Lemma 3.
Assume η satisﬁes the Tsybakov condition with constants C > 0 and λ > 0 .
Let h ∗ denote the Bayes classiﬁer.
Let (cid:101) η
( x ) = (1 − τ 01 − τ 10 )
η ( x ) + τ 01 denote the corrupted Bayesian classiﬁer.
If ∆ = min( 1+ τ 01 − τ 10 1+ τ 10 − τ 01 , 1+ τ 10 − τ 01 1+ τ 01 − τ 10 ) , then
• If LRT-Correction ( f , ( x , (cid:101) y ) , ∆ ) ﬂips the label of (cid:101)
y , then (cid:101)
y new =
h ∗ ( x ) , • Pr ( x ,y ) ∼ D ( (cid:101)
y new ( x, (cid:101) y )
(cid:54) =
h ∗ ( x )) ≤ C (cid:16) | τ 01 − τ 10 2(1 − τ 10 − τ 10) | (cid:17) λ , and • If τ 01 =
τ 10 , Pr ( x ,y ) ∼ D ( (cid:101)
y new ( x, (cid:101) y )
(cid:54) =
h ∗ ( x ))
= 0 .
Proof of Lemma 2:
We begin with some observations:
Observation 1: If (cid:101) y = 1 ( f > b + a/ 2) , the algorithm never ﬂips.
This is because ∆ = 0 in all such cases, and since the likelihood ratios f/ (1 − f ) or (1 − f ) /f are always positive, (cid:101) y is not ﬂipped.
Observation 2: f ( x ) < b + a/ 2 if, and only if, η ( x ) < 1 / 2 .
This follows by a straightforward calculation using f ( x ) =
aη ( x ) + b .
In the remainder of this proof, we assume that τ 10 > τ 01 .
The case τ 10 < τ 01 follows by symmetry.
Observation 3: If (cid:101) y ( x ) = 1 and f ( x ) < b + a/ 2 (so a non-trivial test (unlike ones in Observation 1) is performed), then f ( x ) / (1 − f ( x ) <
∆( (cid:101) y
= 1 , f ( x ) <
b + a/ 2) if, and only if, η ( x ) < 1 / 2 .
Similarly if (cid:101) y ( x ) = 0 and f ( x ) > b + a/ 2 then (1 − f ( x )) /f
( x ) < ∆( (cid:101)
y
= 0 , f ( x ) > b + a/ 2) if, and only if, η ( x ) > 1 / 2 .
Proof of Observation 3: First notice from the table that ∆( (cid:101) y
=
1 , f ( x ) < 1 / 2) is the reciprocal of ∆( (cid:101)
y
= 0 , f ( x ) > 1 / 2) .
Since the likelihood test ratio is also the reciprocal in the two situations, proving one statement sufﬁces.
We prove the ﬁrst statement.
For convenience we denote ∆( (cid:101) y
= 1 , f ( x ) < 1 / 2) by ∆ .
[Equation]
We can now prove the ﬁrst assertion of Lemma 2 (under the assumption that τ 10 > τ 01 ).
That is, if LRT-Correction ( f , ( x , (cid:101) y ) , a , b ) ﬂips the label of (cid:101)
y , then (cid:101)
y new =
h ∗ ( x ) .
Let us ﬁrst consider the case (cid:101)
y
( x ) = 1 .
If f ( x ) > b + a/ 2 , then it implies that η ( x ) > 1 / 2 (Observation 2).
The corresponding ∆ in the table is 0 , meaning that no test is performed, and (cid:101)
y new ( x , (cid:101) y )
= (cid:101) y
( x ) =
h ∗ ( x ) .
On the other hand, if f ( x ) < b + a/ 2 , we perform a likelihood ratio test on f/ (1 − f ) , which is less than the corresponding value of ∆ iff η ( x ) < 1 / 2 (Observation 3), so again (cid:101)
y new ( x , (cid:101) y )
= (cid:101) y
( x ) =
h ∗ ( x ) .
The case (cid:101) y
( x ) = 0 can be analyzed analogously.
Thus the only points x where we may not match h ∗ ( x ) are ones where the ﬂipping algorithms does not ﬂip, and the result mismatches with h .
By the above observations, the following is where this happens.
Observation 4: Assume τ 10 > τ 01 .
Then
[Equation]
However, according to Observation 2, f ( x ) < b + a/ 2 implies η ( x ) < 1 / 2 .
Thus this case cannot happen.
Hence Pr ( x ,y ) ∼ D ( (cid:101)
y new ( x, (cid:101) y )
(cid:54) =
h ∗ ( x ))
= 0 , and Lemma 2 is proved.
Proof of Lemma
This proof is an extension of the proof of Lemma 2.
All the steps are analogous, except in Observation 4, we get the following
Observation 5: Assume τ 10 > τ 01 .
Then { x : (cid:101) y new ( x , (cid:101) [Equation]
[Equation]
This measure is non zero.
However, it is bounded above by the measure of the set { (cid:101) η ( x ) < 1 / 2 , η ( x ) > 1 / 2 } .
Since (cid:101) η
( x ) = (1 − τ 01 − τ 10 )
η ( x ) + τ 01 , this is the same as the measure of the set { x : 1 / 2 < η ( x ) < 1 / 2 − τ 01 1 − τ 10 − τ 01 .
By the Tsybakov condition, this measure is at most (cid:16)
[Equation]
[Equation]
[Equation]
[Equation]
01 ) (cid:17) λ .
This was for the case τ 10 > τ 01 , and when τ 10 < τ 01 we can check by analogous calculation that the numerator is reversed.
Hence we have proved
[Equation]
The last assertion of Lemma 3 follows by just substituting τ
=
τ 10 in the above formula.
Hence
Lemma 3 is proved.
Proof of Theorem
Let A denote the event (cid:16) f 1 − f < ∆ (cid:48) (cid:17) , and B denote the event (cid:16) 1 − ff < ∆ (cid:48) (cid:17) .
In the following we describe the cases when our algorithm results (ﬂips or does not ﬂip) in a label inconsistent with h ∗ .
• ( (cid:101) y
= 1 , f < 1 / 2 , η > 1 / 2 , LRT ﬂips).
According to the LRT scheme, this happens with probability at most Pr( A ∩ ( η > 1 / 2)) .
• ( (cid:101) y
= 1 , f < 1 / 2 , η < 1 / 2 , LRT does not ﬂip).
According to the LRT scheme, this happens with probability at most Pr( A c ∩ ( η < 1 / 2)) .
Here A c denotes the complement of the event A .
• ( (cid:101) y
= 1 , f > 1 / 2 , η > 1 / 2 , LRT ﬂips).
According to the LRT scheme, this happens with probability at most Pr( A ∩ ( η > 1 / 2)) .
• ( (cid:101) y
= 1 , f > 1 / 2 , η < 1 / 2 , LRT does not ﬂip).According to the LRT scheme, this happens with probability at most Pr( A c ∩ ( η < 1 / 2)) .
• ( (cid:101) y
= 0 , f < 1 / 2 , η > 1 / 2 , LRT does not ﬂip).
According to the LRT scheme, this happens with probability at most Pr( B c ∩ ( η > 1 / 2)) .
• ( (cid:101) y
= 0 , f < 1 / 2 , η < 1 / 2 , LRT ﬂips).
According to the LRT scheme, this happens with probability at most Pr( B ∩ ( η < 1 / 2)) .
• ( (cid:101) y
= 0 , f > 1 / 2 , η < 1 / 2 , LRT ﬂips).According to the LRT scheme, this happens with probability at most Pr( B ∩ ( η < 1 / 2)) .
• ( (cid:101) y
= 0 , f > 1 / 2 , η > 1 / 2 , LRT does not ﬂip).
According to the LRT scheme, this happens with probability at most Pr( B c ∩ ( η > 1 / 2)) .
In all of these cases, one observes (after using the fact that || f − (cid:101) η || ∞ ≤ (cid:15) that the probability of these 8 events is either of the form
• Pr (cid:18) 1 / 2 < η ( x ) < 1 1+∆ (cid:48) − τ 01 1 − τ 01 − τ 10 + (cid:15) (cid:19) , or • Pr (cid:18) 1 1+∆ (cid:48) − τ 01 1 − τ 01 − τ 10 − (cid:15) <
η < 1 / 2 (cid:19) , or • Pr (cid:18) 1 / 2 < η ( x ) <
∆ (cid:48) 1+∆ (cid:48)
− τ 01 1 − τ 01 − τ 10 + (cid:15) (cid:19) , or • Pr (cid:18) ∆ (cid:48) 1+∆ (cid:48) − τ 01 1 − τ 01 − τ 10 − (cid:15) <
η < 1 / 2 (cid:19) .
Next, we observe that |
[Equation]
Using these observations
[Equation]
the four expressions can be bounded by the analogous four expressions found by replacing ∆ (cid:48) by ∆ .
{+• Pr (cid:16) 1 / 2 < η ( x ) < 
+}
[Table]
∆ − τ 01 1 − τ 01 − τ 10 + 2 (cid:15) (cid:17) , or • Pr (cid:18) 1 1+∆ (cid:48) − τ 01 1 − τ 01 − τ 10 − 2 (cid:15) <
η < 1 / 2 (cid:19) , or • Pr (cid:18) 1 / 2 < η ( x ) <
∆ (cid:48) 1+∆ (cid:48)
− τ 01 1 − τ 01 − τ 10 + 2 (cid:15) (cid:19) , or • Pr (cid:18) ∆ (cid:48) 1+∆ (cid:48) − [Equation]
Assuming
Tsybakov condition, one can bound each of these probabilities by
C (cid:16) |
[Equation]
Notice that although there are 8 cases, they are mutually disjoint, so the ﬁnal failure probability is bounded by the maximum of these 8 cases, also of the same form.
Hence we have proved
[Equation]
which ﬁnishes the proof of Theorem 1.