A S IMPLE A PPROACH TO THE N OISY L ABEL P ROBLEM T HROUGH THE G AMBLER ’ S L OSS
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning in the presence of label noise is a challenging yet important task, and it is crucial to design models that are robust when part of the dataset are mislabeled.
In this paper, we discover that a new class of loss functions called the gambler’s loss provides strong robustness to label noise across various levels of corruption.
Training with this modiﬁed loss function reduces memorization of data points and is a simple yet effective method to improve robustness and generalization.
Moreover, using this loss function allows us to derive an analytical early stopping criterion that accurately estimates when over-memorization begins to occur.
Our overall approach achieves strong results and outperforming existing baselines.
L EARNING FROM N OISY L ABELS
As a ﬁrst step of supervised learning tasks, the user often collects a large amount of data in the form of ( x,y ) pairs where x represents the input data and y the desired labels.
However, parts of real world data can often be mislabeled due to 1) annotator mistakes as a natural consequence of large-scale crowdsourcing procedures (Howe, 2008), 2) the difﬁcultly in ﬁne-grained labeling across a wide range of possible labels (Russakovsky et al., 2015), 3) subjective differences when annotating emotional content (Busso et al., 2008), and 4) the use of large-scale weak supervision (Dehghani et al., 2017).
Learning in the presence of noisy labels using neural networks is challenging since overparametrized neural networks are known to be able to memorize all labels even with strong regularization (Zhang et al., 2017).
When the model memorizes the noisy labels, its generalization deteriorates (e.g., see Figure 1 and Section 1.1).
In fact, this agrees with the theoretical ﬁnding in (Nakkiran et al., 2019) that neural networks trained with SGD often learns functions stage by stage from simpler functions to more complex ones, and functions whose output is uniformly random are the functions with highest complexity, and so memorization of random labels often come late in the training.
Based on these, we may devise special methods to extract signal from noisy labels.
Similar to Patrini et al.
Yu et al.
; Han et al.
(2018), we deal with an idealized noisy label setting, where a noise transition matrix is invoked to deﬁne the problem, and each label has the probability deﬁned by the transition matrix to be ﬂipped to another label (see appendix for a formal mathematical deﬁnition).
Organization:
In this section, we ﬁrst hypothesize and then empirically verify the existence of three stages during training in the presence of noisy labels; in Section 2, we propose that the gambler’s loss function is a noise robust loss function that can be used for noisy label problems, and we further propose an analytical early stopping criterion using the gambler’s loss.
We then verify our methods through a series of experiments and comparisons with existing approaches.
1. T RAINING T RAJECTORY OF N OISY L ABELS
In practice, one often faces the problem of learning from a dataset D
= { x,y } ni
= 1 whose labels are partially corrupted, i.e. D = D clean ∪ D corrupt .
We consider a setting where the labels are symmetrically corrupted (i.e. any label has equal probability to be ﬂipped to any other class).
Let (cid:15) =
1 − r denote the probability that a data point ( x,y ) is clean, r being the probability of mislabeling.
A classiﬁcation problem can be denoted as learning a function f (⋅)
∶ X → Y , where x ∈ X =
R n are the data points, and
y ∈ Y =
R m are the targets.
In practice, we parametrize f with parameters θ (i.e. f θ ) and learn θ by minimizing the empirical risk E [ (cid:96) ( f θ ( x ) ,y )] for some loss function (cid:96) .
The standard loss function for classiﬁcation is cross entropy (or the nll loss).
However, naive training on D using nll loss often results in a “peaked” learning curve as shown in Figure 1b.
We observe that such learning curves often consist of three stages (See Figure 1):
[Figure]
1) Fast Learning Stage : The model quickly learns the underlying mapping from data to clean labels; one observes rapid decreases in training loss and increases in test accuracy.
2) Gap Stage : From Figure 1a, we notice learning on the clean labels is almost complete ( (cid:96) clean ∼ 0 . 5 ) but training on noisy labels has not started yet ( (cid:96) corrupt ∼ 2 . 5 ), and a large gap in training loss exists between D clean and D corrupt . Both the train loss and the test accuracy reach
a plateau, and this is the time at which the generalization performance is the best
Memorization : When there are only noisy labels left to learn from, the model will memorize these noisy labels and the train loss decreases slowly to 0 .
However, memorizing noisy labels hurts generalization and the test accuracy decreases as well.
These stages seem to be present across datasets and model architectures when label noise is present.
This is problematic because the testing accuracy reaches a peak early in training and then decreases steadily because the noisy gradient from D corrupt starts to dominate that from D clean , leading to memorization of noisy labels and affecting generalization.
Therefore, tackling the problem of learning from noisy labels involves reducing the negative effect from D corrupt .
This is hard in general, since one does not know a priori which points are mislabeled.
We mainly deal with symmetric noise, and we denote the corruption rate (the probability that a data point has a label that is not its true label) as r = 1 − (cid:15) , where (cid:15) is the clean rate.
Various approaches have been proposed (Patrini et al., 2017; Han et al., 2018; Yu et al., 2019), but, unfortunately, the best current methods can do is only to slow down the drop in test accuracy instead of preventing it completely (Han et al., 2018).
Our method will be based on the above observation; to summarize the key point, we note:
Observation.
When training on a label-corrupted dataset, there is a stage when the average loss on the correct points is much lower that of the corrupted points.
Intuitively, since features are easier to learn than random noises on average.
This phenomenon also motivates a label noise method sometimes called the “small loss method” (Han et al., 2018;
Yu et al., 2019), in which the data points with loss higher than a threshold is screened to prevent the network from learning wrong information from such point.
The observation also holds theoretically.
As shown in (Nakkiran et al., 2019), neural networks trained with SGD often learns functions stage by stage from simpler functions to more complex ones, and functions whose output is uniformly random are the functions with highest complexity.
From this observation, we make the following idealized assumption that our method is based on:
Assumption.
(Idealized gap stage)
During the gap stage, the model has not learned anything about the data points in D corrupt , and predicts strictly uniform score across all the labels, and it achieves perfect accuracy on D clean .
We ﬁnd that this assumption holds well for simple datasets such as MNIST and on datasets with very high corruption rate, where our method achieves best results, and less so on more complicated datasets such as CIFAR10 (for example, on gap stage, the training accuracy on the clean dataset do not reach close to 100% ).
T HE G AMBLER ’ S L OSS F UNCTION AND H OW TO E ARLY S TOP
We propose an approach that uses the gambler’s loss function (Ziyin et al., 2019).
Intuitively speaking, the gambler’s loss builds on the analogy that making a classiﬁcation given a data point is the same as a gambler making a bet on a horse race given the horses, and that minimizing the nll loss in the classiﬁcation problem is the same as maximizing the doubling rate of wealth.
The gambler’s loss may be generalized to include a reservation option in which the gambler can play “safe” and reserve part of his money in pocket.
We refer to this generalized version as the gambler’s loss .
Mathematically, the gambler’s loss augments the target space by an “reservation” dimension: Y → Y ′ =
R m
+ 1 , where the ( m + 1 ) -th dimension is deﬁned as the reservation score.
In the case of a neural network, using this loss involves adding one more output neuron followed by a softmax layer to normalize the output.
The gambler’s loss function for a single data point ( x,y ) with y being a 0 − 1 label, and with neural network f θ (⋅)
∶ X → Y ′ is
[Equation]
[Equation]
) o is a hyperparameter for the loss function, and reasonable range of o is o ∈ ( 1 ,m ] (also see ).
Note that we omit θ when unnecessary, but f always denote a neural network in this work, and should always come with subscript θ , denoting its parameters.
When training with gambler’s loss, we notice that usually the following intuition hold.
Larger o encourages learning while smaller o discourages learning.
When o is appropriate, lowering o makes the neural network “cautious” about learning noises present in the dataset, and so lowering o makes the model more robust to such noise.
However, when o is below a critical value, the minimum becomes to refrain from learning completely, and so achieving best performance using gambler’s loss often involves approaching this critical value from above.
2. E ARLY S TOPPING C RITERION F OR THE G AMBLER ’ S L OSS
Now we propose a criterion to accurately estimate the location of the gap and perform early stopping at the peak in test accuracy.
It is shown to be theoretically true that early stopping is effective against label noise in Li et al.
(2019) for overparametrized networks trained with gradient descent.
However, it is surprising that the existing literature only focuses on delaying or relieving the negative effect of learning the noisy labels, but not on ﬁnding the best time to do early-stop (Patrini et al., 2017; Han et al., 2018; Yu et al., 2019).
As in the previous works Han et al.
Yu et al.
(2019), we assume that the (cid:15) is known (to perform early stopping); otherwise, it may be estimated.
Let o be the gambler’s hyperparameter.
Let ˆ p be the predicted probability on the true label y , and let ˆ k denote the prediction made on all the wrong classes added altogether, ˆ l be the predicted conﬁdence score by the gambler’s loss.
By deﬁnition of a probability distribution, we have ˆ p + ˆ k + ˆ l = 1 .
Based on our observations, we make the following idealized assumption: during the the gap stage, the model has not learned anything about the data points with wrong labels, i.e., it makes strictly uniformly random guess across the all labels, the predicted probabilities are the same and are all equal to ˆ km .
This allows us to write down the general loss function during the gap stage:
[Equation]
where the expectation is taken over the dataset, and we will omit it since it does not affect clarity.
The following theorem uses the above assumptions and predict the loss of the gap stage.
Theorem 1.
During the idealized gap stage, the optimal solution has ˆ k = 0 , and so the loss function takes the form:
[Equation]
which exhibits an optimal solution at
[Equation]
which only depends on (cid:15) and o , and we have p ∗ =
(cid:15)o − 1 o − 1 .
The proof is given in the appendix.
This theorem tells us that a network trained with the gambler’s loss function with hyperparameter o on a symmetrically corrupted dataset with corruption rate 1 − (cid:15) should have training loss around − ˜ (cid:96) ∗ ( (cid:15),o ) when entering the gap stage.
This motivates using − ˜ (cid:96) ∗ ( (cid:15),o ) as the early stopping criterion.
Notice that we expect our loss to have the following limits:
As (cid:15) → 1 , we recover the no-noise setting and the optimal loss is 0 .
As (cid:15) → 0 , we go to the full noise case, and the dis-conﬁdence score would dominate, and the gambler’s loss converge to log 1 o (when o is large) (Ziyin et al., 2019). We summarize our proposed method in Algorithm 1. 2. R OBUSTNESS T O L ABEL N OISE WITHOUT EARLY STOPPING [Figure]
One common approach (among others) to label noise involves introducing a surrogate loss function that is specialized for the corrupted dataset at hand and is not very useful when one is not aware of the existence of label noise.
Therefore, one surprising ﬁnding in this work is that simply using the gambler’s loss improves the performance of the model at convergence, thus provides robustness to label noise automatically.
For example, in Figure 4a, we see that, without knowing (cid:15) , the gambler’s improves over the baseline by over 40% testing accuracy at convergence. We hypothesize that the reason is that the gambler’s loss function makes learning on corrupted points slower than the clean points. As a result, this widens the test accuracy plateau by slowing down the memorization phenomenon. Let a data points be x , whose “observable” label is y , and true label is y ′ .
Intuitively speaking, the mechanism is that it is easier to learn to reserve (by increasing f ( x ) m + 1 ) than to memorize the noisy points (by increasing f ( x ) y ), and so f ( x ) m +saturates the softmax layer, making the gradient vanish.
Also, o cannot be too small, since that will also make the gradient vanish on the clean data points.
To be slightly more formal.
Notice that the gambler’s loss function has derivative (deﬁning g ( x ) y ∶= f
{+( x ) y + 
+}
o f ( x ) m
+ 1 ):
[Equation]
[Equation]
and by Theorem 1 (in Section 2.1), during the gap stage we have:
[Equation]
and the probability we update a clean data point is (cid:15) , while that of a corrupted data is 1 − (cid:15)
m − 1 , and, therefore, in expectation (sloppy notation), the gradient is of order
[Equation]
[Figure]
In general, o − 1 < m − 1 and so a corrupt data, in expectation, has gradient o − 1 m − 1 times smaller than that of the clean data, and, within a proper range of o , the lower the o is, the slower the model learns the corrupt labels, providing stronger robustness (in fact, one might go one step further with this qualitative analysis, see section B).
This trend is what we observe in Section 4.1 when o lies above a critical value.
Moreover, notice that this is independent of (cid:15) , and we expect the gambler’s loss to help even when (cid:15) is unknown, and this is shown in section 4.1.
On the other hand, this analysis suggests that, if we train the gambler’s loss with a subgradient methods such as Adam (Kingma & Ba, 2014), the constant factor in front of the gradient would be canceled by the preconditioner 1 /√ E g 2 , and so the widening effect would not be observed.
See Figure 2.
The time it takes for Adam to reach and to start exiting the plateau seem invariant to o , while using SGD, we see that the plateau is widened as we decrease o .
For Adam, even if the widening effect is not changed, we see that at convergence, smaller o does lead to higher performance using both optimizers.
More plots and experiments using
Adam are given in the appendix.
Moreover, as shown in Yu et al.
; Han et al.
(2018), simply training on the raw loss functions features a gap stage that lasts only very shortly ( ∼ 1 epoch), while the gambler’s loss makes the network stays around its performance peak for a much longer time.
The performance on the gambler’s loss has barely dropped by the time the baseline almost dropped to random accuracy, and this an makes it easy to detect the peak and perform early-stopping.
2. T UNING FOR THE O PTIMAL o AND F IRST -O RDER P HASE T RANSITION
We note that there is a “good” range for hyperparameter o and a bad range. The good range is between a critical value o crit and M (non-inclusive) and the bad range is smaller than o crit . We conducted substantial experiments to observe that any o within the good range are seen to provide improved robustness against label noise, and so most of the cases we do not tune for o but only to make sure it is within the good range. Moreover, tuning for the optimal o is fast and can be done in linear time, and we discuss this and its connection to the physical ﬁrst-order phase transition theory in the appendix.
R ELATED W ORK
Label Noise: Modern dataset often contains a lot of labeling errors (Russakovsky et al., 2015;
Schroff et al., 2010).
Existing methods often focus on using a surrogate loss function (Patrini et al., 2017) that is speciﬁc to the label noise problem at hand, or design a special training scheme to alleviate the negative effect of learning the data points with wrong labels (Han et al., 2018; Yu et al., 2019).
In this work, we mainly compare with the following methods.
F-Correction (FC) Patrini et al.
(2017): this method smooths the hard labels with the noise transition matrix; we estimate the transition matrix as suggested by the paper; Co-teaching (CT) (Han et al., 2018): this method trains two networks simultaneously, and update each with the other’s predicted label to decouple the mistakes; Co-teaching+ (CT+)
(Yu et al., 2019): this method improves the previous method by only propagating on the labels the two networks disagree.
Also, we use the subscript ES as a shorthand for early stopping.
Since the early stopping criterion for these methods are not clear, we early stop at where the training accuracy is larger than (cid:15) for comparison, since this is when the learning on the noisy data has deﬁnitely started.
Early Stopping .
Early stopping is an old problem in machine learning Prechelt (1998); Amari (1998), and studying this in the setting of label noise also appeared recently Li et al.
; Hu et al.
In particular, we propose to early stop on a loss function called the gambler’s loss function Ziyin et al.
Our analysis on this loss function allows us to propose a very simple analytic function to predict a early stopping threshhold without using a validation set and is independent of the model (as long as it is a neural network with enough capacity) or the task.
Early stopping without validation set is also recently studied, but does not relate to the label noise problem (Mahsereci et al., 2017).
It has been long noticed (Fr´enay & Verleysen, 2013; Han et al., 2018) and recently theoretically proven (Li et al., 2019) that early stopping can an effective way to defend against label corruption (but no actual method or heuristic for early stopping is given); to the best of our knowledge, this is ﬁrst early stopping criterion to deal with the problem.
E XPERIMENTS
[Figure]
We design several experiments to evaluate 1) the robustness of gambler’s loss to noisy labels, 2) the effectiveness of our early stopping criterion and 3) the performance of gambler’s loss when applied to existing methods to combat noisy labels.
4. R OBUSTNESS OF G AMBLER ’ S L OSS TO L ABEL N OISE
We ﬁrst demonstrate that simple training on the gambler’s loss provides robustness against label noise (also see Section 2.3).
For this and later sections, the exact experimental details are given in the
Appendix.
For demonstration, we choose a simple CNN with 2 convolutional layers followed by 2 fully connected layers.
See Figure 4.
We see that with properly tuned o , the gamblers’s loss alleviates the drop in performance caused by learning the corrupt labels.
For example, in Figure 4a, all three choices of o makes the accuracy plateau wider than training on the raw loss (nll loss).
At convergence, the ﬁnal accuracy is much higher ( ∼ 90% ) than the baseline ( ∼ 55% ).
For the experiments on CIFAR10, the results are slightly subtler.
Here we see the effect of not properly tuning o .
As shown in (Ziyin et al., 2019), the proper choice ranges from 1 to M ; higher o ( ≈ M ) encourages learning while very low o ( ∼ 1 ) slows down learning.
We notice that for a different corruption rate, a different o crit seem to exist.
Also see Table 1 (category Gbler ), we see that at convergence, training on gambler’s loss achieves much higher accuracy than the baseline training on nll loss.
4. E ARLY S TOPPING
Now, we show that stopping at our predicted loss level succeeds in stopping at a point close to where the maximum accuracy is achieved.
See Figure 5.
We choose architectures of very different capacity.
The experiment on MNIST is done as in the previous section and the experiment on CIFAR10 is done using ResNet-18.
We also test on a wide range of noise levels ranging from small, with corruption rate 0 .
2 , to extremely large, with r = 0 .
85 (the intermediate ranges with larger ﬁgures are given in the appendix, see Figure 7).
We see that our method predicts close to optimal early stopping point for all corruption rates we tested on.
Also notice how the learning curves are different for the two different datasets.
This suggests that our theory works even in the presence of very different dynamics.
The fact that this early stopping criterion works suggests in return suggests the validity of our assumption.
In these experiment, we do not tune for optimal o , they are all set to o = 9 . 9 ; since the results are qualitatively similar when o > o crit .
See Table 1 and also compare with other baselines in Table 2.
We notice that doing this already achieves SOTA results on MNIST for all symmetric noise categories according to the results in Han et al.
(2018) and Yu et al.
Early stopping (AES) using our criterion is compared with a very standard early stopping method using validation set: we split 6000 images from the training set to make a validation set, and we early stop when the validation accuracy stops to increase for 5 consecutive epochs.
There are also a few other validation-based early stopping
[Figure]
[Table]
=
[Table]
MN
[Table]
= [Table] [Table] MN
[Table]
=
[-1 ± 0 . 8 ± 0 . MN r = 
 1 ± 3 .91 .-][Table]
[Table]
MN r = 0 .
[Table]
[Table]
MN r =
[Table]
[Table]
criterion, but they are shown to perform qualitatively and quantitatively similar (Prechelt, 1998), so we only compare with this method.
This early stopping is called VES (validation early stopping) and our method is called AES (analytical early stopping).
We see that our proposed method signiﬁcantly outperforms the baseline early stopping method; this is because a small validation set may have large variance and this problem is more serious when label noise is present (since validation sets are often split from the train set).
In the VES vs. AES comparison, we ﬁx o = 9 . 99 and is not directly comparable to the left parts in the table.
In the left part, We also notice that training and stopping on the gambler’s loss is especially effective on tasks with extreme corruption rate ( r = 0 . 85 ), with around 70% accuracy improvement over the baseline and 28% improvement over the runner-up
SOTA methods.
4. C OMBINING G AMBLER ’ S L OSS WITH O THER M ETHODS AND B ENCHMARKING
We also show how our method might be combined with SOTA methods (we choose CT).
To combine gambler’s loss with co-teaching, we ﬁrst replace the loss function in Co-Teaching with the gambler’s loss, and reject fewer than or equal to (cid:15) data points by sorting on the original loss function from the largest to smallest.
This modiﬁcation only shows how we might combine our method with existing methods, and we do not argue that it is the optimal for such combination.
We call our modiﬁed version Co-Gambler, and speciﬁc details of the modiﬁcation is given in the Appendix; we also simplify CT by eliminating four of its hyperparameters.
Namely, we do not use learning rate and momentum schedules, and we use not use drop rate schedule for co-Gambler (while these are used for the CT columns).
See Table 2.
We see that on MNIST, combining our method with Co-Teaching always outperforms the baselines.
On CIFAR10, our method seems to work better when the noise level is extreme ( r ∼ 0 .
7 , 0 . 8 ), agreeing with our ﬁnding in the previous section.
For the CIFAR experiments, we do not compare with CT+ since it is a modiﬁed version of CT and we notice that it fails in the same range as CT ( r = 0 .
We notice that this way of combining gambler’s loss improves on pure AES on gambler’s loss only by 3 − 4% , while the training time more than doubles with twice as many parameters.
The reason for the limitation of this combination is simple:
CT involves a special training procedure that throws away a signiﬁcant percentage of the data, thus changing the data distribution.
Our method, however, assumes that the data distribution is intact, and can be well described by (cid:15) .
Early stopping on gambler’s loss, running much faster than CT and its variants, still achieves SOTA results at r = 0 . 85 , which shows the strength of our simple method.
The method that has similar complexity and runs at similar speed as ours is FC; FC, however, requires knowing the whole transition matrix, and is outperformed signiﬁcantly by our method.
[Table]
C ONCLUSION AND D ISCUSSION
In this work, we have proposed a simple and effective method to deal with label noise problems.
This paper promotes the gambler’s loss as a noise robust loss function, shown to improve performance of existing classiﬁer model, while the early stopping criterion provides a simple criterion for early stopping when the corruption rate is known.
Depending on the problem, one may decide to use the complete framework together (gambler’s loss training + early stopping), or, when the assumption is not well satisﬁed, only use the gambler’s loss to improve the model’s robustness against a labelcorrupted learning task.
This urges for an understanding for why the gambler’s loss actually works.
We plot in training accuracy for MNIST with corruption rate 0 .
8 in Figure 6.
We see that the gambler’s loss seems to work in a fundamentally different way from the standard nll loss.
Training on nll loss achieves 100% accuracy on the training set, with its loss approaching 0 ; however, the gambler’s loss does not do so, its training accuracy stops increasing at around 36% , which is slightly above (cid:15) .
We note that this observation also holds for various corruption rates, and for different optimizers such as SGD.
Comparing with Table 5, we see that the gambler’s loss achieves about 76% testing accuracy, while the baseline drops to 19% at this time.
The vertical line also shows where our early stopping criterion is reached, at this point is training accuracy is ∼ 20% with testing accuracy ∼ 92% .
We hypothesize that the main reason for gambler’s loss’s outperforming the baselines is that it automatically chooses a subset of the training set to learn on, thus avoiding a large part of the dataset that is corrupted.
This result may provide insight into the current debate on the generalization-memorization effect (Zhang et al., 2017;
Wu et al., 2017;
Hu et al., 2019; Arpit et al., 2017).
A key argument from Zhang et al.
(2017) was “ deep neural networks easily ﬁt random labels...; explicit regularization may improve generalization performance, but is neither necessary nor by itself sufﬁcient for controlling generalization error .
Our results above, however, shows that this is not the case when a different loss function is chosen.
While the nll loss encourages learning and memorizing the training set, the gambler’s loss does not seem to do so.
Our ﬁndings suggests that the key to understanding the memorization effect might be on the loss function itself, not the architecture of neural networks or explicit regularization, this is also what is suggested by Abiodun et al.
We hypothesize that it may be seen as a natural generalization of the nll loss function, since it only involves adding a task independent term to the original nll loss, and using o > o crit seems always beneﬁcial.
We would like to study the memorization effect using the gambler’s loss in more detail.
Applications to real-world large-scale datasets with real label noise are also ripe avenues for future work.
A A BRIEF INTRODUCTION TO THE GAMBLER ’ S LOSS
Let f (⋅) denote the model and f ( x ) is the m + 1 -dimensional normalized prediction, i.e., [Equation] [Equation]
The gambler’s loss for a classiﬁcation problem takes the form
[Equation]
where p k,x = Pr ( k ∣ x ) is the true label.
Notice that if o → ∞ , the loss function is equivalent to the KL divergence.
The empirical target is usually a 1 − 0 loss, where p y,x = 1 for the true label y and zero for k ≠ y ; thus the loss function takes the following simple form (and we change max (cid:96) to min − (cid:96) ):
[Equation]
and this reduces to the cross entropy (or nll loss) if o → ∞ .
It is also useful to notice the following relation.
Let ˆ p be the predicted probability on the true label y , and let ˆ k denote the prediction made on all the wrong classes added altogether, ˆ l be the predicted conﬁdence score by the gambler’s loss.
To relate this to the deﬁnition above (Equation 11), we note that this is equivalent to deﬁning ˆ p ∶= f
( x ) y , ˆ k ∶=
∑ mi ≠ y f
( x ) i , ˆ l ∶= f
( x ) m + 1 , and, therefore, by the normalization condition in Equation
[Equation]
and this condition will be used to prove Theorem 1.
B A T RAINING S CHEDULE
Recall equation 7:
[Equation]
We can go one step further and plug in g ( x ) y
{+= f ( x ) y + 
+}
o f ( x ) m
+ 1 to consider the effect on the prediction for f ( x ) j = y ′ ,f ( x ) j ≠ y ′ ,
j < m + 1 , and f ( x ) m + 1 .
If ∣∇ f ( x ) m + 1 ∣ > ∣∇ f ( x ) j ≠ m + 1 ∣ , we say that the reseravation score f ( x ) m + 1 is prone to saturation, and learning on the label class j is prevented by such saturation.
We obtain:
[Equation]
where the ﬁrst term is the gradient for the correct label, and the second term he gradient for the incorrect labels and the third term is the gradient for the reservation score.
Roughly estimating, to prevent saturation delaying learning on correct labels, we want 1 > 1 o + o − 1 o ( m − 1 ) , for MNIST and
CIFAR 10 we have m = 10 , and this condition translates to o > 8 . 88 ... , which agree in magnitude to the experimental threshold o crit ≈ 8 .
This high-level agreement suggests that our qualitative analysis is good.
On the other hand, if we want the saturation on f ( x ) m + 1 to prevent learning on the corrupt labels completely, then we should have 1 o + o − 1 o ( m − 1 ) >
o − 1 m − 1 , and for m = 10 , this translates to o < 4 .
This suggests the following training scheme for m = 10 problems:
start training with o > 8 .2.
when the early stopping criterion is met, either stop, or switch to o <
In fact, similar training scheme is used in (Ziyin et al., 2019) in order to achieve SOTA results, while unaware of this analysis.
For CIFAR10, they started with o = 9 .
9 to for the ﬁrst ﬁfty epochs, and then switched to o = 2 afterwards to achieve the best performance.
C M ATHEMATICAL SYMBOLS
N ∶ number of input dimension
M ∶ number of output dimension
D ∶ dataset, ∣D∣ ∶ size of the dataset
D E XPERIMENT D ETAILS
We use Pytorch as the framework of implementation.
Code to this paper will be released at https://********
E T HEOREM PROOF
In this section we prove theorem 1.
We ﬁrst show that ˆ k = 0 .
Intuitively speaking, this simply means that a gambler betting randomly will not make money, and so the better strategy is to reserve money in the pocket, and so it sufﬁces to show that for any solution ˆ p, ˆ k, ˆ l , the solution ˆ p
=
ˆ p, ˆ k ′ = 0 , ˆ l ′ =
ˆ l + ˆ k achieves better or equal doubling rate.
For a mislabeled point (we drop ˆ ⋅ ), the loss is log ( kM + lo ) but
M > o , and so log ( kM + lo ) < log ( k + lo ) , and we have that optimal solution always have ˆ k = 0 .
Now, we ﬁnd the optimal solution to
[Equation]
[Equation]
by taking the derivative with respect to p:
[Equation]
and then setting it equal to
[Equation]
is the ˆ p optimal :
[Equation]
and then plugging into the original equation [9]:
[Equation]
F A DDITIONAL E XPERIMENTS
In this section, we give more experiment results on Gambler + Early Stopping.
See Figure
[Figure]
G D EFINITION OF TRANSITION MATRIX
Transition matrix is used to replace some correct labels with wrong labels because the dataset used in our experiments such as MNIST and CIFAR are clean.
There are two different types of noise transition matrix and explanation follows.
Symmetric Flipping :
Each label has an uniform possibility to transform to one of the rest class.
In most cases of manual label work, a correct label has equal possibility to be any other class label.
n is the number of class and δ is the rate of label that will be modiﬁed as a wrong label
[Equation]
⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦
[Equation]
Pair Flipping :
Each class can only be corrupted as a speciﬁc class.
For example, it’s more likely to label 6 as 9 than any other class except 9.
δ is the rate of label that will be modiﬁed as a wrong label:
[Equation]
[Equation]
H CNN ARCHITECTURE
In this section, we show the architectures that we used but did not describe in the main text.
[Table]
[Table]
I EFFECT OF TUNING O AND CRITICAL BEHAVIOR
In this section [Table] we give more experiments on the critical behavior discussed in 2.3.
First see Figure
[Table] [Figure] In Figure 2, we show that lower value of o provides stronger robustness to label noise consistently; this raises the question about whether we can continue to lower o to its lower limit ( = 1 ) to achieve the best performance or not - the answer is no, but the reason is very surprising.
We empirically ﬁnd that a ﬁrst order phase transition exists when tuning o (such as transition of water to ice).
In particular, we ﬁnd that there exists a critical value o crit such that, when o > o crit , the robustness to label noise increases as we lower o ; when o < o crit , the learning proceeds badly: the change in performance is discontinuous .
This is characteristic of a physical ﬁrst order phase transition (Landau & Lifshitz, 2013).
See Figure 3.
In this setting, we have r = 0 .
5 for MNIST, and we pin down the o crit to lie between 8 . 21 and 8 . 22 ; qualitatively different behaviors are found for the two different phase.
See appendix for plots ranging from o = 10 .
to o = 5 . 0 .
A qualitatively similar behavior is observed using Adam 1 and on CIFAR10, and using other architectures.
Clearly studying this critical behavior is very interesting but beyond the scope of this work.
The above observation, however, suggests the following heuristic to tune hyperparameter o :
Start with o very close to M (for reference, for MNIST, o ≈ 9 .
5 seems a good starting point; for CIFAR10, o ≈ 9 . 9 ; CIFAR100: o ≈ 99 . 9 ).
If the learning curve starts similarly as before, then decrease o until the learning stops abruptly, and use the previous o
This procedure for tuning o does not rely on a validation set at all.
Also, as show in in Figurefor o = 8 . 3 , 8 . 5 , the performance is very similar when o is close to o crit , there is no need to be inﬁnitesimally close to the critical value.
Unless otherwise noted, we tune our hyperparameter this way in the experiment section.
Moreover, One can prove that o
= M will always work at least as good as the baseline, and so o cr should always < M .
[Figure]
[Figure]