N OISE AGAINST NOISE : STOCHASTIC LABEL NOISE HELPS COMBAT INHERENT LABEL NOISE
Anonymous authors
Paper under double-blind review
A BSTRACT
The noise in stochastic gradient descent (SGD) provides a crucial implicit regularization effect, previously studied in optimization by analyzing the dynamics of parameter updates.
In this paper, we are interested in learning with noisy labels, where we have a collection of samples with potential mislabeling.
We show that a previously rarely discussed SGD noise, induced by stochastic label noise (SLN), mitigates the effects of inherent label noise.
In contrast, the common SGD noise directly applied to model parameters does not.
We formalize the differences and connections of SGD noise variants, showing that SLN induces SGD noise dependent on the sharpness of output landscape and the conﬁdence of output probability, which may help escape from sharp minima and prevent overconﬁdence.
SLN not only improves generalization in its simplest form but also boosts popular robust training methods, including sample-selection and label-correction.
Speciﬁcally, we present an enhanced algorithm by applying SLN to label-correction.
I NTRODUCTION
The existence of label noise is a common issue in classiﬁcation since real-world samples unavoidably contain some noisy labels, resulting from annotation platforms such as crowdsourcing systems (Yan et al., 2014).
In the canonical setting of learning with noisy labels, we collect samples with potential mislabeling, but we do not know which samples are mislabeled since true labels are unobservable.
It is troubling that overparameterized Deep Neural Networks (DNNs) have sufﬁcient capacity to memorize noise (Zhang et al., 2017), leading to poor generalization performance.
Thus, we are urgent for robust training methods that can mitigate the effects of label noise.
The noise in stochastic gradient descent (SGD) (Wu et al., 2020) provides a crucial implicit regularization effect for training overparameterized models.
SGD noise is previously studied in optimization by analyzing the dynamics of parameter updates, whereas its utility in learning with noisy labels has not been explored to the best of our knowledge.
In this paper, we ﬁnd that the common SGD noise directly applied to model parameters does not endow much robustness, whereas a variant induced by controllable label noise does.
Interestingly, inherent label noise is harmful to generalization, while we can mitigate its effects using additional controllable label noise.
To prevent confusion, we use stochastic label noise (SLN) to indicate the label noise we introduce.
Inherent label noise is biased and unknown, ﬁxed when the data is given.
SLN is mean-zero and independently sampled for each instance in each training step.
Our main contributions are as follows.
• We formalize the differences and connections of three SGD noise variants (Proposition 13), and show that SLN induces SGD noise that is dependent on the sharpness of output landscape and the conﬁdence of output probability.
• Based on the noise covariance, we analyze and illustrate two effects of SLN (Claim 1 and Claim 2): escaping from sharp minima 1 and preventing overconﬁdence 2 .
• We empirically show that SLN not only improves generalization in its simplest form but also boosts popular robust training methods, including sample-selection and labelcorrection.
We present an enhanced algorithm by applying SLN to label-correction.
[Figure]
In Fig.
1, we present a quick comparison between models trained with/without SLN on CIFARwith symmetric/asymmetric/instance-dependent/open-set label noise.
Throughout this paper, we use
CE to indicate a model trained with cross-entropy (CE) loss without any robust learning techniques, and the canonical CE loss is used by default for methods like SLN.
In Section 4, we will provide more experimental details and more results that comprehensively verify the robustness of SLN on different synthetic noise and real-world noise.
Here, the test curves in Fig.
1 show that SLN avoids the drop of test accuracy, with converged test accuracy even higher than the peak accuracy of the model trained with CE.
The right two subplots in Fig.
1 show the average loss on clean and noisy samples.
When trained with CE, the model eventually memorizes noise, indicated by the drop of average loss on noisy samples.
In contrast, SLN largely avoids ﬁtting noisy labels.
R ELATED W ORK 2. SGD N OISE AND THE REGULARIZATION EFFECT
The noise in SGD (Wu et al., 2020; Wen et al., 2019; Keskar et al., 2016) has long been studied in optimization.
It is believed to provide a crucial implicit regularization effect (HaoChen et al., 2020; Arora et al., 2019; Soudry et al., 2018) for training overparameterized models.
The most common SGD noise is spherical Gaussian noise on model parameters (Ge et al., 2015; Neelakantan et al., 2015; Mou et al., 2018), while empirical studies (Wen et al., 2019; Shallue et al., 2019) demonstrate that parameter-dependent SGD noise is more effective.
It is shown that the noise covariance containing curvature information performs better for escaping from sharp minima (Zhu et al., 2019; Daneshmand et al., 2018).
On a quadratically-parameterized model (Vaskevicius et al., 2019;
Woodworth et al., 2020), HaoChen et al.
(2020) prove that in an over-parameterized regression setting, SGD with label perturbations recovers the sparse groundtruth, whereas SGD with Gaussian noise directly added on gradient descent overﬁts to dense solutions.
In the deep learning scenario,
HaoChen et al.
(2020) present primary empirical results showing that SGD noise - induced by Gaussian noise on the gradient of the loss w.r.t.
the model’s output - avoids performance degeneration of large-batch training.
Xie et al.
(2016) discuss the implicit ensemble effect of random label perturbations and demonstrate better generalization performance.
In this paper, we provide new insights by analysing SGD noise variants and the effects, and showing the utility in learning with noisy labels.
2. R OBUST T RAINING M ETHODS
Mitigating the effects of label noise is a vital topic in classiﬁcation, which has a long history (Ekholm & Palmgren, 1982; Natarajan et al., 2013) and attracts much recent interest with several directions explored.
1) Malach & Shalev-Shwartz (2017); Han et al.
(2018b);
Yu et al.
; Chen et al.
(2019a)
; Wei et al.
propose sample-selection methods that train on trusted samples, identiﬁed according to training loss, cross-validation or (dis)agreement between two models.
2) Liu & Tao (2015); Jiang et al.
; Ren et al.
Shu et al.
; Li et al.
(2019a) develop sample-weighting schemes that aim to add higher weights on clean samples.
Reed et al.
Sukhbaatar et al. (2015); Patrini et al.
; Han et al.
(2018a) apply loss-correction based on an estimated noise transition matrix.
4) Tanaka et al.
Arazo et al.
; Zheng et al.
(2020) propose label-correction based on the model’s predictions.
5) Ghosh et al.
(2017); Zhang & Sabuncu (2018)
; Xu et al.
; Charoenphakdee et al.
; Lyu & Tsang (2020); Ma et al.
(2020) study robust loss functions that have a theoretical guarantee for noisy risk minimization, typically with the assumption that the noise is class-conditional (Scott et al., 2013; Natarajan et al., 2013).
Chen et al.
(2019b)
; Menon et al.
Hu et al.
Harutyunyan et al.
Lukasik et al.
(2020) apply regularization techniques to improve generalization under label noise, including explicit regularizations such as manifold regularization (Belkin et al., 2006) and virtual adversarial training (Miyato et al., 2018), and implicit regularizations such as dropout (Srivastava et al., 2014), temporal ensembling (Laine & Aila, 2017), gradient clipping (Pascanu et al., 2012;
Zhang et al., 2019; Menon et al., 2020) and label smoothing (Szegedy et al., 2016).
7) One can combat label noise with reﬁned training strategies (Li et al., 2019b; 2020; Nguyen et al., 2020) that potentially incorporate several techniques, including sample-selection/weighting, label-correction, meta-learning (Li et al., 2019b) and semi-supervised learning (Tarvainen & Valpola, 2017; Berthelot et al., 2019).
Among these methods, regularization techniques are closely related to the essence of training networks, and studying robustness under label noise provides a new lens of understanding the regularization apart from the optimization lens.
M ETHOD 3. T HE DIFFERENCES AND CONNECTIONS OF SGD NOISE VARIANTS
Notations.
Let D =
( x ( i ) , y ( i ) ) }
ni =1 be a dataset with noisy labels.
For each sample ( x, y ) , its label y may be incorrect and the true label is unobservable.
Let f ( x ; θ ) be the neural network model with trainable parameter θ ∈ R p .
For a c -class classiﬁcation problem, we have the output f
( x ; θ ) ∈ R c .
We use a softmax function S ( f ( x ; θ ))
∈ R c to obtain the probability of each class.
The loss on a sample is denoted as (cid:96) ( f, y ) .
For classiﬁcation, we use the cross-entropy (CE) loss by default.
In parameter updates, a sample contributes ∇ θ (cid:96) ( f, y ) to the gradient descent.
With SGD noise, the model is trained with a noisy gradient ˜ ∇ θ (cid:96) ( f, y ) .
Following the standard notation of the Jacobian matrix, we have ∇ θ (cid:96) ∈ R 1 × p , ∇ f (cid:96) ∈ R 1 × c , ∇ θ f ∈ R c × p , ∇ θ i f ∈ R c and ∇ θ f i ∈ R 1 × p .
Noise on the gradient of loss w.r.t.
parameters.
The most common SGD noise is the spherical
Gaussian noise directly added to the gradient to parameters (Neelakantan et al., 2015) as follow,
[Equation]
where σ θ > 0 , z θ ∈ R 1 × p and z θ ∼ N (0 , I p × p ) .
Noise on the gradient of loss w.r.t.
the model output.
Taking a step further, HaoChen et al.
(2020) study SGD noise induced by label noise on a quadratically-parameterized regression model, whereas for classiﬁcation, they add mean-zero noise to ∇ f (cid:96) ( f, y ) as follow
[Equation]
and ,
[Equation]
where σ f > 0 , z f ∈ R 1 × c and z f ∼ N (0 , I c × c ) .
Noise induced by SLN.
The label perturbation is a common technique (Xie et al., 2016), while we provide new insights by analyzing the effects from the lens of SGD noise.
Our SLN adds mean-zero
Gaussian noise to the one-hot labels, where the noise is independently sampled for each instance in each training step.
[Equation]
where σ y > 0 , z y ∈ R c and z y ∼ N (0 , I c × c ) .
Here, σ y z y is the SLN on the label y .
In above Eq.
(1-3), the standard deviation σ is a hyperparameter.
Since the SGD noise we introduce is i.i.d for each sample, we consider each single sample ( x, y ) independently in the following propositions.
For convenience, we use f , S to indicate the model output f ( x ; θ ) and the softmax output
S ( f ( x ; θ )) on a sample.
The proofs are presented in Appendix A.
Proposition 1.
Compared with Eq.
(1), Eq.
(2) induces noise z ∼ N (0 , σ 2 f M ) on ∇ θ (cid:96) ( f, y ) , s.t.,
M ∈ R p × p and M i,j = ( ∇ θ i f ) T ∇ θ j f , ∀ i, j ∈ { 1 , · · · , p } .
Note that the standard deviation of noise on the i -th parameter θ
i is σ f (cid:107)∇ θ i f (cid:107) 2 , where (cid:107)·(cid:107) 2 denotes the L 2 norm.
Proposition 2.
For the cross-entropy loss, compared with Eq.
(2), Eq.
(3) induces noise z ∼
N (0 , σ 2 y M ) on ∇ f (cid:96) ( f, y ) , s.t., M ∈ R c × c , M i,i =
c ( S i − 1 /c )
+ ( c − 1) /c , and M i,j = c · S i S j − S
i − S j , if i (cid:54) =
j .
Note that the standard deviation of noise on the i -th entry is σ y (cid:112) c
( S i − 1 /c )
+ ( c − 1) /c .
Proposition 3.
For the cross-entropy loss, compared with Eq.
(1), Eq.
(3) induces noise z ∼
N (0 , σ 2 y M ) on ∇ θ (cid:96) ( f, y ) , s.t., M ∈ R p × p and M i,j = ( ∇
θi S S ) T ∇ θj S S , ∀
i, j ∈ { 1 , · · · , p } , where · S denotes the element-wise division.
Note that the standard deviation of noise on the i -th parameter is
3. T HE EFFECTS OF SGD NOISE
Xie et al.
(2016) discuss the effect of random label perturbations as implicit ensemble.
In this paper, based on Proposition 1-3, we can show that SLN induces SGD noise with high variance when the output landscape is sharp or the conﬁdence of the output is high .
In this way, SLN helps escape from sharp minima and prevents overconﬁdence .
This provides intuition on the robustness of SLN under noisy labels, because 1) ﬂat minima typically generalize well (Hochreiter & Schmidhuber, 1997;
Keskar et al., 2017; Neyshabur et al., 2017); 2) preventing overconﬁdence can mitigate overﬁtting on noisy labels (Menon et al., 2020; Lukasik et al., 2020).
The SGD noise perturbs θ in each training step.
Thus, the training can not converge when the noise has high variance.
(i) For the most common spherical Gaussian noise shown in Eq.
(1), its standard deviation is a constant throughout training, independent of the landscape.
(ii) For Eq.
(2), Proposition 1 shows that the standard deviation of noise is σ
[Equation]
(cid:107)∇
[Equation]
can be very large around the sharp landscape, which means the SGD noise has high variance.
The high variance makes the training difﬁcult to converge, and helps escape from sharp minima.
Note that for training without SGD noise, it can converge to sharp minima because θ always follows the direction of gradient descent, whereas the direction of noise is random.
(iii)
For our SLN in Eq.
(3), Proposition 3 shows that the standard deviation of the SGD noise is σ y (cid:107)∇ θ i S/S (cid:107) 2 with ∇ θ i S in the numerator.
SLN similarly induces SGD noise with high variance around the sharp landscape.
Hence we have Claim 1.
(iv) Moreover, Proposition 2 shows that SLN induces SGD noise dependent on the conﬁdence of
S .
The standard deviation of noise on an entry of ∇ f (cid:96) ( f, y ) is minimized if S i = 1 /c , maximized if S i = 1 .
Proposition 3 directly characterizes the equivalent noise on θ , where there is a S in the denominator (element-wise division).
If S is conﬁdent, s.t., the entropy H ( S ) → 0 , which means an entry of S approaches 1 and others approach 0 , then the variance will be high since there are small numbers in the denominator.
Hence we have Claim 2.
Claim 1.
With SGD noise induced by Eq.
(2) or Eq.
(3), the training is difﬁcult to converge when the output ( f or S ) landscape is sharp.
Claim 2.
With SGD noise induced by Eq.
(3), the training is difﬁcult to converge when the output ( S ) is overconﬁdent,
s.t. H ( S ) → 0 , where H ( · ) is the entropy.
To illustrate sharp/ﬂat minima, we present visualizations of loss landscapes in Fig.
The ﬁgure also shows that the model trained with SLN converges to a solution that has small SGD noise.
More discussions are presented in Appendix E.
In terms of escaping from sharp minima, there are previous works (Zhu et al., 2019; Daneshmand et al., 2018) that study the inheret noise in SGD, showing that the noise covariance containing curvature information performs better.
They use the second-order approximation near the minima and apply integral to training steps to characterize the ability of escaping from sharp minima.
In this paper, we study the noise induced by SLN and draw a more direct intuition between the noise covariance and the ability of escaping from sharp minima.
Moreover, the dependency between the noise induced by SLN and the conﬁdence of output probability further provides intuition on avoiding overﬁtting under noisy labels.
In terms of preventing overconﬁdence, we shall discuss label smoothing (LS) (Lukasik et al., 2020).
It smooths the given one-hot label y into a soft one ˜ y , s.t., ˜ y
=
(1 − α )
y + αe/c , where e is an all-one vector and α > 0 .
In this way, LS introduces a ﬁxed and biased perturbation on the label, whereas SLN introduces dynamic and mean-zero perturbations.
LS does not introduce noise in each training step, while SLN adaptively perturbs the parameter once the landscape is sharp or the prediction is overconﬁdent.
Hence, the robustness of SLN may not result from preventing overconﬁdence alone, but also escaping from sharp minima.
In Fig.
3, we plot the sample density w.r.t.
predictions on the labeled class, using CIFAR-10 with 40% symmetric noise as an example.
It shows that SLN does reduce overconﬁdence, and it mostly affects noisy samples.
3. A DETAILED DISSECTION ON PREDICTIONS OF DNN S TRAINED WITH SLN
In this section, we show that SLN boosts popular robust training methods, including sampleselection and label-correction.
Many methods, demonstrated to work well, select or add higher
[Figure] [Figure]
[Figure] [Figure]
weight on small-loss samples (Han et al., 2018b; Jiang et al., 2018; Li et al., 2020), or use the model’s predictions to correct noisy labels (Tanaka et al., 2018; Arazo et al., 2019).
A warm-up phase is usually required to initialize the model before sample-selection or label-correction, yet the model will memorize noise if the warm-up phase is too long.
With SLN, we can simply train the model until convergence.
In Fig.
4, we compare converged models on CIFAR-10, where the model is trained with SLN or merely CE loss.
The detailed noise setting can be found in Section 4.
We ﬁrst sort training samples in ascending order of loss, then uniformly divide them into 1000 samples per interval, and ﬁnally obtain the number of four types of samples in each interval based on the correctness of the given label and the prediction.
When trained with CE, there are many small-loss samples with incorrect labels (the blue region), then selecting small-loss samples is not reliable.
The model trained with SLN largely address the issue.
Moreover, SLN is suitable for label-correction since it yields correct predictions for many originally incorrect samples (the orange region).
As a concrete example, we present an enhanced algorithm by applying SLN to label-correction.
With SLN, we train the model for sufﬁcient epochs until convergence, without the need of carefully tuning a warm-up phase.
Then we start label-correction using y correction =
ω · y + (1 − ω ) · S , where S is the softmax prediction, ω ∈ [0 , 1] is the weight obtained by normalizing the training loss, s.t., for the i -th training sample, ω
i =
( (cid:96) i − (cid:96) min ) /
( (cid:96) max − (cid:96) min ) .
More discussions on the label-correction are presented in Appendix D.
E XPERIMENTS 4. T HE OVERVIEW OF EXPERIMENTAL SETUP
We comprehensively verify the utility of SLN on different types of label noise, including symmetric noise, asymmetric noise, instance-dependent noise and open-set noise synthesized on CIFAR-10 and
CIFAR-100, and real-world noise on Clothing1M (Xiao et al., 2015).
Symmetric noise assumes each label has the same probability of ﬂipping to any other class.
We uniformly ﬂip the label to other classes with an overall probability 40% . 2)
Asymmetric noise contains noisy labels ﬂipped between similar classes.
Following Zhang & Sabuncu (2018), on CIFAR-10, we ﬂip labels between
TRUCK → AUTOMOBILE, BIRD → AIRPLANE, DEER → HORSE, and CAT ↔ DOG, with a probability 40% ; on CIFAR-100, we ﬂip each class into the next class circularly with a probability 40% .
3) Instance-dependent noise assumes that the mislabeling probability should be dependent on each instance’s input features (Xia et al., 2020).
We ﬁrst train a model on clean data, then ﬂip labels of 40% samples to their second-conﬁdent class in the predictions.
4) Open-set noise contains samples that do not belong to any class considered in the classiﬁcation task.
Following Wang et al.
(2018), we yield CIFAR-10 with open-set noise by randomly replacing 40% of its training images with images from CIFAR-100.
For real-world noise, we use the large-scale benchmark Clothing1M, which contains 1 M training images with noisy labels from online shops.
We use the backbone wide ResNet-28-2 (Zagoruyko & Komodakis, 2016) on CIFAR-10 and
CIFAR-100, and ResNet-50 on Clothing1M. More training details can be found in Appendix B.
Apart from SLN, we introduce a momentum (MO) model (Tarvainen & Valpola, 2017) and labelcorrection (LC), with which we get SLN-MO and SLN-MO-LC.
Parameters of the momentum model ( θ (cid:48) ) is updated as a moving average of the the training model ( θ ).
At t -th training step, θ (cid:48) ( t ) =
αθ (cid:48)
( t − 1) + (1 − α )
θ ( t ) .
We use α = 0 .
999 in all experiments.
For SLN-MO-LC, we have discussed in Section 3.3 that SLN is reliable when applied to label-correction.
4. T HE EFFECTS OF SGD NOISE VARIANTS
We ﬁrst show that compared with other variants, SLN stands out in mitigating the effects of label noise.
The SGD noise variants have been formalized in Eq.
(1-3), including z θ directly added to ∇ θ (cid:96) , z f on ∇ f (cid:96) and our SLN ( z y ).
For SLN, the standard deviation is σ = 1 under symmetric label noise and σ = 0 .
5 in all other cases.
For other SGD noise, the standard deviation is equally tuned, and the performance under three different σ is separately shown in Fig.
SLN signiﬁcant improves the generalization, while other SGD noise variants can not achieve such impressive performance even if σ is heavily tuned.
It is worth noting that for all these variants, when σ is too small, the model overﬁts noise since the training is similar to merely using CE loss; when σ is too large, the model fails to ﬁt the training data since the SGD noise is too high.
[Figure]
[Table] [Table]
[Table]
GCE
[Table]
Co-Teaching
[Table]
PHuber-CE
[Table]
[Figure]
[Table] [Table]
[Figure]
[Table]
[Figure]
MO
[Table]
[Figure]
[Table]
[Figure]
LC
[Table]
[Figure]
Zhang &
The ﬁgure implies we may achieve better results with a more detailed hypermarameter search for σ .
In Appendix C.2, we further illustrate the case of overﬁtting/underﬁtting with small/large σ .
4. CIFAR-10 AND CIFAR-
On CIFAR-10 and CIFAR-100, we compare with the following baselines: 1) only crossentropy (CE) loss, 2) Generalized Cross-Entropy (GCE) (Zhang & Sabuncu, 2018) loss, 3)
CoTeaching (Han et al., 2018b) that uses co-training and sample-selection, 4)
PHuber-CE
(Menon et al., 2020) that uses gradient clipping and 5) label-smoothing (LS) Lukasik et al.
(2020) that clips the label to be less conﬁdent before training.
We use 5 k noisy samples as the validation to tune hyperparameters, then train the model on the full training set and report the test accuracy at the last epoch.
SLN simply requires tuning the standard deviation σ , which is tuned in { 0 . 1 , 0 . 2 , 0 .
On
CIFAR-10, the best σ is 1 under symmetric noise and 0.5 otherwise; On CIFAR-100, it is 0.1 under instance-dependent noise and 0.2 otherwise.
The label-correction in SLN-MO-LC is applied in the last 50 epochs.
The softmax prediction is converted into one-hot label in correction.
We repeat each experiment 5 times.
The average test accuracy at the last epoch is reported in Table 1 and Table 2.
To illustrate the inﬂuence of σ , an ablation study is presented in Fig.
In the tables, we mark the top-3 results in bold and present the average training time of each method, evaluated on a single
[Table]
V100 GPU.
Without the momentum model and label-correction
[Table]
vanilla SLN achieves impressive test performance, which is consistent with results in Fig.
In Section 3.
[Table]
we have analyzed that
SLN can boost popular robust training methods, based on a detailed dissection on DNNs’ predictions.
As expected, we obtain further improvement with SLN-MO and SLN-MO-LC.
Notably, SLN,
SLN-MO and SLN-MO-LC sweep the top-3 results in almost all cases and require low computational cost.
We do not expect vanilla SLN to achieve state-of-the-art performance compared with many integrated methods.
Still, SLN can be a promising option in the family of robust learning methods.
In Appendix C.1, we show that SLN can improve many existing methods.
4. C LOTHING 1M
Clothing1M (Xiao et al., 2015) is a large-scale benchmark of clothing images from online shops with 14 classes, containing real-world label noise.
It has 1 million noisy samples for training, 14 k and 10 k clean samples for validation and test.
The number of images labeled as each class is unbalanced, ranging from 18976 to 88588 in the noisy training set.
In previous works, some experiments are conducted by sampling a class-balanced training subset in each epoch (Li et al., 2020), while others directly train on the full training set (Patrini et al., 2017).
Since the balanced training sampling itself affects the test performance, it is difﬁcult to compare results across papers.
Therefore, we make it clear here and conduct experiments in both setting, including the standard sampling and the noisy-class-balanced sampling.
For the latter, in each epoch, 18976 instances per class are randomly sampled from the noisy training set.
Other training details strictly follow the standard benchmark setting (Patrini et al., 2017), presented in Appendix B.
We set the standard deviation of SLN as σ = 0 .
For SLN-MO-LC, the label-correction is applied since the ﬁrst epoch.
Results are listed in
Table 3, with the best result in bold and previous published results marked by a star.
DivideMix trains two models in each run and we average their test accuracy, rather than using additional ensemble of two models.
Our SLN outperforms many baselines.
The variants SLN-MO and SLN-MO-LC further achieve higher test accuracy.
Speciﬁcally, SLN-MO-LC achieves the best test accuracy in both settings.
Our methods also stand out for training efﬁciency.
C ONCLUSION
In this paper, we establish that SLN induces SGD noise dependent on the sharpness of output landscape and the conﬁdence of output probability, and analyze the effects of escaping from sharp local minima and preventing overconﬁdence.
This partially explains the robustness of SLN under noisy labels since various works show that ﬂat minima typically generalize well and preventing overconﬁdence helps mitigate overﬁtting on noisy labels.
We empirically verify the robustness of SLN under various synthetic label noise and real-world noise.
Moreover, we show that SLN boosts popular robust training methods, including sample-selection and label-correction.
In particular, we justify that SLN can enhance existing methods based on a detailed dissection on DNNs’ predictions, then present a practical algorithm by applying SLN to label-correction.
A P ROOFS
Proposition 1.
Compared with Eq.
(1), Eq.
(2) induces noise z ∼ N (0 , σ 2 f M ) on ∇ θ (cid:96) ( f, y ) , s.t.,
M ∈ R p × p and M i,j = ( ∇ θ i f ) T ∇ θ j f , ∀ i, j ∈ { 1 , · · · , p } .
Note that the standard deviation of noise on the i -th parameter θ
i is σ f (cid:107)∇ θ i f (cid:107) 2 , where (cid:107)·(cid:107) 2 denotes the L 2 norm.
[Equation]
For Eq.
(2), the noisy gradient is
[Equation]
y
[Equation]
+ σ
[Equation]
The noise on ∇ θ (cid:96) ( f, y ) is z = σ f z f · ∇ θ f ∈ R 1 × p .
Note that z f ∼ N (0 , I c × c ) , let z i be the i -th entry of z , we have z i
[Equation]
Hence,
[Equation] [Equation]
Proposition 2.
For the cross-entropy loss, compared with Eq.
(2), Eq.
(3) induces noise z ∼
N (0 , σ 2 y M ) on ∇ f (cid:96) ( f, y ) , s.t., M ∈ R c × c , M i,i =
c ( S i − 1 /c )
+ ( c − 1) /c , and M i,j = c · S i S j − S
i − S j , if i (cid:54) =
j .
Note that the standard deviation of noise on the i -th entry is σ y (cid:112) c
( S i − 1 /c )
+ ( c − 1) /c .
Proof.
[Equation]
Firstly, for the softmax function S = S ( f ( x )) , we have the derivative matrix
[Equation]
where Diag( S ) is a diagonal matrix with S i on its i -th diagonal element and 0 otherwise.
For the cross-entropy loss, (cid:96) ( f, y ) =
− (cid:80) ck
=1 y k log S k .
Let · S denote element-wise division, we have
[Equation]
Then it is equivalent to induce noise z = − σ y ( z y − (cid:80) ck
=1 z y k · S ) on ∇ f (cid:96) ( f, y ) , whose i -th entry is z i = − σ y ( z y i − (cid:80) ck
=1 z y k · S i ) .
Note that z y ∼ N (0 , I c × c ) , then we can derive for i = j ,
E [ z
[Equation]
i ]
= σ
[Equation]
y (
[Equation]
S i + c · S 2 i ) =
[Equation]
y (
[Equation] [Equation]
[Equation]
) , (9) and for i (cid:54) =
j ,
E [ z
[Equation]
z
[Equation]
] = σ
[Equation]
( c
[Equation]
S
[Equation]
− S
[Equation]
− S
[Equation]
Proposition 3.
For the cross-entropy loss, compared with Eq.
(1), Eq.
(3) induces noise z ∼
N (0 , σ 2 y M ) on ∇ θ (cid:96) ( f, y ) , s.t., M ∈ R p × p and M i,j = ( ∇
θi S S ) T ∇ θj S S , ∀
i, j ∈ { 1 , · · · , p } , where · S denotes the element-wise division.
Note that the standard deviation of noise on the i -th parameter θ
i is σ y (cid:13)(cid:13)(cid:13) ∇ θi S S (cid:13)(cid:13)(cid:13) 2 .
Proof.
Dimension: z y ∈ R c , ∇ θ (cid:96) ∈ R 1 × p , ∇ S (cid:96)
∈ R 1 × c , ∇ θ S ∈ R c × p , ∇ θ i S ∈ R c .
For Eq.
(3), the noisy gradient is
[Equation]
The noise on ∇ θ (cid:96) ( f, y ) is z = − (cid:0) σ y z y S (cid:1) T · ∇ θ S .
Note that z y ∼ N (0 , I c × c ) , let z i be the i -th entry of z , we have
[Equation]
Hence,
[Equation]
B M ORE DETAILS ON THE TRAINING SETTINGS B. CIFAR-10 AND CIFAR-
The backbone and general training hyperparameters.
In all experiments on CIFAR-10 and
CIFAR-100, we train wide ResNet-28-2 (Zagoruyko & Komodakis, 2016) for 300 epochs using the SGD optimizer with learning rate 0 .
001 , momentum 0 .
9 , weight decay 5 × 10 − 4 , and a batchsize of 128 .
Standard data augmentation is applied, including per-pixel normalization, horizontal random ﬂip and 32 × 32 random crop after padding with 4 pixels on each side.
The criterion for setting the training hyperparameters includes 1) all methods should converge (the training accuracy converges), 2) all methods share the same general training hyperparameters for a fair comparison.
Method-speciﬁc hyperparameters.
The backbone is not uniﬁed in previous papers and we reimplement all methods in the same backbone for a fair comparison.
Regarding this, we may not directly follow the default hyperparameters.
Following Zhang & Sabuncu (2018), we use 5 k noisy samples ( 10% of the training data) as the validation set to tune method-speciﬁc hyperparameters.
We then train the model on the full training set and report the test accuracy at the last epoch.
• SLN/SLN-MO/SLN-MO-LC (ours).
We tune σ ∈ { 0 . 1 , 0 . 2 , 0 . 5 , 1 } .
On CIFAR-10, we use σ = 1 for symmetric noise and σ = 0 .
5 otherwise; On CIFAR-100, we use σ = 0 .for
instance-dependent noise and σ = 0 .
2 otherwise.
The momentum model is introduced with hyperparemeter 0 .
999 without tuning.
The label-correction (LC) is applied after convergence of training with SLN.
All models are trained for 300 epochs and we introduce LC at the 250 th epoch without tuning beacuse the training accuracy does not increase much after the 250 th epoch.
In this way, we do not increase the computation cost.
• GCE (Zhang & Sabuncu, 2018).
GCE loss is applied as the training starts and there is a warm epoch after which truncated GCE loss is applied every 10 epochs.
We tune the warm epoch in { 0 , 50 , 100 , 150 , 200 } and use 50 for CIFAR-10, 150 for CIFAR-100.
There is a hyperparameter q for the GCE loss.
We set q = 0 .
7 since it is used in all experiments on CIFAR-10 and CIFAR-100 in its original paper (Zhang & Sabuncu, 2018).
• Co-Teaching (Han et al., 2018b).
The rate of selecting small-loss samples is linearly decreased from 1 to 1 − ε at the ﬁrst 10 epochs, where ε is the noise rate.
This setting is used in all experiments in the original paper (Han et al., 2018b) and it works well in our setting.
• PHuber-CE (Menon et al., 2020).
There is a hyperparameter τ that controls the gradient clipping.
The original paper (Menon et al., 2020) uses τ = 2 on CIFAR-10 and τ = 10 on CIFAR-100, but the default setting does not work well in our experiments.
Hence, we tune τ ∈ { 2 , 5 , 10 , 30 , 50 } and ﬁnally, on CIFAR-10, we use τ = 10 for asymmetric noise and τ = 2 otherwise; on CIFAR-100, we use τ = 30 .
• LS (Lukasik et al., 2020).
There is a hyperparameter α that controls how much the label is smoothed.
We tune α ∈ { 0 . 2 , 0 .
5 , 0 . 8 } and ﬁnally, on CIFAR-10, we use α = 0 .
5 for asymmetric noise and α = 0 .
8 otherwise; on CIFAR-100, we use α = 0 .
• SIGUA (Han et al., 2020).
The hyperpramemeter γ is a factor that is multiplied on the loss on ‘bad’ samples.
We tune it in { 0 . 01 , 0 .
and ﬁnally use γ = 0 .
001 in all experiments.
• DivideMix (Li et al., 2020).
We tune the warm-up epoch and λ u - the weight for unsupervised loss.
In the ofﬁcial implementation, the warm-up epoch is 10 on CIFAR-10 and 30 on CIFAR-100.
The default hyperparameters do not work well in our experiments (we observe a decrease of test accuracy after warm-up).
Hence we tune the warm-up epoch in { 10 , 30 , 50 , 100 } but the performance does not improve compared with the default settings.
Hence we use the default warm-up and tune λ u ∈ { 0 , 1 , 5 , 10 , 25 } .
Finally, we obtain impressive results with λ u = 0 in all experiments.
B. C LOTHING 1M
The backbone and general training hyperparameters.
On Clothing1M, following the common setting (Patrini et al., 2017), we train an Imagenet-pretrained ResNet-50 using the SGD optimizer with momentum 0 .
9 , weight decay 10 − 3 and batchsize 32 .
The initial learning rate is 10 − 3 and decreased to 10 − 4 after 5 epochs.
We use standard data augmentation with per-pixel normalization, horizontal random ﬂip and 224 × 224 random crop.
Note that DivideMix uses the same backbone
ResNet-50 but a different training schedule, and we follow its ofﬁcial implementation released on the GitHub.
Method-speciﬁc hyperparameters
Since the backbone ResNet-50 is used by default for most published results (Patrini et al., 2017; Li et al., 2020), we can easily follow the default hyperparameters.
• SLN/SLN-MO/SLN-MO-LC (ours).
Following previous methods (Patrini et al., 2017; Li et al., 2020), the validation set cobtaining 14 k clean samples is adopted to tune our hyperparameters.
We tune σ ∈ { 0 . 1 , 0 . 2 , 0 . 5 } and choose σ = 0 .
The momentum model is implemented with hyperparemeter 0 .
999 without tuning.
We ﬁx the overall training epoch as 10 and tune the epoch for applying label-correction in { 1 , 5 , 9 } .
Finally, we apply label-correction after the ﬁrst epoch in all experiments.
• Forward/Backword
(Patrini et al., 2017).
The results is reproduced by reimplementing the method exactly following hyperparameters in Patrini et al.
• Co-Teaching (Han et al., 2018b).
On Clothing1M, the estimated noise rate is around 0 .
4 (Xiao et al., 2015).
Hence, we linearly reduce the rate of selecting small-loss samples from 1 to 0 .
6 in 10 epochs.
• DivideMix (Li et al., 2020).
The result is reproduced from its ofﬁcial implementation.
C M ORE EMPIRICAL RESULTS AND DISCUSSIONS C. SLN ENHANCES EXISTING METHODS
With a detailed dissection on predictions of DNNs trained with SLN (Section 3.3), we have shown that SLN can boost popular robust training methods such as label-correction and sample-selection.
In this section, we verify this by integrating SLN with the following methods.
• Co-teaching (Han et al., 2018b).
It uses co-training and sample-selection.
Two modes select small-loss samples to train each other.
• Stochastic integrated gradient underweighted ascent (SIGUA) (Han et al., 2020).
It adopts gradient descent on good data as usual, and learning-rate-reduced gradient ascent on bad data.
• DivideMix
(Li et al., 2020).
It combines co-training of two models, sample-selection based on the loss, label-correction/guessing based on semi-supervised learning MixMatch (Berthelot et al., 2019), and other techniques including regularization, augmenting each image twice.
[Table]
[Figure]
Speciﬁcally, we use models trained with SLN as the initialization of these methods.
As shown in
Table 4, all methods obtain consistent improvement when integrated with SLN.
Moreover, with SLN as initialization, we do not need to tune the warm-up phase for methods like DivideMix, because we can train with SLN until convergence.
In contrast, without SLN, we need to carefully warm up the model so that it learns enough correct patterns and does not memorize too much noise.
Note that better results for DivideMix are reported in the original paper with a different backbone and carefully scheduled learning rate.
We focus on fairly comparing the robustness of all methods: in all experiments, we train the same backbone wide ResNet-28-2 for 300 epochs without learning rate change.
Detailed training settings are presented in Appendix B.
C. T -SNE VISUALIZATION OF FEATURES ON THE MODEL ’ S PENULTIMATE LAYER
In Fig.
7, we show the t-SNE visualization of features on the model’s penultimate layer, taking all training samples as input.
We visualize the embedding on CIFAR-10 with symmetric noise since the noise yields the most severe damage to the generalization and SLN provides a signiﬁcant improvement.
Fig.
7 shows that the model trained with SLN can yield a better embedding.
It also demonstrates overﬁtting and underﬁtting when σ is too small or too large.
[Figure]
[Figure]
C. T EST ACCURACY W . R . T . TRAINING EFFICIENCY
In Fig. 8, we visualize the training efﬁciency and test accuracy.
The ﬁgure clearly illustrates the superior generalization performance and high efﬁciency of our methods.
D M ORE DISCUSSIONS ON LABEL - CORRECTION
The convergence issue in label-correction.
When using the model’s prediction to correct noisy labels, we ﬁnd a convergence issue such that the test accuracy decreases, as shown in Fig.
The convergence issue is also reported in Arazo et al.
, but it has not been widely discussed.
In this section, we provide an intuitive hypothesis.
We can analyze the effect of label-correction on the four types of samples, as shown in Fig. 9.
• 1) label × , prediction × .
Label-correction can not correct these samples, yet may even make the case worse because the prediction error should be easier for the model to overﬁt compared to given noisy labels.
• 2) label × , prediction (cid:88) .
These samples beneﬁt from label-correction.
• 3) label (cid:88) , prediction × .
Label-correction is harmful for these samples.
• 4) label (cid:88) , prediction (cid:88) .
Label-correction does not signiﬁcantly impact these samples because both the prediction and the label are correct.
In a word, there exist samples of case 1) and case 3) that are not desired in label-correction, but they are unavoidable since we do not expect a model with 100% prediction accuracy.
In label-correction, when trained with modiﬁed labels obtained from wrong predictions, the model may accumulate its own error due to a positive feedback: yielding worse predictions after training on prediction errors.
Therefore, we hypothesize that samples of case 1) and case 3) result in the convergence issue.
How do we assign weights in label-correction?
We use y correction =
ω · y + (1 − ω ) · S , as opposed to y correction =
(1 − ω ) · y + ω · S , where ω is an instance-dependent weight positive correlated with the loss.
We observe that our scheme mitigates the convergence issue, as shown in Fig.
Our intuition that - samples that need correction have large losses - is consistent with the method of Arazo et al.
, but this does not mean that the weight on S should be positive correlated with the loss.
We can consider the following cases.
• For small-loss samples, we have S ≈ y , as illustrated by the red region in Fig.
Labelcorrection does not affect these samples much regardless of the weight .
Hence, we simply need to consider the effects of label-correction on large-loss samples.
• Large-loss samples of case 2) can beneﬁt from label-correction, as illustrated by the orange region in Fig.
However, in this case, a higher loss does not mean that it requires a higher weight on S for label-correction 3 .
• There exist large-loss samples of case 1) and case 3) for which label-correction can be harmful , as has been discussed in the above paragraph and illustrated by the blue and green regions in Fig.
Therefore, we assign a small weight of 1 − ω on S for large-loss samples
In
this
way
, samples of case 2) still beneﬁt from label-correction, while we mitigate the undesired effects on other large-loss samples for which label-correction can be harmful.
The effectiveness of our scheme is veriﬁed in
Fig.
E
M ORE DISCUSSIONS ON THE CONVERGENCE
In Fig.
the visualizations of loss landscapes show that the model trained with SLN converges to a solution that has small SGD noise.
The center point on the visualized landscape (i.e., the loss of the given model) is a local minimum.
From Fig.
2 (d), we observe that the minimum has the following properties
• The gradient around the minimum is small since it is ﬂat.
• The predictions do not approach one-hot labels because the loss at the local minimum is high.
As shown in Fig.
3, the prediction probabilities are much lower than 1.
With the above two properties
Proposition 3 implies that around the ﬂat minimum illustrated in
Fig.
2 (d), the noise on gradients is small.
Therefore, the model converges in the local ﬂat minimum.