N OISE AGAINST NOISE : STOCHASTIC LABEL NOISE HELPS COMBAT INHERENT LABEL NOISE
Anonymous authors
Paper under double-blind review
A BSTRACT
The noise in stochastic gradient descent (SGD) provides a crucial implicit regularization effect, previously studied in optimization by analyzing the dynamics of parameter updates.
In this paper, we are interested in learning with noisy labels, where we have a collection of samples with potential mislabeling.
We show that a previously rarely discussed SGD noise, induced by stochastic label noise (SLN), mitigates the effects of inherent label noise.
In contrast, the common SGD noise directly applied to model parameters does not.
We formalize the differences and connections of SGD noise variants, showing that SLN induces SGD noise dependent on the sharpness of output landscape and the conﬁdence of output probability, which may help escape from sharp minima and prevent overconﬁdence.
SLN not only improves generalization in its simplest form but also boosts popular robust training methods, including sample-selection and label-correction.
Speciﬁcally, we present an enhanced algorithm by applying SLN to label-correction.
I NTRODUCTION
The existence of label noise is a common issue in classiﬁcation since real-world samples unavoidably contain some noisy labels, resulting from annotation platforms such as crowdsourcing systems (Yan et al., 2014).
In the canonical setting of learning with noisy labels, we collect samples with potential mislabeling, but we do not know which samples are mislabeled since true labels are unobservable.
It is troubling that overparameterized Deep Neural Networks (DNNs) have sufﬁcient capacity to memorize noise (Zhang et al., 2017), leading to poor generalization performance.
Thus, we are urgent for robust training methods that can mitigate the effects of label noise.
The noise in stochastic gradient descent (SGD) (Wu et al., 2020) provides a crucial implicit regularization effect for training overparameterized models.
SGD noise is previously studied in optimization by analyzing the dynamics of parameter updates, whereas its utility in learning with noisy labels has not been explored to the best of our knowledge.
In this paper, we ﬁnd that the common SGD noise directly applied to model parameters does not endow much robustness, whereas a variant induced by controllable label noise does.
Interestingly, inherent label noise is harmful to generalization, while we can mitigate its effects using additional controllable label noise.
To prevent confusion, we use stochastic label noise (SLN) to indicate the label noise we introduce.
Inherent label noise is biased and unknown, ﬁxed when the data is given.
SLN is mean-zero and independently sampled for each instance in each training step.
Our main contributions are as follows.
• We formalize the differences and connections of three SGD noise variants (Proposition 13), and show that SLN induces SGD noise that is dependent on the sharpness of output landscape and the conﬁdence of output probability.
• Based on the noise covariance, we analyze two effects of SLN (Claim 1 and Claim 2): escaping from sharp minima 1 and preventing overconﬁdence 2 .
• We empirically show that SLN not only improves generalization in its simplest form but also boosts popular robust training methods, including sample-selection and labelcorrection.
We present an enhanced algorithm by applying SLN to label-correction.
[Figure]
In Fig.
1, we present a quick comparison between models trained with/without SLN on CIFARwith symmetric/asymmetric/instance-dependent/open-set label noise.
Throughout this paper, we use
CE to indicate a model trained with cross-entropy (CE) loss without any robust learning techniques, and the canonical CE loss is used by default for methods like SLN.
In Section 4, we will provide more experimental details and more results that comprehensively verify the robustness of SLN on different synthetic noise and real-world noise.
Here, the test curves in Fig.
1 show that SLN avoids the drop of test accuracy, with converged test accuracy even higher than the peak accuracy of the model trained with CE.
The right two subplots in Fig.
1 show the average loss on clean and noisy samples.
When trained with CE, the model eventually memorizes noise, indicated by the drop of average loss on noisy samples.
In contrast, SLN largely avoids ﬁtting noisy labels.
R ELATED W ORK 2. SGD N OISE IN O PTIMIZATION
The noise in SGD (Wu et al., 2020; Wen et al., 2019; Keskar et al., 2016) has long been studied in optimization.
It is believed to provide a crucial implicit regularization effect (HaoChen et al., 2020; Arora et al., 2019; Soudry et al., 2018) for training overparameterized models.
To study SGD noise, the most common way is explicitly adding spherical Gaussian noise to model parameters in each training step (Ge et al., 2015; Neelakantan et al., 2015; Mou et al., 2018).
However, empirical studies (Wen et al., 2019; Shallue et al., 2019) demonstrate that parameter-dependent SGD noise is far more effective.
It is shown that the noise covariance containing curvature information performs better for escaping from sharp minima (Zhu et al., 2019; Daneshmand et al., 2018).
On a quadratically-parameterized model introduced by Vaskevicius et al.
and Woodworth et al.
(2020), HaoChen et al.
(2020) prove that in an over-parameterized regression setting, SGD with label perturbations recovers the sparse groundtruth, whereas SGD with Gaussian noise directly added on gradient descent overﬁts to dense solutions.
In the deep learning scenario, HaoChen et al.
(2020) present primary empirical results for a VGG19 model (Simonyan & Zisserman, 2014) on CIFAR-100, showing that SGD noise - induced by Gaussian noise on the gradient of the loss w.r.t. the model’s output - avoids performance degeneration of large-batch training.
To the best of our knowledge, the utility of SGD noise in learning with noisy labels has not been studied.
2. R OBUST T RAINING M ETHODS
Mitigating the effects of label noise is a vital topic in classiﬁcation, which has a long history (Ekholm & Palmgren, 1982; Natarajan et al., 2013) and attracts much recent interest with several directions explored.
1) Malach & Shalev-Shwartz (2017); Han et al.
(2018b);
Yu et al.
; Chen et al.
(2019a)
; Wei et al.
propose sample-selection methods that train on trusted samples, identiﬁed according to training loss, cross-validation or (dis)agreement between two models.
2) Liu & Tao (2015); Jiang et al.
; Ren et al.
Shu et al.
; Li et al.
(2019a) develop sample-weighting schemes that aim to add higher weights on clean samples.
Reed et al.
Sukhbaatar et al. (2015); Patrini et al.
; Han et al.
(2018a) apply loss-correction based on an estimated noise transition matrix.
4) Tanaka et al.
Arazo et al.
; Zheng et al.
(2020) propose label-correction based on the model’s predictions.
5) Ghosh et al.
(2017); Zhang & Sabuncu (2018)
; Xu et al.
; Charoenphakdee et al.
; Lyu & Tsang (2020); Ma et al.
(2020) study robust loss functions that have a theoretical guarantee for noisy risk minimization, typically with the assumption that the noise is class-conditional (Scott et al., 2013; Natarajan et al.,
Chen et al.
(2019b)
; Menon et al.
Hu et al.
Harutyunyan et al.
Lukasik et al.
(2020) apply regularization techniques to improve generalization under label noise, including explicit regularizations such as manifold regularization (Belkin et al., 2006) and virtual adversarial training (Miyato et al., 2018), and implicit regularizations such as dropout (Srivastava et al., 2014), temporal ensembling (Laine & Aila, 2017), gradient clipping (Pascanu et al., 2012;
Zhang et al., 2019; Menon et al., 2020) and label smoothing (Szegedy et al., 2016).
7) One can combat label noise with reﬁned training strategies (Li et al., 2019b; 2020; Nguyen et al., 2020) that potentially incorporate several techniques, including sample-selection/weighting, label-correction, meta-learning (Li et al., 2019b) and semi-supervised learning (Tarvainen & Valpola, 2017; Berthelot et al., 2019).
Among these methods, regularization techniques are closely related to the essence of training networks, and studying robustness under label noise provides a new lens of understanding the regularization apart from the optimization lens.
M ETHOD 3. T HE DIFFERENCES AND CONNECTIONS OF SGD NOISE VARIANTS
Notations.
Let D =
( x ( i ) , y ( i ) ) }
ni =1 be a dataset with noisy labels.
For each sample ( x, y ) , its label y may be incorrect and the true label is unobservable.
Let f ( x ; θ ) be the neural network model with trainable parameter θ ∈ R p .
For a c -class classiﬁcation problem, we have the output f ( x ; θ ) ∈ R c .
We use a softmax function S ( f ( x ; θ ))
∈ R c to obtain the probability of each class.
The loss on a sample is denoted as (cid:96) ( f, y ) .
For classiﬁcation, we use the cross-entropy (CE) loss by default.
In parameter updates, a sample contributes ∇ θ (cid:96) ( f, y ) to the gradient descent.
With SGD noise, the model is trained with a noisy gradient ˜ ∇ θ (cid:96) ( f, y ) .
Following the standard notation of the Jacobian matrix, we have ∇ θ (cid:96) ∈ R 1 × p , ∇ f (cid:96) ∈ R 1 × c , ∇ θ f ∈ R c × p , ∇ θ i f ∈ R c and ∇ θ f i ∈ R 1 × p .
Noise on the gradient of loss w.r.t.
parameters.
The most common SGD noise is the spherical Gaussian noise directly added to the gradient to parameters (Neelakantan et al., 2015) as follow,
[Equation]
where σ θ > 0 , z θ ∈ R 1 × p and z θ ∼ N (0 , I p × p ) .
Noise on the gradient of loss w.r.t.
the model output.
Taking a step further, HaoChen et al.
(2020) study SGD noise induced by label noise on a quadratically-parameterized regression model, whereas for classiﬁcation, they add mean-zero noise to ∇ f (cid:96) ( f, y ) as follow
[Equation]
and
[Equation]
where σ f > 0 , z f ∈ R 1 × c and z f ∼ N (0 , I c × c ) .
Noise induced by SLN.
We induce SGD noise by label perturbations, namely the stochastic label noise (SLN), which is independently sampled for each instance in each training step.
[Equation]
where σ y > 0 , z y ∈ R c and z y ∼ N (0 , I c × c ) .
Here, σ y z y is the SLN on the label y .
In above Eq.
(1-3), the standard deviation σ is a hyperparameter.
Since the SGD noise we introduce is i.i.d for each sample, we consider each single sample ( x, y ) independently in the following propositions.
For convenience, we use f , S to indicate the model output f ( x ; θ ) and the softmax output S ( f ( x ; θ )) on a sample.
The proofs are presented in Appendix A.
Proposition 1.
Compared with Eq.
(1), Eq.
(2) induces noise z ∼ N (0 , σ 2 f M ) on ∇ θ (cid:96) ( f, y ) , s.t.,
M ∈ R p × p and M i,j = ( ∇ θ i f ) T ∇ θ j f , ∀ i, j ∈ { 1 , · · · , p } .
Note that the standard deviation of noise on the i -th parameter θ
i is σ f (cid:107)∇ θ i f (cid:107) 2 , where (cid:107)·(cid:107) 2 denotes the L 2 norm.
Proposition 2.
For the cross-entropy loss, compared with Eq.
(2), Eq.
(3) induces noise z ∼
N (0 , σ 2 y M ) on ∇ f (cid:96) ( f, y ) , s.t., M ∈ R c × c , M i,i =
c ( S i − 1 /c )
+ ( c − 1) /c , and M i,j = c · S i S j − S
i − S j , if i (cid:54) =
j .
Note that the standard deviation of noise on the i -th entry is σ y (cid:112) c
( S i − 1 /c )
+ ( c − 1) /c .
Proposition 3.
For the cross-entropy loss, compared with Eq.
(1), Eq.
(3) induces noise z ∼
N (0 , σ 2 y M ) on ∇ θ (cid:96) ( f, y ) , s.t., M ∈ R p × p and M i,j = ( ∇
θi S S ) T ∇ θj S S , ∀
i, j ∈ { 1 , · · · , p } , where · S denotes the element-wise division.
Note that the standard deviation of noise on the i -th parameter θ
i is σ y (cid:13)(cid:13)(cid:13) ∇ θi S S (cid:13)(cid:13)(cid:13) 2 . 3. T HE EFFECTS OF SGD NOISE Based on Proposition 1-3, we can show that SLN induces SGD noise with high variance when the output landscape is sharp or conﬁdence of the output is high . In this way, SLN helps escape from sharp minima and prevents overconﬁdence .
This provides intuition on the robustness of SLN under noisy labels, because 1) ﬂat minima typically generalize well (Hochreiter & Schmidhuber, 1997; Keskar et al., 2017; Neyshabur et al., 2017); 2) preventing overconﬁdence can mitigate overﬁtting on noisy labels (Menon et al., 2020; Lukasik et al., 2020). The SGD noise perturbs θ in each training step. Thus, the training can not converge when the noise has high variance.
(i) For the most common spherical Gaussian noise shown in Eq.
(1), its standard deviation is a constant throughout training, which means we can not use a too large σ for convergence. (ii) For Eq. (2), Proposition 1 shows that the standard deviation of noise is
[Equation]
can be very large around the sharp landscape, which means the SGD noise has high variance. The high variance makes the training difﬁcult to converge, and helps escape from sharp minima. Note that for training without SGD noise, it can converge to sharp minima because θ always follow the direction of gradient descent, whereas the direction of noise is random. (iii)
For our SLN in Eq.
(3), Proposition 3 shows that the standard deviation of the SGD noise is σ y (cid:107)∇ θ i S/S (cid:107) 2 with ∇ θ i S in the numerator. SLN similarly induces SGD noise with high
variance
around the sharp landscape Hence we have Claim 1. (iv
Moreover Proposition shows that SLN induces SGD noise dependent
on the conﬁdence of
S .
The standard deviation of noise on an entry of ∇ f (cid:96) ( f , y ) is minimized if S i = 1 /c , maximized if S i = 1 .
Proposition 3 directly characterizes the equivalent noise on θ , where there is a S in the denominator (element-wise division).
If S is conﬁdent, s.t., the entropy H ( S ) → 0 , which means an entry of S approaches 1 and others approach 0 , then the variance will be high since there are small numbers in the denominator. Hence we have Claim 2.
Claim 1.
With SGD noise induced by Eq. (2) or Eq. (3), the training is difﬁcult to converge when the output ( f or S ) landscape is sharp.
Claim 2.
With SGD noise induced by Eq.[-(2), we have E [ | ∆ f i | 2 ] = δ 
-]
(3), the training is difﬁcult to converge when the output ( S ) is overconﬁdent,
s.t. H ( S ) → 0 , where H ( · ) is the entropy.
In terms of escaping from sharp minima, there are previous works (Zhu et al., 2019; Daneshmand et al., 2018) that study the inheret noise in SGD, showing that the noise covariance containing curvature information performs better.
They use the second-order approximation near the minima and apply integral to training steps to characterize the ability of escaping from sharp minima.
In this paper, we study the noise induced by SLN and draw a more direct intuition between the noise covariance and the ability of escaping from sharp minima.
Moreover, the dependency between the noise induced by SLN and the conﬁdence of output probability further provides intuition on avoiding overﬁtting under noisy labels.
In terms of preventing overconﬁdence, we shall discuss label smoothing (LS) (Lukasik et al., 2020).
It smooths the given one-hot label y into a soft one ˜ y , s.t., ˜ y
=
(1 − αy )
+ αe , where e is an all-one vector and α > 0 .
In this way, LS introduces a ﬁxed and biased perturbation on the label, whereas SLN introduces dynamic and mean-zero perturbations.
LS does not introduce noise in each training step, while SLN adaptively perturbs the parameter once the landscape is sharp or the prediction is overconﬁdent. Hence, the robustness of SLN may not result from preventing overconﬁdence alone, but also escaping from sharp minima.
In Fig.
2, we plot the sample density w.r.t.
predictions on the labeled class, using CIFAR-10 with 40% symmetric noise as an example.
It shows that SLN does reduce overconﬁdence, and it mostly affects noisy samples.
3. A DETAILED DISSECTION ON PREDICTIONS OF DNN S TRAINED WITH SLN
In this section, we show that SLN boosts popular robust training methods, including sampleselection and label-correction.
Many methods, demonstrated to work well, select or add higher
[Figure] [Figure]
weight on small-loss samples (Han et al., 2018b; Jiang et al., 2018; Li et al., 2020), or use the model’s predictions to correct noisy labels (Tanaka et al., 2018; Arazo et al., 2019).
A warm-up phase is usually required to initialize the model before sample-selection or label-correction, yet the model will memorize noise if the warm-up phase is too long.
With SLN, we can simply train the model until convergence.
In Fig.
3, we compare converged models on CIFAR-10, where the model is trained with SLN or merely CE loss.
The detailed noise setting can be found in Section 4.
We ﬁrst sort training samples in ascending order of loss, then uniformly divide them into 1000 samples per interval, and ﬁnally obtain the number of four types of samples in each interval based on the correctness of the given label and the prediction.
When trained with CE, there are many small-loss samples with incorrect labels (the blue region), then selecting small-loss samples is not reliable. The model trained with SLN largely address the issue.
Moreover, SLN is suitable for label-correction since it yields correct predictions for many originally incorrect samples (the orange region).
As a concrete example, we present an enhanced algorithm by applying SLN to label-correction.
With SLN, we train the model for sufﬁcient epochs until convergence, without the need of carefully tuning a warm-up phase.
Then we start label-correction using y correction =
ω · y + (1 − ω ) · S , where S is the softmax prediction, ω ∈ [0 , 1] is the weight obtained by normalizing the training loss, s.t., for the i -th training sample, ω
i =
( (cid:96) i − (cid:96) min ) /
( (cid:96) max − (cid:96) min ) .
More discussions on the label-correction strategies are presented in Appendix C.1.
E XPERIMENTS 4. T HE OVERVIEW OF EXPERIMENTAL SETUP
We comprehensively verify the utility of SLN on different types of label noise, including symmetric noise, asymmetric noise, instance-dependent noise and open-set noise synthesized on CIFAR-10 and
CIFAR-100, and real-world noise on Clothing1M (Xiao et al., 2015).
Symmetric noise assumes each label has the same probability of ﬂipping to any other class.
We uniformly ﬂip the label to other classes with an overall probability 40% . 2)
Asymmetric noise contains noisy labels ﬂipped between similar classes.
Following Zhang & Sabuncu (2018), on CIFAR-10, we ﬂip labels between
TRUCK → AUTOMOBILE, BIRD → AIRPLANE, DEER → HORSE, and CAT ↔ DOG, with a probability 40% ; on CIFAR-100, we ﬂip each class into the next class circularly with a probability 40% .
3) Instance-dependent noise assumes that the mislabeling probability should be dependent on each instance’s input features (Xia et al., 2020).
We ﬁrst train a model on clean data, then ﬂip labels of 40% samples to their second-conﬁdent class in the predictions.
4) Open-set noise contains samples that do not belong to any class considered in the classiﬁcation task.
Following Wang et al.
(2018), we yield CIFAR-10 with open-set noise by randomly replacing 40% of its training images with images from CIFAR-100.
For real-world noise, we use the large-scale benchmark Clothing1M, which contains 1 M training images with noisy labels from online shops.
We use the backbone wide ResNet-28-2 (Zagoruyko & Komodakis, 2016) on CIFAR-10 and
CIFAR-100, and ResNet-50 on Clothing1M. More training details can be found in Appendix B.
Apart from SLN, we introduce a momentum (MO) model (Tarvainen & Valpola, 2017) and labelcorrection (LC), with which we get SLN-MO and SLN-MO-LC.
Parameters of the momentum model ( θ (cid:48) ) is updated as a moving average of the the training model ( θ ).
At t -th training step, θ (cid:48) ( t ) =
αθ (cid:48)
( t − 1) + (1 − α )
θ ( t ) .
We use α = 0 .
999 in all experiments.
For SLN-MO-LC, we have discussed in Section 3.3 that SLN is reliable when applied to label-correction.
[Figure]
[Figure]
4. T HE EFFECTS OF SGD NOISE VARIANTS
We ﬁrst show that compared with other variants, SLN stands out in mitigating the effects of label noise.
The SGD noise variants have been formalized in Eq. (1-3), including z θ directly added to ∇ θ (cid:96) , z f on ∇ f (cid:96) and our SLN ( z y ).
For SLN, the standard deviation is σ = 1 under symmetric label noise and σ = 0 .
5 in all other cases.
For other SGD noise, the standard deviation is equally tuned, and the performance under three different σ is separately shown in Fig.
SLN signiﬁcant improves the generalization, while other SGD noise variants can not achieve such impressive performance even if σ is heavily tuned.
It is worth noting that for all these variants, when σ is too small, the model overﬁts noise since the training is similar to merely using CE loss; when σ is too large, the model fails to ﬁt the training data since the SGD noise is too high.
This ﬁnding provides a guideline for tuning σ .
In Fig.
5, we show the t-SNE visualization of features on the model’s penultimate layer, taking all training samples as input.
We visualize the embedding under symmetric noise since it yields the most severe damage to the generalization and SLN provides a signiﬁcant improvement.
[Table] [Table]
Fig.
5 shows that the model trained with SLN can yield a better embedding.
It also demonstrates overﬁtting and underﬁtting when σ is too small or too large.
4. CIFAR-10 AND CIFAR-
On CIFAR-10 and CIFAR-100, we compare with the following baselines: 1) only crossentropy (CE) loss, 2) Generalized Cross-Entropy (GCE) (Zhang & Sabuncu, 2018) loss, 3)
CoTeaching (Han et al., 2018b) that uses co-training and sample-selection, 4)
PHuber-CE
(Menon et al., 2020) that uses gradient clipping and 5) label-smoothing (LS) Lukasik et al.
(2020) that clips the label to be less conﬁdent before training.
We use 5 k noisy samples as the validation to tune hyperparameters, then train the model on the full training set and report the test accuracy at the last epoch.
SLN simply requires tuning the standard deviation σ , which is tuned in { 0 . 1 , 0 . 2 , 0 .
On CIFAR-10, the best σ is 1 under symmetric noise and 0.5 otherwise; On CIFAR-100, it is 0.
under instance-dependent noise and 0.2 otherwise.
The label-correction in SLN-MO-LC is applied in the last 50 epochs.
The softmax prediction is converted into one-hot label in correction.
We repeat each experiment 5 times.
The average test accuracy at the last epoch is reported in Tableand Table 2,.
We mark the top-3 results in bold.
To compare the efﬁciency, we present the average training time of each method, evaluated on a single V100 GPU.
Without the momentum model and label-correction, SLN itself achieves impressive test performance, which is consistent with results in Fig.
In Section 3.3, we have analyzed that SLN can boosts popular robust training methods, based on a detailed dissection on DNNs’ predictions.
As expected, we obtain further improvement with SLN-MO and SLN-MO-LC.
Notably, SLN, SLN-MO and SLN-MO-LC sweep the top-3 results in almost all cases and require low computational cost.
In Fig.
6, we visualize the training efﬁciency and test accuracy.
The ﬁgure clearly illustrates the superior generalization performance and high efﬁciency of our methods.
To further verify our discussions that SLN can enhance existing methods, we present additional results in Appendix C.2.
Speciﬁcally, we apply SLN to a practical baseline that combines many robust training strategies, and show that SLN not only eliminates the need of tuning warm-up phase but also improves the test performance.
[Figure]
[Table]
[Table]
4. C LOTHING 1M
Clothing1M (Xiao et al., 2015) is a large-scale benchmark of clothing images from online shops with 14 classes, containing real-world label noise.
It has 1 million noisy samples for training, 14 k and 10 k clean samples for validation and test.
The number of images labeled as each class is unbalanced, ranging from 18976 to 88588 in the noisy training set.
In previous works, some experiments are conducted by sampling a class-balanced training subset in each epoch (Li et al., 2020)
[Table]
while others directly train on the full training set (Patrini et al., 2017).
Since the balanced training sampling itself affects the test performance, it is difﬁcult to compare results across papers.
Therefore, we make it clear here and conduct experiments in both setting, including the standard sampling and the noisy-class-balanced sampling.
For the latter, in each epoch, 18976 instances per class are randomly sampled from the noisy training set.
Other training details strictly follow the standard benchmark setting (Patrini et al., 2017), presented in Appendix B.
We set the standard deviation of SLN as σ = 0 .
For SLN-MO-LC, the label-correction is applied since the ﬁrst epoch.
Results are listed in Table [Table] with the best result in bold and previous published results marked by a star.
DivideMix trains two models in each run and we average their test accuracy, rather than using additional ensemble of two models.
Our SLN outperforms many baselines.
The variants SLN-MO and SLN-MO-LC further achieve higher test accuracy.
Speciﬁcally, SLN-MO-LC achieves the best test accuracy in both settings.
Our methods also stand out for training efﬁciency.
C ONCLUSION
In this paper, we establish that SLN induces SGD noise dependent on the sharpness of output landscape and the conﬁdence of output probability, and analyze the effects of escaping from sharp local minima and preventing overconﬁdence.
This partially explains the robustness of SLN under noisy labels since various works show that ﬂat minima typically generalize well and preventing overconﬁdence helps mitigate overﬁtting on noisy labels.
We empirically verify the robustness of SLN under various synthetic label noise and real-world noise.
Moreover, we show that SLN boosts popular robust training methods, including sample-selection and label-correction.
In particular, we justify that SLN can enhance existing methods based on a detailed dissection on DNNs’ predictions, then present a practical algorithm by applying SLN to label-correction.
A P ROOFS
Proposition 1.
Compared with Eq.
(1), Eq.
(2) induces noise z ∼ N (0 , σ 2 f M ) on ∇ θ (cid:96) ( f, y ) , s.t.,
M ∈ R p × p and M i,j = ( ∇ θ i f ) T ∇ θ j f , ∀ i, j ∈ { 1 , · · · , p } .
Note that the standard deviation of noise on the i -th parameter θ
i is σ f (cid:107)∇ θ i f (cid:107) 2 , where (cid:107)·(cid:107) 2 denotes the L 2 norm.
[Equation]
For Eq.
(2), the noisy gradient is
[Equation]
y
[Equation]
+ σ
[Equation]
The noise on ∇ θ (cid:96) ( f, y ) is z = σ f z f · ∇ θ f ∈ R 1 × p .
Note that z f ∼ N (0 , I c × c ) , let z i be the i -th entry of z , we have z i
[Equation]
Hence,
[Equation] [Equation]
Proposition 2.
For the cross-entropy loss, compared with Eq.
(2), Eq.
(3) induces noise z ∼
N (0 , σ 2 y M ) on ∇ f (cid:96) ( f, y ) , s.t., M ∈ R c × c , M i,i =
c ( S i − 1 /c )
+ ( c − 1) /c , and M i,j = c · S i S j − S
i − S j , if i (cid:54) =
j .
Note that the standard deviation of noise on the i -th entry is σ y (cid:112) c
( S i − 1 /c )
+ ( c − 1) /c .
Proof.
[Equation]
Firstly, for the softmax function S = S ( f ( x )) , we have the derivative matrix
[Equation]
where Diag( S ) is a diagonal matrix with S i on its i -th diagonal element and 0 otherwise.
For the cross-entropy loss, (cid:96) ( f, y ) =
− (cid:80) ck
=1 y k log S k .
Let · S denote element-wise division, we have
[Equation]
Then it is equivalent to induce noise z = − σ y ( z y − (cid:80) ck
=1 z y k · S ) on ∇ f (cid:96) ( f, y ) , whose i -th entry is z i = − σ y ( z y i − (cid:80) ck
=1 z y k · S i ) .
Note that z y ∼ N (0 , I c × c ) , then we can derive for i = j , E [ z [Equation] i ]
= σ
[Equation]
y (
[Equation]
S i + c · S 2 i ) = [Equation] y ( [Equation]
[Equation]
[Equation]
) , (9) and for i (cid:54) =
j ,
E [ z
[Equation]
z [Equation] ] = σ [Equation]
( c
[Equation]
S
[Equation]
− S
[Equation]
− S
[Equation]
Proposition 3.
For the cross-entropy loss, compared with Eq.
(1), Eq.
(3) induces noise z ∼
N (0 , σ 2 y M ) on ∇ θ (cid:96) ( f, y ) , s.t., M ∈ R p × p and M i,j = ( ∇
θi S S ) T ∇ θj S S , ∀
i, j ∈ { 1 , · · · , p } , where · S denotes the element-wise division.
Note that the standard deviation of noise on the i -th parameter θ
i is σ y (cid:13)(cid:13)(cid:13) ∇ θi S S (cid:13)(cid:13)(cid:13) 2 .
Proof.
Dimension: z y ∈ R c ,
[Equation]
∈
[Equation]
p
[Equation]
∈ R 1 × c , ∇ θ S ∈ R c × p , ∇ θ i S ∈ R c .
For Eq.
(3), the noisy gradient is
[Equation]
The noise on ∇ θ (cid:96) ( f, y ) is z = − (cid:0) σ y z y S (cid:1) T · ∇ θ S .
Note that z y ∼ N (0 , I c × c ) , let z i be the i -th entry of z , we have
[-δ 
-]
[Equation]
Hence,
[Equation]
B T RAINING DETAILS
In all experiments on CIFAR-10 and CIFAR-100, we train wide ResNet-28-2 (Zagoruyko & Komodakis, 2016) for 300 epochs using the SGD optimizer with learning rate 0 .
001 , momentum 0 .
9 , weight decay 5 × 10 − 4 , and a batchsize of 128 .
This is sufﬁcient for all methods to converge.
Standard data augmentation is applied, including per-pixel normalization, horizontal random ﬂip and 32 × 32 random crop after padding with 4 pixels on each side.
On Clothing1M, following the common setting (Patrini et al., 2017), we train an Imagenet-pretrained ResNet-50 using the SGD optimizer with momentum 0 .
9 , weight decay 10 − 3 and a batchsize of 32 .
The initial learning rate is 10 − 3 and decreased to 10 − 4 after 5 epochs.
We use standard data augmentation with per-pixel normalization, horizontal random ﬂip and 224 × 224 random crop.
C A DDITIONAL R ESULTS AND D ISCUSSIONS C. T HE WEIGHT IN LABEL - CORRECTION
When applying our SLN to label-correction, we use y correction =
ω · y +(1 − ω ) · S , which is different to the correction strategy in Arazo et al.
Firstly, the common idea is that both Arazo et al.
(2019) and our method tend to yield higher ω for large-loss samples.
Arazo et al.
(2019) have shown that assigning weight according to the loss is superior than using a ﬁxed weight (Reed et al., 2014).
Secondly, the key difference is that Arazo et al.
(2019) use y correction =
(1 − ω ) · y + ω · S .
We ﬁnd that our correction strategy helps mitigate the convergence issue.
The problem of unstable convergence is reported in Arazo et al.
It may be a common issue for label-correction methods, but has not been widely discussed, nor do we have an explanation on the issue.
Here, we provide a hypothesis, with which we justify our label-correction strategy.
When we use the model’s predictions to correct noisy labels, since the predictions unavoidably contain errors, the model easily accumulates its own errors in each label-correction iteration if the weight on predictions is too high.
This should be more clear by analysing how much the label is modiﬁed:
[Equation]
For small-loss samples, S is close to y , hence labels of these samples will not be modiﬁed much no matter how we apply ω .
The key is how we correct large-loss samples, for which S is very different from y , and ω is high.
These large-loss samples cover three cases in Fig.
7, 1) label × , prediction × , 2) label × , prediction (cid:88) , 3) label (cid:88) , prediction × .
Samples of case 2) can be improved by label-correction.
Compared with CE, SLN yields more samples of case 2) (the orange region).
Still, we can not guarantee that the prediction S is 100% correct, i.e., there are case 1) and case 3) .
If we correct the label with large ω on S for these samples, it may result in accumulating prediction [Figure] [Figure] errors, because we will train on the modiﬁed label.
Worse still, the weight ω will be higher in the next label-correction iteration, since the loss between S and y shall be higher after training on the modiﬁed label.
Such positive feedback may account for the unstable convergence observed in existing label-correction methods.
To address this issue, we apply a mild correction with weight 1 − ω on S .
Therefore, for small-loss samples, the correction does modify the label much because S is close to y ; for large-loss samples, 1 − ω is small, avoiding high weight on S .
As shown in Fig.
8, our strategy works well in terms of convergence.
[Table]
C. SLN ENHANCES EXISTING METHODS
[Table]
We have analyzed that SLN can boosts popular robust training methods such as label-correction and sample-selection.
Here we use the model trained with SLN as the initialization of a practical algo-
It takes more than 4 times of cost compared , and has extensive
Still, it works well after we heavily tuning the hyperparameters.
With SLN as initialization, we do not need to tune the warm-up phase, because we can train with SLN until convergence without the concern of overﬁtting.
In contrast, without SLN, we need to carefully warm up the model so that it learns enough correct patterns and does not memorizes too much noise.
Moreover, as shown in
Table 4, the of DivideMix is improved with SLN.