Noise Attention Learning: Enhancing Noise Robustness by Gradient Scaling
Abstract
Machine learning has been highly successful in data-driven applications but is often hampered when the data contains noise, especially label noise.
When trained on noisy labels, deep neural networks tend to ﬁt all noisy labels, resulting in poor generalization.
To handle this problem, a common idea is to force the model to ﬁt only clean samples rather than mislabeled ones.
In this paper, we propose a simple yet effective method that automatically distinguishes the mislabeled samples and prevents the model from memorizing them, named Noise Attention Learning.
In our method, we introduce an attention branch to produce attention weights based on representations of samples.
This attention branch is learned to divide the samples according to the predictive power in their representations.
We design the corresponding loss function that incorporates the attention weights for training the model without affecting the original learning direction.
Empirical results show that most of the mislabeled samples yield signiﬁcantly lower weights than the clean ones.
Furthermore, our theoretical analysis shows that the gradients of training samples are dynamically scaled by the attention weights, implicitly preventing memorization of the mislabeled samples.
Experimental results on two benchmarks (CIFAR-10 and CIFAR-100) with simulated label noise and three realworld noisy datasets (ANIMAL-10N, Clothing1M and Webvision) demonstrate that our approach outperforms state-of-the-art methods.
Introduction
With the emergence of highly-curated datasets such as ImageNet [1] and CIFAR [2], deep neural networks (DNNs) have achieved remarkable performance on many classiﬁcation tasks.
However, in real-world applications, it is extremely time-consuming and expensive, sometimes even impossible to label a new large-scale dataset containing fully correct annotations.
To alleviate this problem, one may obtain the data with lower quality annotations efﬁciently through online keywords queries [3] or crowdsourcing [4], but noisy labels (e.g. a cat image is mislabeled as dog) are inevitably introduced consequently.
Previous study [5] has empirically demonstrated that noisy labels in training data are problematic for deep neural networks, resulting in overﬁtting and performance degradation.
Therefore, designing robust algorithms against noisy labels is of great practical importance.
Given the training data consists of clean (correctly labeled) samples and mislabeled samples, DNNs have been observed to correctly predict the true labels for all training samples during a early learning [Figure] stage, and then start to make incorrect predictions as it gradually memorizes the mislabeled samples [6, 7] (see Figure 1(a) and 1(b)).
Based on this empirical ﬁnding, many existing approaches [8–11] use the training loss or prediction conﬁdence to identify mislabeled samples in the early learning stage, i.e., the samples with small loss or high conﬁdence are more likely to be clean samples.
After dividing the samples into clean and mislabeled sets, one can simply train the model only based on clean set [8, 11], or correct these mislabeled samples by pseudo-labels [9, 12], or treat them as unlabeled samples and train the model using semi-supervised learning algorithms [10].
However, these methods typically involve sophisticated training procedure modiﬁcations and sometimes require high computational costs.
In this paper, we propose Noise Attention Learning (NAL) which enjoys simplicity and universality from a new perspective.
The core idea behind NAL is to let the model automatically differentiate the mislabeled samples from clean samples, and prevent the memorization of mislabeled samples in training.
Since the memorization of DNNs has a preference for easy (clean) samples, the predictive power of a sample’s representation aligns with its label cleanliness [6].
To transform this qualitative observation into a quantitive measure, we introduce an attention branch that outputs scalar attention weights to indicate the predictive power of representations.
We incorporate the attention weights in our loss function, so that the attention branch can be learned to divide samples, i.e., clean samples are learned to be associated with large weights, while mislabeled ones with small weights.
In this way, the proposed method does not require extra steps (e.g. modeling loss distribution [9, 10], selecting small-loss samples [8]) to separate the training set, effectively simplifying the training procedure.
Theoretically, we demonstrate that NAL neutralizes the effect of the mislabeled samples on the gradient and ensures the contribution of clean samples to the whole gradient remains dominant, thereby preventing the deep neural networks from overﬁtting mislabeled samples.
The proposed method has a similar effect on gradients as an existing regularization method ELR [7].
In general, the performance of pure regularization methods cannot be compared to complex methods.
Despite that the regularization methods can effectively hinder the memorization of mislabeled samples, the limited number of clean samples makes the model not rich enough to generalize effectively to the held-out data.
To solve this problem, ELR uses temporal ensembling [13] to estimate the targets in its loss function to correctly guide the model for improving performance.
Similarly, instead of directly using noisy labels as target distribution in our loss function, we adopt SELC [12] to estimate the targets, allowing the generation of more clean labels for learning sufﬁciency.
Empirically, we show that the proposed approach achieves better robustness than the state-of-the-art methods on two benchmarks (CIFAR-10 and CIFAR-100 [2]) with simulated label noise and three challenging datasets in noisy label literature (ANIMAL-10N [15], Clothing1M [16] and Webvision [3]) with real-world label noise.
To better understand our method, we conduct many empirical analyses, including memorization analysis, gradient analysis, attention weights, quality of targets, hyperparameter sensitivity and ablation study, to verify our theoretical explanations and design goals.
Preliminaries
Classiﬁcation with Noisy Labels.
Consider the K -class classiﬁcation problem in noisy-label scenario, the ground truth label y is unobservable.
We only have a noisy training set
ˆ D = { ( x [ i ] , ˆ y
[ i ] ) }
Ni =1 , where x [ i ] is an input and ˆ y
[ i ] ∈ Y = { 1 , . . .
, K } is the corresponding noisy label.
We denote ˆ y
[ i ] ∈ { 0 , 1 } K as one-hot vector of noisy label
ˆ y
[ i ] .
A DNN N θ maps an input x
[ i ] to a Kdimensional logits and then feeds the logits to a softmax function to obtain p
[ i ] of the conditional probability of each class.
θ denotes the parameters of the DNN and z [ i ] ∈ R K × 1 denotes the logits (i.e. pre-softmax output).
We refer to it as prediction branch in this work.
We have z
[ i ] =
N θ ( x [ i ] ) and p [ i ] =
softmax ( z [ i ] ) .
Without knowing the ground truth joint probability distribution P ( x , y ) , the typical cross-entropy (CE) loss is often used as empirical risk to measure how well the model ﬁts the training set ˆ D as follows:
[Equation]
Early Learning Phenomenon When optimizing L ce by stochastic gradient descent (SGD), it has been observed that DNNs ﬁt the easy (clean) samples ﬁrst before memorizing of hard (mislabeled) samples [6].
Figure 1(a) and 1(b) shows this qualitative behavior.
The model begins by learning to predict the true labels, even for majority of mislabeled samples, but eventually learns to predicts wrong labels due to memorization of mislabeled samples.
Label Noise Model.
The generation of real-world label noise is unpredictable, a typical methodology to dealing with noisy labels is to posit a noise model and develop robust algorithms under the hypothetical noise model.
The algorithms are then tested on the real-world noisy datasets to see how effective they are.
A widely-accepted noise model is class-conditional noise [17, 18], where label noise is independent of input features and true labels are corrupted by a symmetric or asymmetric noise transition matrix (simulation details are in Appendix B.2).
Instance-dependent noise [19, 20] is another noise model in which the noise is affected not only by the label but also by the input feature.
Noise Attention Learning
In this section, we present our approach NAL for learning with noisy labels.
NAL consists of three key elements: (1) An attention branch based on learned representations to produce the attention weight for each training sample; (2) A noise attention loss is speciﬁcally designed to learn the attention weights; (3) A target estimation strategy to help generate more clean labels for sufﬁcient learning.
3.1 Attention Branch based on Learned Representations
To enable the deep neural networks to produce an attention weight that reﬂects the quality of learned representation, we introduce an attention branch just after the penultimate layer of the original model.
We denote M -dimensional penultimate layer (representation) of input x
[ i ] as H
[ i ] .
H [ i ] is shared in both prediction and attention branches.
For each input, the prediction branch outputs the softmax prediction p [ i ] as usual.
The attention branch contains one fully connected layer to produce a single scalar value h
[ i ] , and sigmoid function is applied to scale it between 0 to 1.
Speciﬁcally, we have h [ i ] = WH
[ i ] + b , where W ∈ R 1 × M denotes the weights and b ∈ R denotes the bias in penultimate layer of the attention branch.
We have the attention weight τ
[ i ] for each sample x
[ i ] as
[Equation]
Note that W and b are extra network parameters need to be learned to produce meaningful τ ’s.
Therefore, attention weights are not ﬁxed but dynamic during the training.
3.2 Noise Attention Loss
The early learning phenomenon reveals that the DNNs memorize the clean samples before the mislabeled samples.
Therefore, clean samples are more likely to have better learned representations than mislabeled samples in the early learning phase.
To enable the attention weights to automatically capture the difference of representations as well as learn the classiﬁcation task, we propose a noise attention loss consisting of two terms: attention term L a and boost term L b .
[Equation]
where λ is the hyperparameter.
Intuitively, the attention term L a can be interpreted as a dynamic expansion of the CE loss, where the prediction is modiﬁed to ‘perceptual shortcut’ τ ( p − ˆ y )
+ ˆ y
(Note that many existing methods [9, 21, 22] use this linear interpolation formula to infer the label distribution, while the proposed NAL applies it to prediction distribution for learning the meaningful attention weights).
Assuming no representations are corrupted and all weights τ → 1 , the attention term L a reduces to exactly L ce in Eq.
On the contrary, when all τ → 0 , the attention term L a becomes “lazy" as it produces many zero gradients.
The mechanism of attention term can be explained as follows: (1) For mislabeled samples, the model does not memorize them in the early learning phase.
Therefore, their representations are under learned compared to clean samples and ( p − ˆ y ) remains large.
By minimizing L a , it forces τ of mislabeled samples toward 0, which is treated as ‘shortcut’.
(2) For clean samples, the model memorizes them ﬁrst, resulting in ( p − ˆ y )
→ 0 .
It makes τ have no inﬂuence on minimizing L a as the prediction distribution is already equal to ˆ y .
As a result, by solely minimizing L a , we may obtain a trivial optimization result that the model always produces τ → 0 for any inputs.
To avoid this ‘lazy’ learning circumstance, we have another boost term L b .
It can be interpreted as a binary CE loss, where the target of τ is always 1 for all inputs.
By adding the boost term, the attention weights of clean samples are pushed to 1.0, and weights of mislabeled samples are still close to 0 as expected.
To verify the effectiveness of L NAL in discriminating the mislabeled samples from clean samples, we empirically analyze the weight distribution of clean and mislabeled samples with simulated symmetric noise.
Figure 2 shows results on CIFAR-10 and CIFAR-100 with different ratios of symmetric noise.
In 0% noise cases, we observe that all samples have weights close to 1.0, which indicates that L NAL reduced to exactly L ce .
Thus, the proposed loss would not affect the performance when training with clean data.
At other ratios of label noise, there is a clear separation between attention weights of clean and mislabeled samples.
The weights of clean samples form a spike close to 1.0, whereas the weights of most mislabeled samples are plainly less than 1.0, which satisﬁes our design goal.
3.3 Target Estimation
For regularization methods, directly using the noisy labels as target distribution in loss function is less effective as the limited number of clean samples makes the model not rich enough to generalize effectively to the held-out data, especially at the high noise level. To yield better performance, ELR [7] 1: Input: DNN N θ ; Training set ˆ D , ˆ
2: for do 3: for each minibatch B do 4: for i B do 5: Obtain z [ i ] , h
[ i ] =
N θ ( x [ i ] ) .
Obtain p [ i ] , τ [ i ] =
softmax
( z [ i ] ) , sigmoid ( h [ i ] ) .
Update t [ i ] [ e ] by Eq.
8: end for 9:
Calculate L NAL = − 1 |
[Equation]
(cid:80) | B | i =1 ( t [ i ] [ e ] ) (cid:62) log (cid:0) τ
[ i ] ( p [ i ] − t [ i ] [ e ] ) + t
[ i ] [ e ] (cid:1) − λ | B
[Equation]
(cid:80) | B | i =1 log( τ [ i ] ) .
Update θ using stochastic gradient descent.
11: end for 12: end for 13: Output: θ . use the temporal ensembling [13] to estimate targets in its regularizer and steer the model towards these new targets.
The ensemble predictions are formed by an exponential moving average of historical outputs.
Speciﬁcally, in the m -th epoch, it can be expressed as ˜ p [ m ] = (cid:80) mj
=1 (1 − α ) α m − j p
[ j ] , where 0 ≤ α < 1 is the momentum and p [ j ] represents the output of j -th epoch.
In this paper, we use the ensemble strategy in SELC [12] to estimate our targets t for the m -th epoch as follows:
[Equation]
initial term
[Equation]
The initial term preserves the original noisy labels with exponential decaying weights α m .
In the early learning phase (small m ), the overall target relies on noisy labels to learn the model.
As the learning continues (larger m ), α m approaches 0.
The overall target eventually depends on the ensemble term ˜ p [ m ] and penalizes model predictions that are inconsistent with this target.
In our experiment, we will demonstrate that estimated targets are more accurate than original noisy labels.
We simply replace noisy label ˆ y with ensemble target t in our loss.
Algorithm 1 shows pseudocode of our method.
Theoretical Justiﬁcation
In this section, we demonstrate the noise robustness of L NAL by analyzing how it scales the gradients accordingly to prevent memorization of mislabeled samples.
The omitted proofs in this section are in Appendix A. For clarity of explanation, we denote the true label of sample x as y ∈ { 1 , ..., K } .
The ground-truth distribution over labels for sample x is q ( y | x ) , and (cid:80) K k
=1 q ( k | x ) = 1 .
Consider the case of a single ground-truth label y , then q ( y | x ) = 1 and q ( k | x ) = 0 for all k (cid:54)
=
y .
We denote the prediction probability as p ( k | x ) and (cid:80)
Kk =1 p ( k | x ) = 1 .
For notation simplicity, we denote p k , q k , p y , q y , p j , q j as abbreviations for p ( k | x ) , q ( k | x ) , p ( y | x ) , q ( y | x ) , p ( j | x ) and q ( j | x ) , where j represents j -th entry.
First, we show how the CE loss fails in noisy-label scenario.
Lemma 4.1.
Given the cross-entropy loss L ce in Eq.
(1), we rewrite the sample-wise loss (cid:96) ce
= − (cid:80)
Kk =
q k log p k .
Its gradient with respect to z j is ∂(cid:96)
[Equation]
where z j is the j -th entry of logits z .
q j = q y = 1 means j equals the true class y .
In clean-label scenario, Lemma 4.1 ensures that, during SGD, learning direction of CE loss continues towards the true class as the corresponding gradient remains negative.
However, in noisy-label scenario, the learning direction ﬂuctuates.
Suppose j is true class and equals y , but q j = 0 due to the label noise, then the contribution of a mislabeled sample to the gradient is reversed (i.e. gradient [Figure] should be negative but get positive instead).
The entry corresponding to the impostor class j (cid:48) , is also reversed because q j (cid:48) = 1 , causing the gradient of mislabeled samples dominates (in Figure 3(a)).
Consequently, training with CE loss results in memorization of the mislabeled samples.
Theorem 4.1.
Given the noise attention loss L NAL in Eq.
(3), we rewrite the sample-wise loss (cid:96) NAL = − (cid:80)
Kk =1 q k log( τ ( p k − q k ) + q k ) −
λ log τ .
Its gradient with respect to the logits z j can be derived as [Equation]
not the true
[Equation]
where the multiplier
[Equation]
Theorem 4.1 ensures that learning on true class persists when training with noise attention loss.
In addition, compared to the gradient of (cid:96) ce , the gradient of (cid:96) NAL is scaled by an positive multiplier term p y p y − 1+1 /τ .
Let’s denote ϕ = p y p
y − 1+1 /τ .
ϕ is monotonically increasing on τ .
We have lim τ → 1 ϕ = 1 , and lim τ → 0 ϕ = 0 .
For the samples with the true class j in Eq.
(6a), the cross-entropy gradient term p j − 1 of clean samples tends to vanish after early learning stage, causing mislabeled samples to dominate the gradient.
However, by multiplying ϕ (note that ϕ
→ 0 for mislabeled samples and ϕ → 1 for clean samples due to attention weight distribution), it counteracts the effect of gradient dominating by mislabeled samples.
For the samples that j is not the true class in Eq.
(6b), the gradient term p j is positive.
Multiplying ϕ < 1 effectively dampens the magnitudes of coefﬁcients on these mislabeled samples, thereby diminishing their effect on the gradient.
Figure 3(b) empirically shows the gradient when training with L NAL .
It can be observed that NAL keeps the gradient of clean samples dominant and diminishes the gradient of mislabeled samples when epoch increases, effectively preventing memorization of mislabeled samples.
Experiments
In this section, we ﬁrst compare NAL with the existing methods on ﬁve widely used datasets.
Then we provide several empirical results towards a better understanding of NAL.
We also provide an ablation study to evaluate the inﬂuence of each component and show the results when integrated with another noise robust method SCE [23]. Finally, we discuss the scenarios when NAL encounters hard samples.
All experiments are implemented in PyTorch and run in a single NIVDIA A100 GPU.
5.1 Comparison with Existing Methods
Datasets and Setups.
We evaluate our approach on two benchmarks CIFAR-10 and CIFAR- [2] with simulated label noise, and three real-world datasets, ANIMAL-10N [15], Clothing1M [16] and WebVision [3].
Since CIFAR-10 and CIFAR-100 are initially clean, we follow [18] to inject [Table] [Table] [Table]
[Table] [Table] class-conditional label noise: symmetric and asymmetric label noises. Symmetric noise is generated by uniformly ﬂipping the label to one of the other class label.
Asymmetric noise is a simulation of ﬁne-grained classiﬁcation, where the label ﬂipping only occurs within very similar classes (e.g. dog ↔ cat).
Following PLC [20], we also evaluate NAL on Polynomial Margin Diminishing (PMD) noise, an instance-dependent noise that allows arbitrary noise strength in a wide buffer near the decision boundary.
ANIMAL-10N contains human-labeled online images for 10 animals with confusing appearance.
Its estimated noise rate is 8%.
Clothing1M consists of 1 million images collected from online shopping websites with labels generated from surrounding texts.
Its estimated noise rate is 38.5%.
WebVision contains 2.4 million images crawled from the web using the 1000 concepts in ImageNet ILSVRC12.
Its estimated noise rate is 20%.
For CIFAR with class-conditional noise, we use a ResNet-34 [39] and train it using SGD with a batch size of 64. For CIFAR with instancedependent noise, we use a PreActResNet34 [14] and train it using SGD with a batch size of 128. For
ANIMAL-10N, we use VGG-19 [40] with batch normalization and train it using SGD with a batch size of 128.
For Clothing1M, we train a ResNet-50 [39] using SGD with a batch size of 64.
For
Webvision, we train a InceptionResNetV2 [41] using SGD with a batch size of 32.
More training details can be found in Appendix B.
Table 1 shows the results on CIFAR-10 and CIFAR-100 with different levels of symmetric and asymmetric label noise.
We compare NAL to the best performing approaches that only modify the training loss.
All of these methods use the same backbone (ResNet34).
NAL shows substantial improvements over other methods and obtains the highest accuracy in most cases.
Table 2 shows the results on three types of instance-dependent (PMD) noise from PLC [20].
We observe that the proposed method outperforms baselines across different noise settings. Table 3 shows the results on ANIMAL-10N. NAL achieves state-of-the-art performance, even better than Nested which uses Co-teaching to boost performance.
Table 4 shows the results on the Clothing1M. All compared methods are trained using ResNet pretrained on ImageNet.
However, as NAL needs to learn the parameters W and b in the attention branch, we directly train our model from scratch.
As we can observe from Table 4, NAL still slightly outperforms other methods even without using an ImageNet pretrained model.
Table 5 shows the results on the (mini) WebVision following [35].
All the compared methods are evaluated on WebVision and ImageNet ILSVRC12 validation sets.
We observe that NAL consistently outperforms state-of-the-art methods on both validation sets in terms of top1 and top5 accuracy.
Note that we do not compare with some state-of-the-art methods like RoCL [42] and DivideMix [10] as baselines, because these methods are aggregations of multiple techniques (e.g. mixup [43], multiple networks [8], and complex augmentation [44]), while this paper only focuses on one, so the comparison seems unfair.
5.2 Empirical Analysis of NAL
Memorization Procedure of NAL.
Figure 1(c) and 1(d) show the memorization procedure of NAL on clean and mislabeled samples, respectively.
Compared to CE loss in Figure 1(a) and 1(b), NAL effectively prevents memorization of mislabeled samples, allowing the model to continue learning on the clean samples to attain high classiﬁcation accuracy.
Gradient of L ce and L NAL .
Figure 3(a) and 3(b) show the gradients of CE loss and noise attention loss on CIFAR-10 with 40% symmetric noise using ResNet34 [39] respectively.
In comparison to CE loss, the proposed method keeps the gradient of clean samples dominant and dampens the gradient of mislabeled samples, resulting in the whole gradient being guided by clean samples.
Attention Weights.
When trained on noisy labels, we have observed that the mislabeled samples have smaller weights than the clean samples in Figure 2.
Here, we report the average weights of samples in Figure 4(a).
The ( i, j ) -th block represents the average weights of samples with clean label i and noisy label j .
We observe that the attention weights on the diagonal blocks are higher than those on non-diagonal blocks, which indicates NAL does properly down-weight the mislabeled samples.
Quality of Estimated Targets.
We evaluate the quality of new targets t by correction accuracy deﬁned via 1 N (cid:80) N i 1 { argmax j y
[ i ] j = argmax j t
[ i ] j } , where y [ i ] is the true label of input x [ i ] .
Figure 4(d) shows the correction accuracy vs. epochs on CIFAR-10 with different levels of label noise.
The target estimation in Section 3.3 steadily improves the quality of new targets.
Figure 4(b) and Figure 4(c) shows the confusion matrix of noisy labels and corrected labels (i.e. the hard version of new targets) w.r.t the clean labels on CIFAR-10 with 60% symmetric noise respectively.
We observe that target estimation corrects the noisy labels impressively well for all classes. [Figure] [Figure] [Figure]
Sensitivity of Hyperparameters λ and α .
NAL contains two hyperparameters: λ is the coefﬁcient for boost term L b , α is the momentum in target estimation.
Figure 4(e) and 4(f) show their sensitivity to the performance.
We observe that the sensitivity to α is quite mild as long as it is set close to 1. λ need to be tuned according to the complexity of dataset.
It cannot be set to be very large or very small, resulting in neglecting attention term or falling into the ‘lazy’ learning circumstance.
In our experiments, we ﬁxed α = 0 .
9 for all datasets.
For CIFAR-10 and ANIMAL-10N, we set λ = 0 .
For CIFAR-100, we set λ
= 10 .
For Webvision and Clothing1M, we set λ = 50 .
5.3 Ablation Study
Table 6 reports the inﬂuence of the two components in NAL: target estimation and attention branch.
Removing the target estimation leads to a signiﬁcant performance drop.
This suggests that using target estimation is crucial as it generates more clean samples for the model to learn.
To validate the effect of the attention branch, we conduct another way to calculate the attention weights: using the conﬁdence score (highest probability in output) as the weight, i.e., max j p j , j ∈ [1 , K ] .
We observe that the model does not converge in hard cases.
We conjecture that using the conﬁdence from model output does interfere with the original prediction branch, while adding the attention branch to get attention weight effectively solve this problem.
We also explore the possibility of integrating
NAL with other methods.
Here we adopt the idea from symmetric cross entropy (SCE) [23], which adds the reverse term to achieve noise robustness.
As shown in Table 6, adding RNAL affects the performance under CIFAR-10, but enjoys performance boost under CIFAR-100 without extra cost. 5.4 Discussion on Hard Samples
We assume the training set consists of mislabeled samples, easy (clean) samples and hard (clean) samples.
In this work, we do not design a speciﬁc technique to distinguish hard samples but the proposed NAL can handle hard samples as well.
In the early learning phase, both mislabeled samples and hard samples have lower weights compared to the easy samples, which forces the model to learn only from easy samples due to gradient scaling (Theorem 4.1).
After the model has learned from easy samples for a while, the model then gradually learns from hard samples rather than the mislabeled
[Table] samples. This is because the hard samples are more connected to the easy samples than the mislabeled samples, as the former two share some common features [45, 46]. Therefore, the proposed method can continuously learn from clean samples (both easy and hard) and prevent the memorization of mislabeled samples.
In Figure
[Table]
we can observe less overlap between clean and mislabeled samples.
We also observe that some clean samples are given low weights, and some mislabeled samples are given high weights.
The clean samples given low weights are actually hard clean samples.
There are also mislabeled samples given high weights.
These mislabeled samples are hard samples with "proper" wrong labels.
For example, given a hard sample: a cat image that looks like a dog due to blurry resolution (e.g. 32 × 32).
When generating simulated label noise, it is likely to assign a dog label for this sample.
Compared to regular mislabeled samples, this kind of mislabeled samples can be learned easily by the classiﬁer, resulting in having high weights.
We brieﬂy discuss the existing noise-robust methods that do not require a set of clean training data (as opposed to [16, 47, 48]).
Loss Correction.
Many approaches focus on correcting the loss function explicitly by estimating the noise transition matrix [49, 18, 50].
Robust Loss Functions.
These studies develop loss functions that are robust to label noise, including L DMI [31], MAE [51], GCE [24], SCE [23], NCE [52], TCE [53] and GJS [54].
Above two categories of methods do not utilize the early learning phenomenon.
Sample Selection
During the early learning stage, the samples with smaller loss values are more likely to be the clean samples.
Based on this observation, MentorNet [55] pre-trains a mentor network for selecting small-loss samples to guide the training of the student network.
Co-teaching related methods [8, 56–58] maintain two networks, and each network is trained on the small-loss samples selected by its peer network.
Label Correction Joint Opt [25] and
PENCIL [59] replace the noisy labels with soft (i.e. model probability) or hard (i.e to one-hot vector) pseudo-labels.
[9] weigh the clean and mislabeled samples by ﬁtting a two-component Beta mixture model to loss values, and corrects the labels via mixup combination.
Similarly, DivideMix [10] trains two networks to separate the clean and mislabeled samples via a two-component Gaussian mixture model, and further uses MixMatch [60] to enhance the performance.
Regularization [61] prove the gradient descent with early stopping is an effective regularization to achieve robustness to label noise. [62] explicitly add the regularizer based on neural tangent kernel [63] to limit the distance between the model parameters to initialization.
ELR [7] estimates the target by temporal ensembling [13] and adds a regularization term to cross-entropy loss to avoid memorization.
Other techniques, such as mixup augmentation [43], label smoothing [41] and weight averaging [64], can effectively improve the performance under label noise.
Conclusion
In this work, we propose NAL for learning with noisy labels.
Our method leverages an attention branch and a noise attention loss to learn the attention weights for distinguishing the mislabeled samples from clean samples.
NAL can effectively diminish the gradient of mislabeled samples, mitigating the effect of noisy labels.
We also provide extensive empirical analyses and evaluate its effectiveness across multiple datasets with different types and ratios of label noises.
There are still multiple open problems for future research.
Currently, label noise has been extensively studied in image classiﬁcation task.
Other research areas, such as graph learning and federated learning, remain to be explored.
On the methodological front, we hope that our work will trigger interest in designing novel network architectures that inherently provide robustness to label noise.
A Missing Proofs A.1 Gradient Derivation of (cid:96) NAL
The sample-wise (cid:96) NAL can be rewrite as: [Equation] As τ is learned from attention branch.
The derivation of the (cid:96) NAL with respect to the logits is as follows: [Equation] Since p k = softmax ( z ) =
[Equation] we have [Equation] In the case of k = j :
[Equation]
In the case of k (cid:54) = j :
[Equation]
Combining Eq. (4) and (5) into Eq. (2), we obtain:
[Equation]
Therefore
[Equation]
if q j = q y = [Equation]
then ∂(cid:96)
NAL
[Equation]
If q j = 0 , then [Equation]
Theorem A.1 (4.1) .
Given the noise attention loss L NAL , we rewrite the sample-wise loss (cid:96) NAL = − (cid:80)
Kk =1 q k log( τ ( p k − q k ) + q k ) − λ log τ . Its gradient with respect to the logits z j can be derived as
[Equation] the true [Equation] x
[Equation] where the multiplier [Equation] Proof.
From the Appendix A.1, we obtain the gradient of the sample-wise (cid:96) NAL with respect to the logits
z j is [Equation] where ∂p k ∂z j can be further derived base on whether k = j by follows: [Equation]
According to Eq. (10) and (11), the gradient of (cid:96) NAL can be derived as: [Equation] We denote ϕ = p y p y − 1+1 /τ .
Since p j ≤ 1 , we have p j − 1 ≤ 0 .
As τ ∈ (0 , 1) , the term p y p y − 1+1 /τ ∈ (0 , 1) , we have ( p j − 1) p y p
y − 1+1 /τ ≤ 0 and p j p y p y − 1+1 /τ ≥ 0 .
All experiments are implemented in PyTorch and run in a single Nvidia A100 GPU. For CIFARand CIFAR-100, we do not perform early stopping since we don’t assume the presence of clean validation data.
All test accuracy are recorded from the last epoch of training. For Clothing1M, it provides 50k, 14k, 10k reﬁned clean data for training, validation and testing respectively. Note that we do not use the 50k clean data for fair comparison with existing methods. Similar to the compared methods [7, 23], we report the test accuracy when the performance on validation set is optimal.
All tables of CIFAR-10/CIFAR-100 report the mean and standard deviation from 3 trails with different random seeds for different simulated noise.
The information of datasets are described in Table 7.
CIFAR-10 and CIFAR-100 are clean datasets, we describe the label noise injection in Appendix B.2.
ANIMAL-10N contains human-labeled online images for 10 animals with confusing appearance.
Its estimated noise rate is 8%.
Clothing1M consists of 1 million training images from 14 categories collected from online shopping websites with noisy labels generated from surrounding texts.
Its noise level is estimated as 38.5%. Following [35], we use the mini WebVision dataset which contains the top 50 classes from the Google image subset of WebVision, which results in approximate 66 thousand images.
The noise level of WebVision is estimated at 20% [3].
As for data preprocessing, we apply normalization and regular data augmentation (i.e. random crop and horizontal ﬂip) on the training sets of all datasets.
The cropping size is consistent with existing works [7]. Speciﬁcally, 32 for CIFAR-10 and CIFAR-100, 224 × 224 for Clothing 1M (after resizing to [Table] B.2 Simulated Label Noise Injection
Since the CIFAR-10 and CIFAR-100 are initially clean, we follow [25, 18] for symmetric and asymmetric label noise injection. Speciﬁcally, symmetric label noise is generated by randomly ﬂipping a certain fraction of the labels in the training set following a uniform distribution. Asymmetric label noise is simulated by ﬂipping their class to another certain class according to the mislabel confusions in the real world. For CIFAR-10, the asymmetric noisy labels are generated by mapping truck → automobile , bird → airplane , deer → horse and cat ↔ dog .
For CIFAR-100, the noise ﬂips each class into the next, circularly within super-classes. B.3 Training Procedure CIFAR with class-conditional noise : We use a ResNet-34 and train it using SGD with a momentum of 0.9, a weight decay of 0.001, and a batch size of 64. The network is trained for 500 epochs for both CIFAR-10 and CIFAR-100.
We use the cosine annealing learning rate where the maximum number of epoch for each period is 10, the maximum and minimum learning rate is set to 0.02 and 0.001 respectively.
Note that the reason that we train the model 500 epochs in total is not because of the slow convergence (Our method actually converges around 250 epochs, shown in Section E).
Instead, it is to fully evaluate whether the proposed method will overﬁt mislabeled samples, which avoids the interference caused by early stopping [61].
CIFAR with instance-dependent noise : We evaluate NAL on three types of label noise from the
PMD noise family for consistency with PLC [20].
We use the same network architecture PreAct
ResNet34 [14] as PLC.
We use SGD with a momentum of 0.9 and a weight decay of 5 × 10 − 4 .
Models are trained for 150 epochs with a batch size of 128.
The learning rate is initialized as 0.1 and is divided by 5 after 60 and 120 epochs.
ANIMAL-10N :
Following [15], we use VGG-19 with batch normalization and train it using SGD with a weight decay 0.001, batch size 128 and initial learning rate 0.1, which is reduced by a factor of 5 after 50 and 75 epochs (100 epochs in total).
Clothing1M : Following [16, 23], we use a ResNet-50 without pretrained parameters.
We train the model with batch size 64.
The optimization is done using SGD with a momentum 0.9, and weight decay 0.001.
We use the same cosine annealing learning rate as CIFAR-10 except the minimum learning rate is set to 0.0001 and total epoch is 250.
For each epoch, we sample 2000 mini-batches from the training data ensuring that the classes of the noisy labels are balanced.
Webvision : Following [10, 7], we use an InceptionResNetV2 as the backbone architecture. All other optimization details are the same as for CIFAR-10, except for the weight decay (0.0005) and the batch size (32). B.4 Hyperparameters Selection and Sensitivity We perform hyperparameter tuning via grid search: λ = [0 . 1 , 0 . 5 , 5 , 10 , 50] and α = [0 . 7 , 0 . 9 , 0 . 99] using a noisy validation set sampled from the noisy training set (Note that the reason why a noisy validation set works has been empirically explored in [7] and theoretically proved in [65]).
In our experiments, we set α = 0 . 9 for all datasets. For CIFAR-10, we set λ = 0 . For CIFAR-100, we set λ
= 10 . For ANIMAL-10N, we set λ = 0 . For Webvision and Clothing1M, we set λ = 50 . C More Results on Training Clothing1M from Scratch
Existing methods use the ResNet-50 pretrained on ImageNet.
Here we report the results when existing methods train the model from scratch in Table 4.
We run the ofﬁcial code of SCE, SELC, DivideMix and ELR+. SCE [23] is a noise robust loss function. SELC [12] is a label correction method.
These two methods only focus on modifying the loss function, thus no extra GPU space is required.
In contrast, DivideMix and ELR+ (i
[Table]
e. an improved version of ELR) are complex method that use multiple techniques to boost their performance.
Both of them use the two networks and mixup augmentations.
Other technique, such as weight average is also applied in ELR+.
Therefore, more GPU space is required for DivideMix and ELR+.
As we can observe in Table 8, the proposed method NAL still outperforms these methods when training the ResNet-50 from scratch.
[Table]
D More results of Estimated Targets
We report the confusion matrix for other levels of label noise (symmetric 40%, 80% and asymmetric 40%) on CIFAR-10.
Figure 5, Figure 7 and Figure 9 display the confusion matrix of noisy labels w.r.t.
the clean labels on CIFAR-10 with 40% symmetric, 80% symmetric and 40% asymmetric label noise respectively. Figure 6, Figure 8 and Figure 10 display the confusion matrix of corrected labels w.r.t. the clean labels on CIFAR-10 with 40% symmetric, 80% symmetric and 40% asymmetric label noise after using the proposed method, respectively. As we can observe, the corrected labels (estimated targets) yield better quality than the original noisy labels, even under the extreme label noise (e.g.
80% symmetric noise). [Figure] [Figure]
[Figure]
[Figure] [Figure] [Figure] E Accuracy Curves versus Epochs
In this section
[Figure]
we show the accuracy curves during training to show the noise robustness of NAL. As we can observe in Figure 11, CE ﬁts the whole noisy labels eventually, while NAL only ﬁts around 60% training samples. In Figure 12, the test accuracy of CE decreases due to memorization of noisy labels, while NAL won’t.
We also observe the similar results on CIFAR-10. [Figure] [Figure] [Figure]
A cc
[Figure]
a c y [Figure] [Figure] T e s t
A cc u r a c y
[Figure] NAL
We monitor the weight τ distribution during training on CIFAR-10 with 40% symmetric noise using
NAL. Figure 15 to Figure 17 show the results. We summarize the weight τ distribution into three stages. As can be observed, in the stage 1, clean samples and mislabeled samples have the similar τ .
By continuing to train the model with NAL, the weights for clean samples and mislabeled samples start to separate. Until stage 3, the weight of clean samples is increased towards 1.0, even for most hard samples (since the weight distribution has only a little overlap between clean and mislabeled samples).