Noise Attention Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Machine learning has been highly successful in data-driven applications but isoften hampered when the data contains noise, especially label noise.
When trainedon noisy labels, deep neural networks tend to ﬁt all noisy labels, resulting in poorgeneralization.
To handle this problem, a common idea is to force the model to ﬁtonly clean samples rather than the mislabeled ones.
In this paper, we propose asimple yet effective method that automatically distinguishes the mislabeled samplesand prevents the model from memorizing them, named Noise Attention Learning .
In our method, we introduce an attention branch to produce attention weightsbased on representations of samples.
The attention branch is learned to divide thesamples according to the predictive power in their representations.
We design thecorresponding loss function that incorporates the attention weights for training themodel without affecting the original learning direction.
Empirical results show thatmost of the mislabeled samples yield signiﬁcantly lower weights than clean ones.
Furthermore, our theoretical analysis shows that the gradients of training samplesare dynamically scaled by the attention weights, implicitly preventing memorizationof the mislabeled samples.
Experimental results on two benchmarks (CIFAR-10and CIFAR-100) and three real-world datasets (ANIMAL-10N, Clothing1M and
Webvision) demonstrate that our approach outperforms state-of-the-art methods.
Introduction
With the emergence of highly-curated datasets such as ImageNet [1] and CIFAR [2], deep neuralnetworks (DNNs) have achieved remarkable performance on many classiﬁcation tasks.
However, inreal-world applications, it is extremely time-consuming and expensive, sometimes even impossible tolabel a new large-scale dataset containing fully correct annotations.
To alleviate this problem, onemay obtain the data with lower quality annotations efﬁciently through online keywords queries [3] orcrowdsourcing [4] but noisy labels (e.g. a cat image is mislabeled as dog) are inevitably introducedconsequently.
Previous studies [5] demonstrate that noisy labels in training data are problematic fordeep neural networks, resulting in overﬁtting and performance degradation.
Therefore, designingrobust algorithms against noisy labels is of great practical importance.
Given the training data consists of clean (correctly labeled) samples and mislabeled samples, DNNshave been observed to correctly predict the true labels for all training samples during a early learningstage, and then start to make incorrect predictions as it gradually memorizes the mislabeled samples[6, 7] (see Figure 1(a) and 1(b)).
Based on this empirical ﬁnding, many existing approaches [8–11]use the training loss or prediction conﬁdence to identify mislabeled samples in the early learningstage, i.e., the samples with small loss or high conﬁdence are more likely to be clean samples.
Afterdividing the samples into clean and mislabeled sets, one can train the model only based on clean set[8, 11], or correct these mislabeled samples by pseudo-labels [9], or treat them as unlabeled samplesand train the model using semi-supervised learning algorithms [10].
However, these methods involvesophisticated training procedure modiﬁcations and sometimes require high computational costs.
[Figure]
high accuracy
In this paper we propose Noise Attention Learning (NAL) which enjoys simplicity and universalityfrom a new perspective.
The core idea behind NAL is to let the model automatically differentiatethe mislabeled samples from clean samples, and prevent the memorization of mislabeled samplesin training.
Since the memorization of DNNs has a preference for clean samples, the predictivepower of a sample’s representation aligns with its label cleanliness [6].
To transform this qualitativeobservation into a quantitive measure, we introduce an attention branch that outputs scalar attentionweights to indicate the predictive power of representations.
We incorporate the attention weights inour loss function, so that the attention branch can be learned to divide samples, i.e., clean samples arelearned to be associated with large weights, while mislabeled ones with small weights.
In this way,the proposed method does not require extra steps (e.g. modeling loss distribution [9, 10], selectingsmall-loss samples [8]) to separate the training set, effectively simplifying the training procedure.
Theoretically, we demonstrate that NAL neutralizes the effect of the mislabeled samples on thegradient and ensures the contribution of clean samples to the whole gradient remains dominant,thereby preventing the deep neural networks from overﬁtting mislabeled samples.
The proposed method has a similar effect on gradients as a regularization method ELR [7].
In general,the performance of pure regularization methods cannot be compared to complex methods.
Despitethat the regularization methods can effectively hinder the memorization of mislabeled samples, thelimited number of clean samples makes the model not rich enough to generalize effectively to theheld-out data.
To solve this problem, ELR uses temporal ensembling [12] to estimate the targets in itsloss function to generate more clean labels for improving performance.
Similarly, instead of directlyusing noisy labels as target distribution in our loss function, we adopt SELC [13] to estimate thetargets, allowing the generation of more clean labels for learning sufﬁciency.
Empirically, we show that the proposed approach achieves better robustness than the state-of-the-artmethods on two benchmarks (CIFAR-10 and CIFAR-100 [2]) with simulated label noise and threechallenging datasets in noisy label literature (ANIMAL-10N [14], Clothing1M [15] and Webvision[3]) with real-world label noise.
To better understand our method, we conduct many empiricalanalyses, including memorization analysis, gradient analysis, attention weights, quality of targets,hyperparameter sensitivity and ablation study, to verify our theoretical explanations and design goals.
Preliminaries
Classiﬁcation with Noisy Labels.
Consider the K -class classiﬁcation problem in noisy-label scenario, the ground truth label y is unobservable.
We only have a noisy training set
ˆ D = { ( x [ i ] , ˆ y
[ i ] ) }
Ni =1 ,where x [ i ] is an input and ˆ y
[ i ] ∈ Y = { 1 , . . .
, K } is the corresponding noisy label.
We denoteˆ y
[ i ] ∈ { 0 , 1 } K as one-hot vector of noisy label
ˆ y
[ i ] .
A DNN N θ maps an input x
[ i ] to a Kdimensional logits and then feeds the logits to a softmax function to obtain p
[ i ] of the conditional probability of each class.
θ denotes the parameters of the DNN and z [ i ] ∈ R K × 1 denotes the logits(i.e.
pre-softmax output).
We refer to it as prediction branch in this work.
We have z
[ i ] =
N θ ( x [ i ] )and p [ i ] =
softmax ( z [ i ] ) .
Without knowing the ground truth joint probability distribution P ( x , y ) ,the cross-entropy (CE) loss is often used as empirical risk to measure how well the model ﬁts thetraining set ˆ D as follows:
[Equation]
Early Learning Phenomenon When optimizing L ce by stochastic gradient descent (SGD), it hasbeen observed that DNNs ﬁt the easy (clean) samples ﬁrst before memorizing of hard (mislabeled)samples [6].
Figure 1(a) and 1(b) shows this qualitative behavior.
The model begins by learningto predict the true labels, even for majority of mislabeled samples, but eventually learns to predictswrong labels due to memorization of mislabeled samples.
Label Noise Model.
The generation of real-world label noise is unpredictable, a typical methodologyto dealing with noisy labels is to posit a noise model and develop robust algorithms under thismodel.
The algorithms are then tested on the real-world datasets to see how effective they are.
Awidely-accepted noise model is class-conditional noise [17, 18], where label noise is independentof input features and true labels are corrupted by a symmetric or asymmetric noise transition matrix(simulation details in Section 5).
Instance-dependent noise [19] is another noise model in which thenoise is affected not only by the label but also by the input feature.
Noise Attention Learning
In this section, we present our approach NAL for learning with noisy labels.
NAL consists of three keyelements: (1) An attention branch based on learned representations to produce the attention weightfor each training sample; (2) A noise attention loss is speciﬁcally designed to learn the attentionweights; (3) A target estimation strategy to help generate more clean labels for learning.
3.1 Attention Branch based on Learned Representations
To enable the deep neural networks to produce an attention weight that reﬂects the quality of learnedrepresentation, we introduce an attention branch just after the penultimate layer of the original model.
We denote M -dimensional penultimate layer (representation) of input x
[ i ] as H
[ i ] .
H [ i ] is shared inboth prediction and attention branches.
For each input, the prediction branch outputs the softmaxprediction p
[ i ] as usual.
The attention branch contains one fully connected layer to produce a singlescalar value
h
[ i ] , and sigmoid function is applied to scale it between 0 to 1.
Speciﬁcally, we haveh [ i ] =
WH [ i ] + b , where W ∈ R 1 × M denotes the weights and b ∈ R denotes the bias in penultimatelayer of the attention branch.
We have the attention weight tau
[ i ] for each sample x
[ i ] as
[Equation]
Note that W and b are extra network parameters need to be learned to produce meaningful τ ’s.
Therefore, attention weights are not ﬁxed but dynamic during the training.
3.2 Noise Attention Loss
The early learning phenomenon reveals that the DNNs memorize the clean samples before themislabeled samples.
Therefore, clean samples are likely to have better learned representations thanmislabeled samples in the early learning phase.
To enable the attention weights to automaticallycapture the difference of representations as well as learn the classiﬁcation task, we propose a noiseattention loss consisting of two terms: attention term L a and boost term L b .
[Equation]
where λ is the hyperparameter.
Intuitively, the attention term L a can be interpreted as a dynamicexpansion of the CE loss, where the prediction is modiﬁed to ‘perceptual shortcut’ τ ( p − ˆ y )
+ ˆ y
(Note that many existing methods [20, 21, 9] use this linear interpolation formula to infer the labeldistribution, while the proposed NAL applies it to prediction distribution for learning the meaningfulattention weights).
Assuming no representations are corrupted and all weights τ → 1 , the attentionterm L a reduces to exactly L ce in Eq.
On the contrary, when all τ → 0 , the attention term L abecomes “lazy" as it produces many zero gradients.
The mechanism of attention term can be explained as follows: (1) For mislabeled samples, themodel does not memorize them in the early learning phase.
Therefore, their representations areunder learned compared to clean samples and ( p − ˆ y ) remains large.
By minimizing L a , it forcesτ of mislabeled samples toward 0, which is treated as ‘shortcut’.
(2) For clean samples, the modelmemorizes them ﬁrst, resulting in ( p − ˆ y )
→ 0 .
It makes τ have no inﬂuence on minimizing L aas the prediction distribution is already equal to ˆ y .
As a result, by solely minimizing L a , we mayobtain a trivial optimization result that the model always produces τ → 0 for any inputs.
To avoidthis ‘lazy’ learning circumstance, we have another boost term L b .
It can be interpreted as a binary CEloss, where the target of τ is always 1 for all inputs.
By adding the boost term, the attention weightsof clean samples are pushed to 1, and weights of mislabeled samples are still close to 0 as expected.
To verify the effectiveness of L NAL in discriminating the mislabeled samples from clean samples, weempirically analyze the weight distribution of clean and mislabeled samples with simulated symmetricnoise.
Figure 2 shows results on CIFAR-10 and CIFAR-100 with different ratios of symmetric noise.
In 0% noise cases, we observe that all samples have weights close to 1, which indicates that L NALreduced to exactly L ce .
Thus, the proposed loss would not affect the performance when training withclean data.
At other ratios of label noise, there is a clear separation between attention weights ofclean and mislabeled samples.
The weights of clean samples form a spike close to 1.0, whereas theweights of most mislabeled samples are plainly less than 1.0, which satisﬁes our design goal.
3.3 Target Estimation
For regularization methods, directly using the noisy labels as target distribution in loss function isless effective as the limited number of clean samples makes the model not rich enough to generalizeeffectively to the held-out data, especially at the high noise level.
To yield better performance, ELR[7] use the semi-supervised technique temporal ensembling [12] to estimate targets in its regularizerand steer the model towards these targets.
The ensemble predictions are formed by an exponentialmoving average of network outputs.
Speciﬁcally, the ensemble predictions in the m -th epoch can beexpressed as ˜ p [
[Equation]
] = [Equation] 1 − α ) α m − j p [ j ] , where 0 ≤ α < 1 is the momentum.
In this paper, weuse the ensemble strategy in SELC [13] to estimate our targets t for the m -th epoch as follows:
[Equation]
initial term
[Equation]
softmax sigmoid
Update t [ i ] e 8: end for 9: Calculate L NAL = − 1 | B | (cid:80) | B | i =1 ( ˆ t [ i ] [ e ] ) (cid:62) log (cid:0) τ [ i ] ( p [ i ] − ˆ t [ i ] [ e ] ) + ˆ t [ i ] [ e ] (cid:1) − λ | B | (cid:80) | B | i =1 log( τ [ i ] ) .
Update θ using stochastic gradient descent.
for 12: end for 13: Output: θ .
The initial term preserves the original noisy labels with exponential decaying weights α m .
In the earlylearning phase (small m ), the overall target relies on noisy labels to learn the model.
As the learningcontinues (larger m ), α m approaches 0.
The overall target eventually depends on the ensemble term˜ p [ m ] and penalizes model predictions that are inconsistent with this target.
In our experiment, we willdemonstrate that estimated targets are more accurate than original noisy labels.
We simply replacenoisy label ˆ y with ensemble target t in our loss.
Algorithm 1 shows pseudocode of our method.
Theoretical Justiﬁcation
In this section, we demonstrate the noise robustness of L NAL by analyzing how it scales the gradientsaccordingly to prevent memorization of mislabeled samples.
The omitted proofs in this section arein supplementary materials.
For clarity of explanation, we denote the true label of sample x as y ∈{ 1 , ..., K } .
The ground-truth distribution over labels for sample x is q ( y | x ) , and
(cid:80) Kk
=1 q ( k | x ) = 1 .
Consider the case of a single ground-truth label y , then q ( y | x ) = 1 and q ( k | x ) = 0 for all k (cid:54)
=
y .
We denote the prediction probability as p ( k | x ) and (cid:80)
Kk =1 p ( k | x ) = 1 .
For notation simplicity, wedenote p k , q k , p y , q y , p j , q j as abbreviations for p ( k | x ) , q ( k | x ) , p ( y | x ) , q ( y | x ) , p ( j | x ) and q ( j | x ) .
First, we explain how the CE loss fails in noisy-label scenario.
Lemma 4.1.
Given the cross-entropy loss L ce in Eq.
(1), we rewrite the sample-wise loss (cid:96) ce
=− (cid:80)
Kk =
q k log p k .
Its gradient with respect to z j is
[Equation]
where z j is the j -th entry of logits z .
q j = q y = 1 means j equals the true class y .
In clean-label scenario, Lemma 4.1 ensures that, during SGD, learning direction of CE loss continuestowards the true class as the corresponding gradient remains negative.
However, in noisy-labelscenario, the learning direction ﬂuctuates.
Suppose j is true class and equals y , but q j = 0 due tothe label noise, then the contribution of a mislabeled sample to the gradient is reversed (i.e. gradientshould be negative but get positive instead).
The entry corresponding to the impostor class j (cid:48) , is alsoreversed because q j (cid:48) = 1 , causing the gradient of mislabeled samples dominates (in Figure 3(a)).
Consequently, using CE loss results in memorization of the mislabeled samples.
Theorem 4.1.
Given the noise attention loss L NAL in Eq.
(3), we rewrite the sample-wise loss (cid:96) NAL = −
[Equation]
Kk =1 q k log( τ [Equation] p k − q k ) + q k ) − λ log τ .
Its gradient with respect to the logits z j can be derived as
[Equation]
where the multiplier
[Equation]
[Figure]
Theorem 4.1 ensures that learning on true class persists when training with noise attention loss.
Inaddition, compared to the gradient of (cid:96) ce , the gradient of (cid:96) NAL is scaled by an positive multiplier termp y p y − 1+1 /τ .
Let’s denote ϕ = p y p
y − 1+1 /τ .
ϕ is monotonically increasing on τ .
We have lim τ → 1 ϕ = 1 ,and lim τ → 0 ϕ = 0 .
For the samples with the true class j in Eq.
(6a), the cross-entropy gradient termp j − 1 of clean samples tends to vanish after early learning stage, causing mislabeled samples todominate the gradient.
However, by multiplying ϕ (note that ϕ
→ 0 for mislabeled samples andϕ → 1 for clean samples due to attention weight distribution), it counteracts the effect of gradientdominating by mislabeled samples.
For the samples that j is not the true class in Eq.
(6b), thegradient term p j is positive.
Multiplying ϕ < 1 effectively dampens the magnitudes of coefﬁcientson these mislabeled samples, thereby diminishing their effect on the gradient.
Figure 3(b) empiricallyshows the gradient when training with L NAL .
NAL keeps the gradient of clean samples dominantand diminishes the gradient of mislabeled samples when epoch increases, effectively preventingmemorization of mislabeled samples.
Experiments
In this section, we ﬁrst compare NAL with the existing methods on ﬁve widely used datasets.
Thenwe provide several empirical results towards a better understanding of NAL.
Finally, we provide anablation study to evaluate the inﬂuence of each component and also show the results when integratedwith another noise robust method SCE [22].
All experiments are implemented in PyTorch and run ina single NIVDIA A100 GPU.
Code is provided in supplementary materials.
5.1 Comparison with Existing Methods
Datasets and Setups.
We evaluate our approach on two benchmarks CIFAR-10 and CIFAR-100[2] with simulated label noise, and three real-world datasets, ANIMAL-10N [14], Clothing1M [15]and WebVision [3].
Since CIFAR-10 and CIFAR-100 are initially clean, we follow [18] to injectsymmetric and asymmetric label noises.
Symmetric noise is generated by uniformly ﬂipping the labelto one of the other class label.
Asymmetric noise is a simulation of ﬁne-grained classiﬁcation, wherethe label ﬂipping only occurs within very similar classes (e.g. dog ↔ cat).
ANIMAL-10N containshuman-labeled online images for 10 animals with confusing appearance.
Its estimated noise rate is8%.
Clothing1M consists of 1 million images collected from online shopping websites with labelsgenerated from surrounding texts.
Its estimated noise rate is 38.5%.
WebVision contains 2.4 millionimages crawled from the web using the 1000 concepts in ImageNet ILSVRC12.
Its estimated noiserate is 20%.
For CIFAR-10/100, We use a ResNet-34 [23] and train it using SGD with a batch size of64.
For ANIMAL-10N, we use VGG-19 [24] with batch normalization and train it using SGD with abatch size of 128.
For Clothing1M, we train a ResNet-50 [23] using SGD with a batch size of 64.
For
Webvision, we train a InceptionResNetV2 [25] using SGD with a batch size of 32.
More trainingdetails can be found in supplementary materials.
Table 1 shows the results on CIFAR-10 and CIFAR-100 with different levels of symmetric andasymmetric label noise.
We compare NAL to the best performing approaches that only modify the
[Table] [Table]
[Table] [Table]
training loss.
All of these methods use the same backbone (ResNet34).
NAL shows substantialimprovements over other methods and obtains the highest accuracy in most cases.
Table 2 shows the results on ANIMAL-10N. NAL achieves state-of-the-art performance, even betterthan Nested which uses Co-teaching to boost performance.
Table 3 shows the results on the Clothing1M. All compared methods are trained using ResNet50pretrained on ImageNet.
However, as NAL needs to learn the parameters W and b in the attentionbranch, we directly train our model from scratch.
As we can observe from Table 3, NAL still slightlyoutperforms other methods even without using an ImageNet pretrained model.
Table 4 shows the results on the (mini) WebVision following [37].
All the compared methods areevaluated on WebVision and ImageNet ILSVRC12 validation sets.
We observe that NAL consistentlyoutperforms state-of-the-art methods on both validation sets in terms of top1 and top5 accuracy.
Note that we do not compare with some state-of-the-art methods like RoCL [41] and DivideMix[10] as baselines, because these methods are aggregations of multiple techniques (e.g. mixup [42],multiple networks [8], and complex augmentation [43]), while this paper only focuses on one, so thecomparison seems unfair.
[Figure]
[Figure] [Figure]
5.2 Empirical Analysis of NAL
Memorization Procedure of NAL.
Figure 1(c) and 1(d) show the memorization procedure of NALon clean and mislabeled samples, respectively.
Compared to CE loss in Figure 1(a) and 1(b), NALeffectively prevents memorization of mislabeled samples, allowing the model to continue learning onthe clean samples to attain high classiﬁcation accuracy.
Gradient of L ce and L NAL .
Figure 3(a) and 3(b) show the gradients of CE loss and noise attentionloss on CIFAR-10 with 40% symmetric noise using ResNet34 [23] respectively.
In comparison to CEloss, the proposed method keeps the gradient of clean samples dominant and dampens the gradient ofmislabeled samples, resulting in the whole gradient being guided by clean samples.
Attention Weights.
When trained on noisy labels, we have observed that the mislabeled sampleshave smaller weights than the clean samples in Figure 2.
Here, we report the average weights ofsamples in Figure 4(a).
The ( i, j ) -th block represents the average weights of samples with clean labeli and noisy label j .
We observe that the attention weights on the diagonal blocks are higher than thoseon non-diagonal blocks, which indicates NAL does properly down-weight the mislabeled samples.
Quality of Estimated Targets.
We evaluate the quality of new targets t by correction accuracydeﬁned via 1 N (cid:80) Ni 1 { argmax j y
[ i ] j = argmax j t
[ i ] j } , where y [ i ] is the true label of input x [ i ] .
Figure4(d) shows the correction accuracy vs. epochs on CIFAR-10 with different levels of label noise.
Thetarget estimation in Section 3.3 steadily improves the quality of new targets.
Figure 4(b) and Figure4(c) shows the confusion matrix of noisy labels and corrected labels (i.e. the hard version of newtargets) w.r.t the clean labels on CIFAR-10 with 60% symmetric noise respectively.
We observe thattarget estimation corrects the noisy labels impressively well for all classes.
Sensitivity of Hyperparameters λ and α .
NAL contains two hyperparameters: λ is the coefﬁcientfor boost term L b , α is the momentum in target estimation.
Figure 4(e) and 4(f) show their sensitivityto the performance.
We observe that the sensitivity to α is quite mild as long as it is set close to 1.λ need to be tuned according to the complexity of dataset.
It cannot be set to be very large or verysmall, resulting in neglecting attention term or falling into the ‘lazy’ learning circumstance.
In ourexperiments, we ﬁxed α = 0 .
9 for all datasets.
For CIFAR-10 and ANIMAL-10N, we set λ = 0 .
For CIFAR-100, we set λ
= 10 .
For Webvision and Clothing1M, we set λ = 50 .
[Table]
5.3 Ablation Study
Table 5 reports the inﬂuence of the two components in NAL: target estimation and attention branch.
Removing the target estimation leads to a signiﬁcant performance drop.
This suggests that usingtarget estimation is crucial as it generates more clean samples for the model to learn.
To validatethe effect of the attention branch, we conduct another way to calculate the attention weights: usingthe conﬁdence score (highest probability in output) as the weight,
i
[Table]
e., max j p j , j ∈ [1 , K ] .
Weobserve that the model does not converge in hard cases.
We conjecture that using the conﬁdence frommodel output does interfere with the original prediction branch, while adding the attention branchto get attention weight effectively solve this problem.
We also explore the possibility of integrating
NAL with other methods.
Here we adopt the idea from symmetric cross entropy (SCE) [22], whichadds the reverse term to achieve noise robustness.
As shown in Table 5, adding RNAL affects theperformance under CIFAR-10, but enjoys performance boost under CIFAR-100 without extra cost.
Related Work
We brieﬂy discuss the existing noise-robust methods that do not require a set of clean training data (asopposed to [15, 44, 45]).
Loss Correction.
Many approaches focus on correcting the loss functionexplicitly by estimating the noise transition matrix [46, 18, 47].
Robust Loss Functions.
Thesestudies develop loss functions that are robust to label noise, including L DMI [33], MAE [48], GCE[26], SCE [22], NCE [49], TCE [50] and GJS [51].
Above two categories of methods do not utilizethe early learning phenomenon.
Sample Selection
During the early learning stage, the samples withsmaller loss values are more likely to be the clean samples.
Based on this observation, MentorNet[52] pre-trains a mentor network for selecting small-loss samples to guide the training of the studentnetwork.
Co-teaching related methods [8, 53–55] maintain two networks, and each network istrained on the small-loss samples selected by its peer network.
Label Correction Joint Opt [27] and
PENCIL [56] replace the noisy labels with soft (i.e. model probability) or hard (i.e to one-hot vector)pseudo-labels.
[9] weigh the clean and mislabeled samples by ﬁtting a two-component Beta mixturemodel to loss values, and corrects the labels via mixup combination.
Similarly, DivideMix [10] trainstwo networks to separate the clean and mislabeled samples via a two-component Gaussian mixturemodel, and further uses MixMatch [57] to enhance the performance.
Regularization [58] prove thegradient descent with early stopping is an effective regularization to achieve robustness to label noise.[59] explicitly add the regularizer based on neural tangent kernel [60] to limit the distance betweenthe model parameters to initialization.
ELR [7] estimates the target by temporal ensembling [12] andadds a regularization term to cross-entropy loss to avoid memorization.
Other techniques, such asmixup augmentation [42], label smoothing [25] and weight averaging [61], can effectively improvethe performance under label noise.
Conclusion and Discussion
In this work, we propose NAL for learning with noisy labels.
Our method leverages an attentionbranch and a noise attention loss to learn the attention weights for distinguishing the mislabeledsamples from clean samples.
NAL can effectively diminish the gradient of mislabeled samples,mitigating the effect of noisy labels.
We also provide extensive empirical analyses and evaluate itseffectiveness across multiple datasets with different types and ratios of label noises.
There are still multiple open problems for future research.
Currently, label noise has been extensivelystudied in image classiﬁcation task.
Other research areas, such as graph learning and federatedlearning, remain to be explored.
On the methodological front, we hope that our work will triggerinterest in designing new network architectures that inherently provide robustness to label noise.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our work
advances the development of deep learning methods and is useful because
the accurate
annotations are hard and expensive to gather in practice.
Currently, we do not ﬁnd any
potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Appendix
in supplementary materials.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix in
supplementary materials.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The code isincluded in supplementary material.
Two datasets CIFAR-10 and CIFAR-100 will be
automatically downloaded when you run the code.
The information of ANIMAL-10N,
Clothing1M and Webvision dataset can be found online.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] We run experiments in three times with different random
seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We cite the papers
when using the datasets.
(b) Did you mention the license of the assets?
[No] The datasets used in this paper are
public.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]