L EARNING WITH N OISY L ABELS R EVISITED : A S TUDY U SING R EAL -W ORLD H UMAN A NNOTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Existing research on learning with noisy labels mainly focuses on synthetic label noise.
Synthetic label noise, though has clean structures which greatly enabled statistical analyses, often fails to model the real-world noise patterns.
The recent literature has observed several efforts to offer real-world noisy datasets, e.g., Food101N, WebVision, and Clothing1M. Yet the existing efforts suffer from two caveats: ﬁrstly, the lack of ground-truth veriﬁcation makes it hard to theoretically study the property and treatment of real-world label noise.
Secondly, these efforts are often of large scales, which may result in unfair comparisons of robust methods within reasonable and accessible computation power.
To better understand real-world label noise, it is important to establish controllable and moderate-sized real-world noisy datasets with both ground-truth and noisy labels.
This work presents two new benchmark datasets, which we name as CIFAR-10N, CIFAR-100N, equipping the training datasets of CIFAR-10 and CIFAR-100 with human-annotated real-world noisy labels that we collect from Amazon Mechanical Turk.
We quantitatively and qualitatively show that real-world noisy labels follow an instance-dependent pattern rather than the classically assumed and adopted ones (e.g., class-dependent label noise).
We then initiate an effort to benchmarking a subset of the existing solutions using CIFAR-10N and CIFAR-100N. We further proceed to study the memorization of correct and wrong predictions, which further illustrates the difference between human noise and class-dependent synthetic noise.
We show indeed the real-world noise patterns impose new and outstanding challenges as compared to synthetic label noise.
These observations require us to rethink the treatment of noisy labels, and we hope the availability of these two datasets would facilitate the development and evaluation of future learning with noisy label solutions.
I NTRODUCTION
Image classiﬁcation task in deep learning requires assigning labels to speciﬁc images.
Annotating labels for training use often requires tremendous expenses on the payment for hiring human annotators.
The pervasive noisy labels from data annotation present signiﬁcant challenges to training a quality machine learning model.
The problem of dealing with label noise has been receiving increasing attentions.
Typical approaches include unbiased estimators and weighted loss functions (Natarajan et al., 2013; Liu & Tao, 2015), loss correction (Patrini et al., 2017; Liu & Guo, 2020), sample selection aided (Jiang et al., 2018; Han et al., 2018; Yu et al., 2019), etc.
The majority of existing solutions are often developed under stylish synthetic noise model, where the noise rates are either class-dependent or homogeneous across data instances.
Recently, there has been efforts for establishing results for the more realistic instance-dependent setting, i.e, the noise transition differs in different parts of features (Xia et al., 2020b) or sub-populations (Wang et al., 2021; Zhu et al., 2021a), etc.
All of these noise models are proposed with some statistical assumptions, which facilitate the derivation of theoretical solutions.
Yet, it is unclear how the existing models captured the real-world scenario.
To empirically validate the robustness of proposed methods, synthetic noisy labels on CIFARand CIFAR-100 (Krizhevsky et al., 2009) are the most widely accepted benchmarks.
The literature has also observed approaches to the simulation of human annotators in data labeling (Hua et al., 2013; Long & Hua, 2015; Liao et al., 2021), and real-world label noise benchmarks, including
Food-101 (Bossard et al., 2014), Clothing-1M (Xiao et al., 2015), WebVision (Li et al., 2017), etc.
We summarize the above real-world noisy label datasets in Table 1.
While a more detailed description
[Table]
and discussion of the existing datasets can be found in the related works, we want to highlight several outstanding issues in existing benchmarks and evaluations.
As noted in Table 1, except for CIFAR related noisy label datasets, all other datasets suffer from at least one of the three caveats:
• Complex task (High-resolution): when learning with large-scale and relative high-resolution data, the complex data pattern, various augmentation strategies, the use of extra train or clean data, different computation power (for hyper-parameter tuning such as batch-size, learning rate, etc) jointly contribute to the model performance and then result in unfair comparison.
• Missing clean labels: the lack of clean labels for veriﬁcation in most existed noisy-label datasets makes the evaluation of robust methods intractable.
• Interventions: human interventions in data generation (Jiang et al., 2020) and non-representative data collection process (Song et al., 2019) might disturb the original noisy-label pattern.
In addition, despite synthetically labeled CIFAR datasets are popular and highly used benchmarks for evaluating the robustness of proposed methods, there exists no publicly available human annotated labels for CIFAR training datasets to perform either validation of existing methods or veriﬁcation of popular noise models 1 .
A human-annotated version of CIFAR datasets would greatly facilitate the evaluations of existing and future solutions, due to the already standardized procedures for experimenting with CIFAR datasets.
All above issues motivate us to revisit the problem of learning with noisy labels and establish accessible and easy-to-use, veriﬁable datasets that would be broadly usable to the research community.
Our contributions can be summarized as follows
: • We present two new benchmarks CIFAR-10N, CIFAR-100N which provide CIFAR-10 and CIFAR100 with human annotated noisy labels.
Our efforts built upon the CIFAR datasets and provide easily usable benchmark data for the weakly supervised learning community (Section 3).
We expect to continue to maintain the datasets to facilitate future development of results.
• We introduce new observations for the distribution of human annotated noisy labels on tiny images,
i
e., imbalanced annotations, the ﬂipping of noisy labels among similar features, co-existence of multiple clean labels for CIFAR-100 train images (which leads to a new pattern of label noise), etc.
We further distinguish noisy labels in CIFAR-10N and CIFAR-100N with synthetic classdependent label noise, from the aspect of noise transitions for different features qualitatively and quantitatively (via hypothesis testing)
(Section 4).
• We empirically compare the robustness of a comprehensive list of popular methods when learning with CIFAR-10N,
CIFAR-100N. We observe consistent performance gaps between human noise and synthetic noise.
The different memorization behavior further distinguishes the human noise and synthetic noise (Section 5).
1. R ELATED WORKS
Learning from noisy labels Earlier approaches for learning from noisy labels mainly focus on loss adjustment techniques.
To mitigate the impact of label noise, a line of approaches modify the loss of image samples by multiplying an estimated noise transition matrix (Patrini et al., 2017; Hendrycks et al., 2018;
Xia et al., 2019; Yao et al., 2020), re-weight the loss to encourage deep neural nets to ﬁt on correct labels (Liu & Tao, 2015), propose robust loss functions (Natarajan et al., 2013; Ghosh et al., 2017; Zhang & Sabuncu, 2018; Amid et al., 2019;
Wang et al., 2019; Liu & Guo, 2020), or introduce a robust regularizer (Liu et al., 2020; Xia et al., 2020a; Cheng et al., 2021; Wei et al., 2021).
Another line of popular approaches behaves like a semi-supervised manner which begins with a clean sample selection procedure, then makes use of the wrongly-labeled samples.
For example, several methods (Jiang et al., 2018; Han et al., 2018; Yu et al., 2019) adopt a mentor/peer network to select small-loss samples as “clean” ones for the student/peer network.
To further explore the beneﬁts of wrongly-labeled samples and improve the model performance, Li et al.
(2020) chose the Mix-match (Berthelot et al., 2019) technique which has shown success in semi-supervised learning.
Benchmarks noisy labels datasets Food-101 (Bossard et al., 2014),
Clothing-1M
(Xiao et al., 2015), WebVision (Li et al., 2017) are three large-scale noisy labeled web-image databases which consist of food images, clothes images or other web images, respectively.
However, the majority of images in these three datasets do not have a corresponding clean label to perform controlled veriﬁcation (e.g., verifying the noise levels).
Later, a much larger-scale food dataset is collected by
Lee et al.
(2018), which contains exactly the same classes as Food-101 (Bossard et al., 2014).
More recently, Peterson et al.
(2019) present a noisily labeled benchmarks on CIFAR-10 test dataset where each test image has 51 human annotated labels in average.
Jiang et al.
(2020) construct noisily labeled
Mini-ImageNet (Vinyals et al., 2016) and Stanford Cars datasets (Krause et al., 2013) with controlled noise levels by substituting human annotated incorrect labels for synthetic wrong labels.
S YNTHETIC LABEL NOISE
In this section, we discuss a few popular synthetic models for generating noisy labels.
We focus on a K -class classiﬁcation task.
Denote by D := { ( x n , y n ) } n ∈ [ N ] the training samples where [ N ] := { 1 , 2 , ..., N } .
( x n , y n ) s are given by random variables (
X, Y ) ∈
X × Y drawn from the joint distribution D , where X , Y can be viewed as the space of feature and label, respectively.
In real-world scenarios, a classiﬁer f only has access to noisily labeled training sets (cid:101) D := { ( x n , ˜ y n ) } n ∈ [ N ] .
We assume the noisy samples ( x n , ˜ y n ) s are given by random variables ( X, ˜ Y )
∈
X × ˜ Y which are drawn from the joint distribution (cid:101) D .
Clearly, there may exist n ∈ [ N ] such that y n (cid:54) =
˜ y n .
The ﬂipping from clean to noisy label is usually formulated by a noise transition matrix T ( X ) , with elements:
T i,j ( X ) :
= P ( ˜ Y =
j | Y =
i, X ) .
We shall specify different modeling choices of T ( X ) below.
2. C LASS - DEPENDENT LABEL NOISE
The ﬁrst family of noise transition matrix is the class-dependent noise where the label noise is assumed to be conditionally independent of the feature X .
Mathematically, T ( X ) ≡ T and
T i,j ( X ) = P ( ˜ Y =
j | Y = i ) , ∀ i, j ∈ [ K ] .
Symmetric T
The symmetric noise transition matrix (Natarajan et al., 2013) describes the scenario where an amount of human labelers maliciously assign a random label for the given task.
It assumes that the probability of randomly ﬂipping the clean class to the other possible class with probability (cid:15) .
Assume the noise level is (cid:15) , the diagonal entry of the symmetric T is denoted as T i,i = 1 − (cid:15) .
For any other off-diagonal entry T i,j where i (cid:54) =
j , the corresponding element is T i,j = (cid:15)/ ( K − 1) .
Asymmetric T
The asymmetric noise transition matrix (Patrini et al., 2017) simulates the case where there exists ambiguity classes, i.e, human labelers may wrongly annotate the truck as automobile due to the low-resolution images.
There are two types of widely adopted asymmetric T .
The assymetric-next T assumes that the clean label ﬂips to the next class with probability (cid:15) , i.e, i → ( i + 1) mod K for i ∈
[ K ] .
The assymetric-pair T considers [ K/ 2] disjoint class pairs ( i c , j c ) where i c < j c .
For c ∈ [ K/ 2] , T i c ,j c = T j c ,i c
= (cid:15) , and the diagonal entries are 1 − (cid:15) .
2. I NSTANCE - DEPENDENT LABEL NOISE
Beyond the feature-independent assumption, recent works pay more attention to a challenging case where the label noise is jointly determined by feature X and clean label Y .
There are some techniques for synthesizing the instance-dependent label noise, such as the polynomial margin diminishing label noise (Zhang et al., 2021b) where instances near decision boundary are easier to be mislabeled, the part-dependent label noise (Xia et al., 2020b) where different parts of feature may contribute different noise transition matrices, and the group-dependent label noise (Wang et al., 2021; Zhu et al., 2021a) where different sub-populations may have different noise rates.
All of these noise models are proposed with some statistical assumptions, which facilitate the derivation of theoretical solutions.
H UMAN ANNOTATED NOISY LABELS ON CIFAR-10, CIFAR-
In this section, we introduce two new benchmark datasets for learning with noisy labels: CIFAR10N and CIFAR-100N.
Both datasets are built using human annotated labels collected on Amazon
Mechanical Turk (M-Turk): We post CIFAR-10 and CIFAR-100 training images as the annotation
Human Intelligence Tasks (HITs), and workers receive payments by completing HITs.
3. CIFAR-10N REAL - WORLD NOISY LABEL BENCHMARK
CIFAR-10 (Krizhevsky et al., 2009) dataset contains 60k 32 × 32 color images, 50k images for training and 10k images for testing.
Each image belongs to one of ten completely mutually exclusive classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck.
Dataset collection We randomly split the training dataset of CIFAR-10 without replacement into ten batches.
In the Mturk interface, each batch contains 500 HITs with 10 images per HIT.
The training images and test dataset remain unchanged.
Each HIT is then randomly assigned to three independent workers.
Workers gain base reward $0.03 after submitting the answers of each HIT.
We reward workers with bonus salary if the worker contributes more HITs than the averaged number of submissions.
We did not make use of any ground-truth clean labels to approve or reject submissions.
We only block and reject workers who submit answers with ﬁxed/regular distribution patterns.
We defer more details of the dataset collection to Appendix A.
Dataset statistics For CIFAR-10N dataset, each training image contains one clean label and three human annotated labels.
We provide ﬁve noisy-label sets as follows.
• Aggregate: aggregation of three noisy labels by majority voting.
If the submitted three labels are different for an image, the aggregated label will be randomly selected among the three labels.
• Random i
( i ∈ { 1 , 2 , 3 } ) :
the i -th submitted label for each image.
Note our collection procedure ensures that one image cannot be repeatedly labeled by the same worker.
• Worst: dataset with the highest noise rate.
For each image, if there exist any wrongly annotated labels in three noisy labels, the worst label is randomly selected from wrong labels.
Otherwise, the worst label is equal to the clean label.
60.27% of the training images have received unanimous label from three independent labelers.
The noise rates of prepared ﬁve noisy label sets are 9.03% (Aggregate), 17.23% (Random 1), 18.12% (Random 2), 17.64% (Random 3) and 40.21% (Worst) .
A complete dataset comparison among existing benchmarks and ours are given in Table 1.
We defer the noise level of each batch to Table 4 (Appendix).
Aggregating the annotated labels signiﬁcantly decreases the noise rates.
All three random sets have ≈ 18% noise level.
To provide a challenging noisy setting, we also prepare a worst label set which serves to cover highly possible mistakes from human annotators on CIFAR-10.
3. CIFAR-100N REAL - WORLD NOISY LABEL BENCHMARK
CIFAR-100 (Krizhevsky et al., 2009) dataset contains 60K 32 × 32 color images of 100 ﬁne classes, 50000 images for training and 10000 images for testing.
Each (ﬁne) class contains 500 training images and 100 test images.
The 100 classes are grouped into 20 mutually exclusive super-classes.
Data collection We split the training dataset of CIFAR-100 without replacement into ten batches with ﬁve images per HIT.
Only one worker is assigned to each HIT.
We group the 100 classes into 20 disjoint super-classes (see Table 5 in the Appendix) which are slightly different from the“coarse” categories named in CIFAR-100.
The ﬁne label in each super-class is summarized in Table(Appendix).
Workers are instructed to ﬁrstly select the super-class for each image.
And will then be re-directed to the best matched ﬁne label.
Since some super-classes are hard to recognize without prior knowledge in biology, we provide workers with easy access to re-select the super-class, and every ﬁne class has an example image for references.
The rejecting rule and the bonus policy are the same as those in CIFAR-10N.
We defer more details of the dataset collection to Appendix B.
Dataset statistics For CIFAR-100N dataset, each image contains a coarse label and a ﬁne label given by a human annotator.
Most batches have approximately 40% noisy ﬁne labels and 25 % noisy coarse labels.
The overall noise level of coarse and ﬁne labels are 25.60% and 40.20% , respectively.
We defer the noise level of each batch to Table 7 (Appendix).
P RELIMINARY OBSERVATIONS ON CIFAR-10N, CIFAR-100N
In this section, we analyze the human annotated labels for CIFAR-10 and CIFAR-100.
We will empirically compare the noisy labels in CIFAR-10N and CIFAR-100N with class-dependent label noise both qualitatively and quantitatively.
4. T HE NOISY LABEL DISTRIBUTION
Observation 1: Imbalanced annotations Our ﬁrst observation is the imbalanced contribution of labels.
Note that while the number of images are the same for each clean label, across all the ﬁve noisy label sets of CIFAR-10N, we observe that human annotators have different preferences for similar classes.
For instance, they are more likely to annotate an image to be an automobile rather than the truck, to be the horse rather than the deer (see Figure 9 in the Appendix).
The aggregated labels appear more frequently in automobile and ship, and less frequently in deer and cat.
This gap of frequency becomes more clear in the worst label set.
In CIFAR-100N, human annotators annotate frequently on classes which are outside of the clean-coarse, i.e., 25% noisy labels fall outside of the super-class and 15% inside the super-class.
And the phenomenon of imbalanced annotations also appears substantially as shown in Figure 1, which presents the distribution of noisy labels for each selected ﬁne class.
“Man” appears ≥ 750 times, while “Streetcar” only has ≈ 200 annotations.
[Figure]
Observation 2: Noisy label ﬂips to similar features In CIFAR-100N, most ﬁne classes are more likely to be mislabeled into less than four ﬁne classes.
In Figure 2, we show top three wrongly annotated ﬁne labels for several ﬁne classes that have a relative large noise rate.
Due to the lowresolution of images, a number of noisy labels are annotated in pair of classes,
i
[Figure]
e, ≈ 20% of “snake” and “worm” images are mislabeled between each other, similarly for “cockroack”-“beetle”, “fox”- “wolve”
[Figure]
etc.
While some other noisy labels are more frequently annotated within more classes, such as “boy”-“baby”-“girl”-“man”, “shark”-“whale”-“dolphin”-“trout”
[Figure]
etc, which share similar features.
[Figure]
Observation 3: The pattern of noise transition matrices In the class-dependent label noise setting, suppose the label noise is conditional independent of the feature, the noise transition matrices of CIFAR-10N and CIFAR-100N are best described by a mixture of symmetric and asymmetric T .
For CIFAR-10N, we heatmap the aggregated noisy labels, random1 noisy labels and worst noise labels w.r.t.
the clean labels.
In Figure 3, the three noisy label sets share a common pattern: the clean label ﬂips into one or more similar classes more often.
The remaining classes largely follow the symmetric noise model with a low noise rate.
For example, “truck” and “automobile” ﬂip between each other more often ( ≈ 25% − 30% percentage), which is much larger than that of all other classes.
Besides, in the central area of each transition matrix, it is quite obvious that the clean label of animal classes ﬂips more often to other animals.
Similar observations hold in CIFAR-100N, where each class ﬂips to a few misleading classes with much higher probability than that of remaining ones (see
Figure 12 in the Appendix).
Apparently, current synthetic class-dependent noisy settings are not as complex as the real-world human annotated label noise.
[Table] [Table]
Worst label airplane automobile bird cat deer dog frog horse ship truck
[Table]
l
[Table]
l
[Table]
[Figure]
Observation 4: Label noise: bad news or good news?
During the label collection, there exist a non-negligible amount of the wrongly annotated classes that indeed co-exist in the corresponding images.
In other words, training images of CIFAR-100 may contain multiple labels rather than a single one.
We select several exemplary training images of CIFAR-100 where multiple labels appear (in Figure 4).
The annotated class also appears in the corresponding image while is deemed as a wrong annotation by referring to the ofﬁcially provided clean label.
The most frequent case is best described by the scenario where a man holding a ﬂatﬁsh in hands.
The clean label usually comes to “ﬂatﬁsh”, while human annotators are more likely to categorize these images into “man”.
We conjecture that with the increasing label dimension, the phenomenon of multiple clean labels might be a more common issue.
We leave more explorations for the future work.
[Figure]
4. H UMAN NOISY LABELS V . S . SYNTHETIC NOISY LABELS
Recall that, for the class-dependent label noise, we have P ( ˜ Y | X, Y ) =
P ( ˜ Y | Y ) , indicating the noise transitions are identical for different features.
For general instance-dependent label noise, the above equality may not hold.
In this subsection, we explore to what degree this feature-dependency holds by checking whether the equality is satisﬁed or not for different features.
In the following, by checking the equality for different features, we will show the feature-dependency from both a qualitative aspect and a quantitative aspect.
4.2. A QUALITATIVE ASPECT
We ﬁrst visualize the noise transitions for different features from a qualitative aspect.
Taking CIFAR10N as an example, each image in CIFAR-10 only appears once.
Without additional assumptions, we only have three noisy labels for each individual feature X , which makes it difﬁcult to accurately estimate the noise transition probability T ( X ) , ∀ X , even though the ground-truth clean label is available.
To make the estimation tractable, we consider estimations on locally homogeneous label noise, i.e., nearby features share the same T ( X ) .
See the rigorous deﬁnition as follows.
Deﬁnition 1 ( M -NN noise clusterability) (Zhu et al., 2021b)
We call (cid:101) D n satisﬁes M -NN ( MNearest-Neighbor) noise clusterability if the M -NN of x n have the same noise transition matrix as x n , i.e., T ( x n ) =
T ( x n i ) , ∀ i ∈
[ M ] where { x n i } i ∈
[ M ] denote the M -NN of x n .
With M -NN noise clusterability, we can estimate T ( X ) on each subset (cid:101) D n .
In our visualization (e.g., Figure 5), rather than manually ﬁx a particular M , we use the k -means algorithm to separate features belonging to the same true class to 5 clusters and adaptively ﬁnd a suitable M for each cluster.
Denote by I i,ν the set of instance indices from the ν -th cluster of clean class i .
Then for label
[Figure] [Figure]
[Figure]
noise in I i,ν , we assume it is feature-independent and denote the corresponding transition vector by p i,ν , where each element p i,ν [ j ] is expected to be P ( ˜ Y =
j | x n , n ∈ I i,ν , Y = i ) .
The transition vector p i,ν can be estimated by counting the frequency of each noisy class given noisy labels in
I i,ν .
For example, in Figure 5a, each row of the top-right subﬁgure titled “Class 5 (deer)” shows
P ( ˜ Y | X, Y = 5) for different clusters of X , i.e., transition vectors p 5 ,ν , ν ∈ { 1 , · · · , 5 } .
Before putting up a more formal testing, we clearly observe that different transition vectors across different feature clusters, signaling the fact that the noise transitions P ( ˜ Y | X, Y = 5) are feature-dependent.
For a controlled comparison, we synthesize the class-dependent label noise with the same expected noised transition matrix T :=
E [ T ( X )] as our human-level label noise and illustrate it in Figure 5b.
We can ﬁnd that different rows of each matrix in Figure 5b are very similar, showing the synthetic noise is feature-independent and our veriﬁcation method is valid.
4.2. A QUANTITATIVE ASPECT
In this section, we quantitatively compare human noise and synthetic class-dependent noise through hypothesis testing.
Formulation Following the clustering results in Section 4.2.1, we further statistically test whether the human-noise is feature-dependent or not.
For CIFAR-10N, the null hypothesis H 0 and the corresponding alternate hypothesis H 1 are deﬁned as:
H
[Equation]
: Human annotated label noise in CIFAR-10N is feature-independent ; H 1 : Human annotated label noise in CIFAR-10N is feature-dependent .
Note that the synthetic label noise illustrated in Figure 5b is supposed to be feature-independent, the above hypotheses are converted to:
H 0 : Human annotated label noise in CIFAR-10N is the same as the corresponding synthetic one; H 1 : Human annotated label noise in CIFAR-10N is different from the corresponding synthetic one.
From Figure 5, one measure of the difference between human noise and synthetic noise is the distance between transition vectors p i,ν across different noise, e.g., d (1)
i
[Equation]
As contrast, we need to compare d (1) i,ν with d (2) i,ν := (cid:107) p synthetic (cid:48)
[Equation]
22 , where p synthetic
[Equation]
i
[Equation]
denotes the transition vector from the same synthetic noise but different clustering result (caused by random data augmentation).
Intuitively, if d (1) i,ν is much greater than d (2)
i,ν , we should accept H 1 .
The above hypotheses are then equivalent to:
H 0 : { d (1) i,ν } i ∈
[ K ] ,ν ∈ [5] come from the same distribution as { d (2)
i,
ν } i ∈
[ K ] ,ν ∈ [5] ; H 1 : { d (1) i,ν } i ∈
[ K ] ,ν ∈ [5] come from different distributions from { d (2) i,ν } i ∈
[ K ] ,ν ∈ [5] .
We repeat the generation of d i,ν for 10 times, where the images are modiﬁed with different data augmentations each time.
We choose the signiﬁcance level α = 0 .
and perform a two-sided t-test w.r.t { d (1) i,ν } i ∈
[ K ] ,ν ∈ [5] and { d (2) i,ν } i ∈ [ K ] ,ν ∈ [5] .
As described in Figure 6, the p -value of the human noise l l ll lll l l l ll l l lll l l l lll
lll l ll ll l lll lllll lll l l l l l l ll l l l l l l l l l l l l l l l l l l l l l l
[Figure]
[Figure] [Figure]
label w.r.t each clean class (except for “automobile") in CIFAR-10N is less than 0 .
Most of the classes (except for three of them) achieve a p -value
that is smaller than 0 .
Since there exists classes in CIFAR-10N such that the null hypothesis is rejected with the signiﬁcance value α , hypothesis “ human annotated label noise in CIFAR-10N is feature-dependent ” is accepted.
Similar conclusion can be reached for CIFAR-100N (details in Appendix D).
L EARNING WITH CIFAR-10N AND CIFAR-100N
We reproduce several popular and state-of-the-art robust methods on synthetic noisy labels (using calculated transition matrices from our collected data) as well as our collected human annotated labels.
Most of the selected methods fall into loss correction, loss re-weighting, and loss regularization related methods, etc.
5. P ERFORMANCE COMPARISONS ON CIFAR-10N AND CIFAR-100N
For a fair comparison, we adopt ResNet-34 (He et al., 2016), the same training procedure and batch-size for all implemented methods.
More experiment details are deferred to the Appendix E. In
Table 2, ELR+ (Liu et al., 2020) maintains its state-of-the-art in most noise settings.
Note that both
ELR+ (Liu et al., 2020) and Divide-Mix (Li et al., 2020) adopt two networks with advanced strategies such as mix-up data augmentation, their performances on CIFAR-100N largely outperforms all other methods.
The performance gap between ELR and ELR+ becomes much larger when the noise level is high.
We also empirically test the performances of the above methods under the class-dependent noise settings which follow exactly the same T as appeared in CIFAR-10N and CIFAR-100N, details are deferred to Table 8 in the Appendix.
[Table]
Observation 5: Performance gap (human noise v.s. synthetic noise)
Continuing the reported observations in Section 4.1, in Table 3 we highlight that for most selected methods, class-dependent synthetic noise is much easier to learn on CIFAR-
[Table]
especially when the noise level is high.
The gap is less obvious for CIFAR-100.
However, we also observe that Divide-Mix (Li et al., 2020) fails to work well in low noise regime.
Besides, ELR (Liu et al., 2020) performs even slight better when learning with real-world human noise than that on the synthetic class-dependent noise settings
[Table] [Table]
5. M EMORIZATION EFFECTS
When learning with noisy labels on CIFAR-10 and CIFAR-100 datasets, empirical observations (Arpit et al., 2017;
Liu et al., 2020; Xia et al., 2020a; Zhang et al., 2021a) on synthetic noise settings suggest that deep neural networks ﬁrstly ﬁt on samples with clean labels, then gradually over-ﬁt and memorize samples with wrong/noisy labels.
We next explore the memorization of clean and noisy labels on CIFAR-10N and CIFAR-100N.
Deﬁnition 2 (Memorized feature)
In a K-class classiﬁcation task, given a trained classiﬁer f , a feature x and conﬁdence threshold η , x is memorized by f if ∃ i ∈
[ K ] s.t. P ( f ( x ) = i ) >
η .
In Figure 7, we train CE loss with a ResNet-34 (He et al., 2016) neural network on three noisy label sets of CIFAR-10N: aggre-label (left column), random-label1 (middle column) and worst-label (right column).
While visualizing the memorization ( η = 0 .
95 ) on training samples, we split the train data into two parts: images with clean labels (the annotation matches the clean label) and wrong labels (the rest).
Denote f H and f S as models trained on human noisy labels and synthetic noisy labels (with the same T ), respectively.
f S gradually memorizes both correct and wrong predictions, while f H only memorizes correct predictions on clean labels and wrong predictions on wrong labels.
[Figure]
Why memorizing effects differ Compared with synthetic label noise, human annotators are prone to providing wrong labels on more misleading/ambiguous images or complex patterns.
Therefore we conclude that, given the same noise level, learning with human noise labels is more challenging and deep neural nets over-ﬁt on features of wrong annotations inevitably.
C ONCLUSIONS
Building upon CIFAR datasets
[Figure]
we provide the weakly supervised learning community with two accessible and easy-to-use benchmarks: CIFAR-10N and CIFAR-100N.
We introduce new observations from human annotations such as imbalanced annotations, the ﬂipping of noisy labels among similar features, co-existing labels in CIFAR-100N, etc.
From the perspective of noise transitions, we qualitatively show that human noise is indeed feature-dependent and differs substantially from synthetic class-dependent label noise using hypothesis testing.
We empirically compare the robustness of a large quantity of popular methods when learning with CIFAR-10N, CIFAR-100N and synthetic noisy CIFAR datasets.
We also consistently observe the large performance gap between human noise and synthetic noise, as well as the different memorization behavior on training samples.
E THICS S TATEMENT
This paper does not raise any ethics concerns.
Our work contains the human subject study which involves only simply image annotation tasks in Amazon Mechanical Turk.
The study has been carefully reviewed and received Institutional Review Board (IRB) approval.
We have followed the outlined protocol to perform the data collection.
Our implemented human subject studies:
• Do not have negative consequences,
i
e., leaking the private information that would identify human annotators on Amazon Mechanical Turk.
• Do not misrepresent work that might be competing or related.
• Do not present misleading insights.
Our work does not present applications that can lead to misuse.
• Do not introduce bias or fairness concerns, and research integrity issues.
• Will publish our collected datasets once there are no conﬂicts of interest/sponsorship and accepted.
A PPENDIX
The Appendix is organized as follows:
• Section A: the details of dataset collection, processing and information of CIFAR-10N. • Section B: the details of dataset collection, processing and information of CIFAR-100N. • Section C: the comparison of label collection procedure between CIFAR and CIFAR-N. • Section D: hypothesis testing of CIFAR-100N. • Section E: additional experiment results and details.
Before proceeding to the appendix, we want to highlight our boarder impacts as follows.
Broader Impacts Our observations and contributions may have following potential broader impacts:
• Crowd-sourcing of data annotations in computer vision: CIFAR-N may be further used for studying/proposing simulations of human annotations in crowd-sourcing, where the expenses of obtaining human annotations are often tremendous.
• A template for hypothesizing label noise patterns: Our hypothesis testing method of instancedependent label noise may provide a quantitative tool for testing the simulated human labels.
• Benchmarking effort is important: Although learning from noisy labels has witnessed thriving developments, we often observed conﬂicting comparisons due to the randomness in the synthetic noisy labels.
While there exist several datasets with real human noise, we view our contribution as complementary to existing ones, due to the elaborated reasons above. We believe benchmarking existing and population solutions is an important technical contribution to the community. • Understanding real-world label noise: Our observations and the provided human-annotated labels help with understanding real-world label noise. Besides, our observations of the multi-label issues in CIFAR-100 impose a new label noise pattern that is largely neglected.
• Motivations for real-world label noise solutions: our observations, especially the memorizing effects of real-world label noise may provide the literature with motivations for addressing realworld label noise.
A CIFAR-10N REAL - WORLD NOISY LABEL BENCHMARK
In this section, we introduce the preparation for data collection, collection procedure of CIFAR-10N, the workers’ behaviors, and more detailed statistics of the obtained labels.
A. C ASE STUDIES BEFORE THE FORMAL COLLECTION
To make the collection procedure reasonable and efﬁcient, we ﬁrstly upload a few batches ( images / batch) to test the behaviors of workers.
Our observations show that the workers on image classiﬁcation tasks may possibly incur following phenomenons:
• Bots: with the appearance of bots, the accepted HIT may result in low-quality or meaningless responses if the bot is able to pick answers and maliciously/randomly submit them.
Otherwise, the bot accepts the HIT but could not submit annotations.
The accepted HIT may have to be re-assigned to another work after this HIT becomes expired which results in inefﬁcient data collection.
• A large variance of the workers’ contribution: empirical observations show that a large amount of workers contribute too few HITs, while some professional workers upload with much less time.
Thus, there exists a large variance in the number of the worker’s submitted HITs and the noise label pattern is substantially controlled by only a few workers.
• Incomplete submission: when there are more than one image per HIT, a worker would possibly neglect to click the label of one or more images, for example, the worker may only choose easy tasks to annotate and skip tough ones.
The incomplete submission complexes the reassignment of images with missing annotations, the processing of annotated results as well as the individual payment.
A. D ATASET COLLECTION
To alleviate the impacts of aforementioned phenomenons, we randomly split the training dataset of
CIFAR-10 without replacement into ten batches.
Each batch contains 500 HITs.
To ensure most workers do not contribute too few to the annotation task, we include ten 32 × 32 images per HIT.
Each HIT is then randomly assigned to three independent workers.
The worker gains a base reward $0.03 after submitting all annotations of one HIT within 2 minutes.
The worker can not submit the annotations unless all appeared images in the assigned HIT are labeled.
The averaged number of submitted HITs per worker is 7.
Workers with no less 7 submissions share $200 bonus rewards.
Note that we constrain the time duration for each assignment and re-design the interface, bots are less likely to ﬁnish our task either on time or under the procedure.
What is more, we didn’t make use of any ground-truth clean labels to approve or reject submissions.
We only block and reject workers who submit answers with ﬁxed/regular distribution patterns.
A. T HE WORKERS ’ BEHAVIORS
There are 747 independent workers contribute to the construction of CIFAR-10N. As depicted in
Figure 8a, most workers can submit the labels for ten images within one minute.
Although we do observe that a small amount of assignments ( ≤ 0 .
2% ) are ﬁnished within 10 seconds, which are likely to be low-quality responses.
The work time in seconds have the mean 46.7, the standard deviation 21.2 and the interquartile (25th percentile — 75th percentile) range is [31 , 58] .
Among theseworkers, most of them annotated more than 80 images.
The number of annotated images per work has the mean 201, the standard deviation 329 and the interquartile range is [30 , 220] .
[Figure]
A. M ORE DETAILED DATASET STATISTICS
Table 4 includes the summarized statistics of CIFAR-10N for each batch.
In Table 4, the statistics “Consensus" means the three labelers have a consensus on the label of the same image.
We do observe that several batches tend out to be more challenging for human workers to annotate, i.e, the noise rate appeared in Batch3 is clearly higher than those of Batch4 and Batch5.
The difference of noise rate is especially signiﬁcant on the “Worst" label set.
We conjecture that there might exist more human annotators who malicious submit low-quality annotations when working on Batch3.
Note that while the number of images are the same for each clean label, across all the ﬁve noisy label sets of CIFAR-10N, we observe that human annotators have different preferences for similar classes, i.e, they are more likely to annotate an image to be an automobile rather than the truck, to be the horse rather than the deer (see Figure 9).
The aggregated labels appear more frequently in automobile and ship, and less frequently in deer and cat.
This gap of frequency becomes more clear in the worst label set.
[Table]
[Figure]
[Table]
[Figure]
B CIFAR-100N REAL - WORLD NOISY LABEL BENCHMARK
In this section, we introduce the preparation for data collection, collection procedure of CIFAR-100N, and more detailed statistics of the obtained labels
. B. C ASE STUDIES BEFORE THE FORMAL COLLECTION
To make the collection procedure reasonable and efﬁcient, we ﬁrstly upload a few batches (500 images / batch) to test the behaviors of workers.
Our observations show that the workers on CIFARclassiﬁcation tasks may not only have mentioned phenomenons in A, but have following issues as well:
• Hard and time consuming: directly let workers to ﬁnd the best matched label for each image from 100 possible classes is time assuming.
Our case study shows that ﬁnding the label per image takes averaged 5-6 minutes.
The work load as well as the difﬁculty level stop many workers from contributing two or more submissions.
• Lack of background knowledge: distinguishing several classes require some preliminary knowledge in biology, for example, it is common for workers to select a wrong super-class, especially for animal related ones: “aquatic mammals" and “ﬁsh".
And the differences among some ﬁne labels are hard to recognize, i.e, “trees" (oak, palm, pine), “medium-sized mammals" (porcupine, possum, raccoon, skunk), etc.
• Label aggregation has few effects: our empirical observations show that the the decrease of noise rates from aggregated labels given by 3 independent workers is less signiﬁcant than the results on CIFAR-10N.
B. D ATASET COLLECTION
To deal with above mentioned issues, we ﬁrstly split the training dataset of CIFAR-100 without replacement into ten batches.
Each batch contains 1000 HITs.
We include ﬁve 96 × 96 images per HIT which are reshaped from 32 × 32 ones in CIFAR-100 train images.
Only one worker is assigned with each HIT.
Instead of requesting workers to ﬁnd the best matched label from 100 labels directly, we group the 100 classes into 20 super-classes which are slightly different from theraw “coarse" categories given by Krizhevsky et al.
The 20 newly deﬁned super-classes are summarized in Table 5.
And the new division in each super-class is stated in Table 6.
In order to reduce the workload of workers, they are instructed to ﬁrstly select the super-class for each image.
They will then re-directed to the corresponding 4-6 ﬁne labels.
Note that some super-classes are hard to recognize without some prior knowledge in biology, we provide workers with easy access to re-select the super-class for the current image, and every ﬁne label has an example image from
Google images for references..
If the worker luckily ﬁnds the most suitable ﬁne label from her point of view, we also supply the jumping button so that the worker efﬁciently goes to the next image or the ﬁnal submission of this HIT.
A worker gains base reward $0.07 after submitting the answers of each HIT within 6 minutes (averaged working time is less than 90 seconds).
Similar to the setting in the collection of CIFAR-10N, huge bonus applied to workers who contribute more HITs than the averaged number of submissions.
Workers who submit answers with ﬁxed/regular distribution patterns will be blocked and rejected all submitted results [Table] [Table]
[Table]
B. M ORE DETAILED DATASET STATISTICS
For CIFAR-100N dataset, each image contains a coarse label and a ﬁne label given by a human annotator.
Most batches have approximately 40% noisy ﬁne labels and 25 % noisy coarse labels.
The overall noise level of coarse and ﬁne labels are 25.60% and 40.20%, respectively.
A detailed summary of noise level is available in Table 7 which covers the statistics of each batch.
Human annotators annotate frequently on classes which are outside of the clean-coarse, i
[Table]
e., 25% noisy labels fall outside of the super-class and 15% inside the super-class
[Table]
[Table]
Imbalanced annotations.
The phenomenon of imbalanced annotations also appears substantially as shown in Figure
[Figure]
Noisy label ﬂips to similar features.
In CIFAR-100N, most ﬁne classes are more likely to be mislabeled into less than four ﬁne classes. In Figure
[Figure] we show top three wrongly annotated ﬁne labels for several ﬁne classes that have a relative large noise rate.
Due to the low-resolution of images, a number of noisy labels are annotated in pair of classes,
i
[Figure]
e, ≈ 20% of “snake” and “worm” images are mislabeled between each other, similarly for “cockroack”-“beetle”, “fox”-“wolve”, etc.
While some other noisy labels are more frequently annotated within more classes, such as “boy”-“baby”- “girl”-“man”, “shark”-“whale”-“dolphin”-“trout”, etc, which share similar features
[Figure]
Transition matrices of CIFAR-100N In the class-dependent label noise setting, suppose the label noise is conditional independent of the feature, the noise transition matrices of CIFAR-100N are best described by a mixture of symmetric and asymmetric T .
For CIFAR-100N, we heatmap the coarse and ﬁne noisy labels w.r [Figure]
t. the clean labels.
In Figure 12, the clean label ﬂips into one or more similar classes more often.
And the remaining classes follow the symmetric noise model with a low noise rate.
Apparently, current synthetic class-dependent noisy settings are not as complex as the real-world human annotated label noise
[Figure]
CIFAR AND CIFAR-N
We summarize the label collection procedure of CIFAR dataset as below:
• Images sources: the images of CIFAR come from down-scaled images from 80 million color images obtained by various search engines.
The corresponding searching terms are viewed as a noisy source label/class; • Remove irrelevant images: the designer provided each student (labeller) with a class.
Students were asked to verify all the images which were found with that class as the search term.
And reject images that are outside of the assigned class.
• Payment: the payment for each student/labeller per hour is ﬁxed.
• Noise rate: with additional personal veriﬁcation, the noise rate of each class is negligible, especially for CIFAR-10.
We highlight the differences of label collection procedure between CIFAR and CIFAR-N as below:
• Annotation method: CIFAR adopted search engines such
that they obtained images by specifying
the class name (conditioned on the label, from label to feature); while CIFAR-N makes use of CIFAR images and pays workers for annotating labels (from feature to label).
• Payment and incentives: CIFAR ﬁxed the payment for student workers (self-collected), there is no incentive to rush.
As for CIFAR-N, we did not have any constraints on the workers via Amazon Mechanical Turk, i.e., the salary, the education level, etc.
Besides, we set time limits and bonuses for each annotation task, there are incentives to rush and meanwhile provide high-quality labels.
• Noise rate: for CIFAR, the noise rate can be viewed as 0.
While for CIFAR-N, we provide several noisy label sets which are of various noise rates.
D H YPOTHESIS TESTING OF CIFAR-100N
We further statistically test whether the human-noise is feature-dependent or not in CIFAR-100N.
The null hypothesis H 0 and the corresponding alternate hypothesis H 1 are deﬁned as:
H 0 : Human annotated label noise in CIFAR-100N is feature-independent ; H 1 : Human annotated label noise in CIFAR-100N is feature-dependent .
Note that the synthetic label noise is supposed to be feature-independent, the above hypotheses are converted to:
H 0 : Human annotated label noise in CIFAR-100N is the
same as the
corresponding synthetic one
; H 1 : Human annotated label noise in CIFAR-100N is different from the corresponding synthetic one.
As implemented for CIFAR-10N, we adopt the distance between transition vectors p i,ν across different noise as measure of the difference between human noise and synthetic noise, e.g., d (1) i,ν := (cid:107) p human
i,ν − p synthetic i,ν (cid:107) 2 2 .
As contrast, we need to compare d (1) i,ν with d (2) i,ν := (cid:107) p synthetic (cid:48)
i,ν − p synthetic i,ν (cid:107) 2 2 , where p synthetic (cid:48)
i,ν denotes the transition vector from the same synthetic noise but different clustering result (caused by random data augmentation).
Intuitively, if d (1) i,ν is much greater than d (2)
i,ν , we should accept H 1 .
The above hypotheses are then equivalent to:
H 0 : { d (1) i,ν } i ∈
[ K ] ,ν ∈ [5] come from the same distribution as { d (2) i,ν } i ∈
[ K ] ,ν ∈ [5] ; H 1 : { d (1) i,ν } i ∈
[ K ] ,ν ∈ [5] come from different distributions from { d (2) i,ν } i ∈
[ K ] ,ν ∈ [5] .
We repeat the generation of d i,ν for 10 times, where the images are modiﬁed with different data augmentations each time.
We choose the signiﬁcance level α = 0 .
and perform a two-sided t-test w.r.t { d (1) i,ν } i ∈
[ K ] ,ν ∈ [5] and { d (2) i,ν } i ∈ [ K ] ,ν ∈ [5] .
As described in Figure 13, the p -value of approximately 50 classes is less than α = 0 .
Thus, the null hypothesis for the human noisy labels in CIFAR-100N is rejected with the signiﬁcance value α .
And we accept the hypothesis:
Human annotated label noise in CIFAR-100N is feature-dependent .
[Figure]
Beyond feature dependency As shown in Figure 13 (b), although the overall noisy ﬁne labels in CIFAR-100N are feature dependent, the noise transition vectors of around 50 classes can indeed be viewed as class dependent by referring to their signiﬁcance levels.
Thus, we can conclude that real-world human annotated noisy labels may be feature independent (class dependent) for certain classes.
E A DDITIONAL EXPERIMENT RESULTS E. P ERFORMANCE COMPARISONS ON SYNTHETIC CIFAR DATASETS
Continuing the empirical observations in Section 5.1, in this section, we adopt ResNet-34 (He et al., 2016), the same training procedure and batch-size for all implemented methods.
The synthetic CIFAR datasets generate synthetic noisy labels by using the same class-dependent noise transition matrices in CIFAR-10N and
CIFAR-100N.
The comparisons of test accuracies are summarized in Table 8.
For most methods, class-dependent synthetic noise is much easier to learn on CIFAR-10, especially when the noise level is high.
The performance difference between human noise and the synthetic noise is less obvious for CIFAR-100.
[Table]
E. E XPERIMENT DETAILS ON CIFAR-10N AND CIFAR-100N
The basic hyper-parameters settings for CIFAR-10N and CIFAR-100N are listed as follows: minibatch size (128), optimizer (SGD), initial learning rate (0.1), momentum (0.9), weight decay (0.0005), number of epochs (100) and learning rate decay (0.1 at 50 epochs).
Standard data augmentation is applied to each dataset.
Special treatments In the reproduced experiments, we use the default setting for Cores* (Cheng et al., 2021), ELR+ (Liu et al., 2020) and DivideMix since either advanced data augmentation strategies or two deep neural networks are adopted.
And we ﬁx a same pre-trained model for methods that require a CE warm-up or a pre-trained model.
E. C OMPUTING INFRASTRUCTURE
All our experiments run on a GPU cluster (500 GPUs of all kinds, mainly use 2080 Ti) for training and evaluation.