S YNTHETIC VS R EAL : D EEP L EARNING ON C ONTROLLED N OISE
Anonymous authors
Paper under double-blind review
A BSTRACT
Performing controlled experiments on noisy data is essential in thoroughly understanding deep learning across a spectrum of noise levels.
Due to the lack of suitable datasets, previous research have only examined deep learning on controlled synthetic noise, and real-world noise has never been systematically studied in a controlled setting.
To this end, this paper establishes a benchmark of realworld noisy labels at 10 controlled noise levels.
As real-world noise possesses unique properties, to understand the difference, we conduct a large-scale study across a variety of noise levels and types, architectures, methods and training settings.
Our study shows that: (1) Deep Neural Networks (DNNs) generalize much better on real-world noise.
(2) DNNs may not learn patterns ﬁrst on real-world noisy data.
(3) When networks are ﬁne-tuned, ImageNet architectures generalize well on noisy data.
(4) Real-world noise appears to be less harmful, yet it is more difﬁcult for robust DNN methods to improve.
(5) Robust learning methods that work well on synthetic noise may not work as well on real-world noise, and vice versa.
We hope our benchmark, as well as our ﬁndings, will facilitate deep learning research on noisy data
I NTRODUCTION
Deep Neural Networks (DNNs) trained on noisy data demonstrate intriguing properties.
For example, DNNs are capable of memorizing completely random training labels but generalize poorly on clean test data (Zhang et al., 2017).
When trained with stochastic gradient descent, DNNs learn patterns ﬁrst before memorizing the label noise (Arpit et al., 2017).
These ﬁndings inspired recent research on noisy data.
As training data are usually noisy, the fact that DNNs are able to memorize the noisy labels highlights the importance of deep learning research on noisy data.
To study DNNs on noisy data, previous work often performs controlled experiments by injecting a series of synthetic noises into a well-annotated dataset.
The noise level p may vary in the range of 0% - 100% , where p = 0% is the clean dataset whereas p = 100% represents the dataset of zero correct labels.
The most commonly used noise in the literature is uniform (or symmetric) labelﬂipping noise, in which the label of each example is independently and uniformly changed to a random (incorrect) class with probability p .
Controlled experiments on noise levels are essential in thoroughly understanding a DNN’s properties across a spectrum of noise levels and faithfully comparing the strengths and weaknesses of different methods.
The synthetic noise enables researchers to experiment on controlled noise levels, and drives the development of theory and methodology in this ﬁeld.
On the other hand, some studies were also veriﬁed on real-world noisy datasets, e.g. on
WebVision (Li et al., 2017a), Clothing-1M (Xiao et al., 2015), Fine-grained Images (Krause et al., 2016), and Instagram hashtags (Mahajan et al., 2018), where the images are automatically tagged with noisy labels according to their surrounding texts.
However, these datasets do not provide true labels for the training images.
Their underlying noise levels are not only ﬁxed but also unknown, rendering them infeasible for controlled studies on noise levels.
In this paper, we refer image-search noise in these datasets as “real-world noise” to distinguish it from synthetic label-ﬂipping noise.
To study real-world noise in a controlled setting, we establish a benchmark of controlled real-world noisy labels, building on two existing datasets for coarse and ﬁne-grained image classiﬁcation: MiniImageNet (Vinyals et al., 2016) and Stanford Cars (Krause et al., 2013).
We collect noisy labels using text-to-image and image-to-image search via Google Image Search.
Every training image is independently annotated by 3-5 workers, resulting in a total of 527,489 annotations over 147, images.
We create ten different noise levels from 0% to 80% by gradually replacing the original images with our annotated noisy images.
Our new benchmark will enable future research on the real-world noisy data with a controllable noise level.
We ﬁnd that real-world noise possesses unique properties in its visual/semantic relevance and underlying class distribution.
To understand the differences, we conduct a large-scale study comparing synthetic noise, namely blue-pilled noise (or Blue noise ), and real-world noise (or Red noise 1 ).
Speciﬁcally, we train DNNs across 10 noise levels, 7 network architectures, 6 existing robust learning methods, and 2 training settings (ﬁne-tuning and training from random initialization).
Our study reveals several interesting ﬁndings.
First, we ﬁnd that DNNs generalize much better on real-world noise than synthetic noise.
Our results verify Zhang et al.
(2017)’s ﬁnding of deep learning generalization on synthetic noise.
However, we observe a considerably smaller generalization gap on real-world noise.
This does not mean that real-world noise is easier to tackle.
On the contrary, we ﬁnd that real-world noise is more difﬁcult for robust DNNs to improve.
Second, our results substantiate Arpit et al.
(2017)’s ﬁnding that DNNs learn patterns ﬁrst on noisy data.
But we ﬁnd this behavior becomes insigniﬁcant on real-world noise and completely disappears on the ﬁne-grained classiﬁcation dataset.
This ﬁnding lets us rethink the role of “early stopping” (Yao et al., 2007; Arpit et al., 2017) on real-world noisy data.
Third, we ﬁnd that when networks are ﬁne-tuned, ImageNet architectures generalize well on noisy data, with a correlation of r = 0 .and 0 .
89 for synthetic and real-world noise, respectively.
This ﬁnding generalizes Kornblith et al.
(2019)’s ﬁnding, i.e. ImageNet architectures generalize well across clean datasets, to the noisy data.
Our contribution is twofold.
First, we establish a large benchmark of controlled real image search noise.
Second, we conduct perhaps the largest study in the literature to understand DNN training across a wide variety of noise levels and types, architectures, methods, and training settings.
We hope our benchmark along with our ﬁndings, resulted from a considerable amount of manual labeling effort ( ∼ 520K annotations) and computing resources ( ∼ 3K experiments), will facilitate future deep learning research on real-world noisy data.
Our main ﬁndings are summarized as follows:
DNNs generalize much better on real-world noise than synthetic noise.
Real-world noise appears to be less harmful, yet it is more difﬁcult for robust DNN methods to improve.
DNNs may not learn patterns ﬁrst on the real-world noisy data.
When networks are ﬁne-tuned, ImageNet architectures generalize well on noisy data.
Adding noisy examples to a clean dataset may improve performance as long as the noise level is below a certain threshold (30% in our experiments).
R ELATED W ORK
Noisy Datasets : to understand deep learning’s properties on noisy training data, research often conducted experiments across a series of levels of synthetic noises.
The most common one is uniform label-ﬂipping noise ( aka.
symmetric noise), in which the label of each example is independently and uniformly changed to a random (incorrect) class with a probability (Zhang et al., 2017; Arpit et al., 2017; Vahdat, 2017; Shu et al., 2019; Jiang et al., 2018; Ma et al., 2018; Han et al., 2018; Li et al., 2019; Arazo et al., 2019).
The synthetic noisy dataset enables us to experiment on controlled noise levels, and drive the development of theory and methodology in this ﬁeld.
Research have also examined other types of noise to better approximate the real-world noise distribution, including class-conditional noises (Patrini et al., 2017; Rolnick et al., 2017), noises from other datasets (Wang et al., 2018), etc .
However, these noises are still synthetic, generated from artiﬁcial distributions.
Furthermore, different types of synthetic noises may lead to inconsistent or even contradicting observations.
For example, Rolnick et al.
(2017) experimented on a slightly different type of uniform noise and surprisingly found that DNNs are robust to massive label noise.
On the other hand, studies have also veriﬁed DNNs on real-world noisy datasets.
While other noise types exist, e.g. image omission and registration noise (Mnih & Hinton, 2012) or image corruption (Hendrycks & Dietterich, 2019), the most common type consists of images that are automatically tagged according to their surrounding texts either by directly crawling the web pages, e.g. Clothing-1M (Xiao et al., 2015), Instagram (Mahajan et al., 2018), or by querying an image search engine, e.g. WebVision (Li et al., 2017a).
Several studies have used these datasets.
For example, Guo et al.
, Jiang et al.
and Song et al.
(2018) veriﬁed their model on WebVision.
Mahajan et al.
(2018) trained large DNNs on noisy Instagram hashtags.
As these datasets did not provide true labels for the training examples, methods could only be tested on a ﬁxed and, moreover, unknown noise level.
To the best of our knowledge, there have been no studies focused on investigating real noisy labels in a controlled setting.
The closest work to ours is annotating a small set for evaluation (Veit et al., 2017) or estimating the noise level of the training data (Krause et al., 2016).
Robust Deep Learning Methods : robust learning is experiencing a renaissance in the deep learning era.
Since training data usually contain noisy examples, the ability of DNNs to memorize all noisy training labels often leads to poor generalization on the clean test data.
Recent contributions based on deep learning handled noisy data in multiple directions including, e.g. , dropout (Arpit et al., 2017) and other regularization techniques (Azadi et al., 2016; Noh et al., 2017), label cleaning/correction (Reed et al., 2014; Goldberger & Ben-Reuven, 2017; Li et al., 2017b; Veit et al., 2017), example weighting (Jiang et al., 2018;
Ren et al., 2018; Shu et al., 2019; Jiang et al., 2015;
Liang et al., 2016), semi-supervised learning (Hendrycks et al., 2018; Vahdat, 2017), data augmentation (Zhang et al., 2018; Cheng et al., 2019), etc .
Few studies have systematically compared these methods across different noise types and training conditions.
In our study, we select and compare six methods from four directions: regularization, label learning, example weighting, and mixup augmentation.
See Section 4 for details.
These examined methods are selected because they (i) represent a reasonable coverage of different ways of handling noisy data; (ii) are comparable to the state-of-the-art on the commonly used CIFAR-100 with synthetic noise.
D ATASETS
Our benchmark is built on two existing datasets: Mini-ImageNet (Vinyals et al., 2016) for coarse image classiﬁcation and Stanford Cars (Krause et al., 2013) for ﬁne-grained classiﬁcation.
MiniImageNet provides images of size 84x84 with 100 classes from the ImageNet dataset (Deng et al., 2009) 2 .
We select 50,000 images for training and 5,000 from ImageNet for testing.
Note that unlike few-shot learning, we train and test on the same 100 classes.
The Stanford Cars contain 16, high-resolution images of 196 classes of cars (Make, Model, Year) splitting in a 50-50 into training and test set.
The standard test split is used.
To recap, let us revisit the construction of existing noisy datasets in the literature. For the realworld noisy datasets (Xiao et al., 2015; Li et al., 2017a), one automatically collects images for a class by matching the class name to the images’ surrounding text (by web crawling or equivalently querying the crawled index). The retrieved images include false positive (or noisy) examples, i.e. text match/search thinks an image is a positive when it is not. As their training images are not manually labeled, the data noise level is ﬁxed and unknown.
As a result, these datasets are unsuitable for controlled studies.
On the other hand, the synthetic noisy dataset is built on the well-labeled dataset. The label of each training example is independently changed to a random incorrect class 3 with a probability p , called noise level, which indicates the percentage of training examples with false labels. Since the ground-truth labels for every image are known, previous studies enumerate p to obtain datasets of different noise levels and use them in controlled experiments. On balanced datasets, such as MiniImageNet and Stanford Cars used in our study, the above process is equivalent to ﬁrst sampling p % training images from a class and then replacing them with the images uniformly drawn from other classes.
The drawback is that their noisy labels are artiﬁcial and not follow the distribution of the real-world noise (Xiao et al., 2015; Li et al., 2017a;
Krause et al., 2016).
For our datasets, we follow the construction of synthetic datasets with only one difference, i.e. we draw false positive (noisy) examples from similar noise distributions as in existing real-world noisy datasets (Li et al., 2017a; Xiao et al., 2015).
To be speciﬁc, we draw images using Google text-to-image search, which is commonly used to get noisy labels in prior works (Bootkrajang & Kab´an, 2012; Li et al., 2017a; Krause et al., 2016; Chen & Gupta, 2015; Wang et al., 2014). In addition, we also include noisy examples using image-to-image search to enrich the type of label noises in our dataset. We manually annotate every retrieved image to identify the ones with false labels. For each class, we replace p % training images in the Mini-ImageNet and
Stanford Cars datasets with these false-positive images. We enumerate p in 10 different levels: { 0% , 5% , 10% , 15% , 20% , 30% , 40% , 50% , 60% , 80% } to study noisy labels in the controlled setting. Since Mini-ImageNet and Stanford Cars are all collected from the web images, their true positive examples should follow a similar distribution as the added false positive images.
The constructed datasets hence contain label noise similar to existing real-world datasets and are suitable to be used in controlled experiments. We sample noisy images using Google Image Search 4 in three steps: images collection, deduplication, and annotation. In the ﬁrst step, we combine images independently retrieved from two sources (1) text-to-image and (2) image-to-image search. For text-to-image (or text-search in short), we formulate a text query for each class using its class name and broader category ( e.g. cars) to retrieve the top 5,000 images.
For image-to-image search (or image-search), we query the search engine using every training image in Mini-ImageNet (50,000) and Stanford Cars (8,144). This collects a large pool of similar images.
As different query images may retrieve the same image, we rank the retrieved images by their number of occurrences in the pool and remove images that occur less than 3 times.
Finally, we union text-search and image-search images, where the text-search images accounts for 72% of our ﬁnal dataset.
For deduplication, following (Kornblith et al., 2019), we run a CNN-based duplicate detector over all images to remove near-duplicates to any of the images in the test set.
All images are under the usage rights “free to use or share” 5 .
These images are then annotated on a cloud labeling platform of high-quality labeling professionals.
The annotator is asked to provide a binary label to indicate whether an image is a true positive of its class.
Every image is independently annotated by 3-5 workers to improve the labeling quality, and the ﬁnal label is reached by majority voting.
In total, we have 372,428 annotations over 94, images on Mini-ImageNet, and 155,061 annotations over 51,687 images on Stanford Cars, out of which there are 28,691 and 12,639 image with false (noisy) labels.
Using these noisy image, we replace p % of the original training images in Mini-ImageNet and Stanford Cars.
Similar to the synthetic noise, p is made uniform across classes, e.g. p = 20% means that every class has roughly 20% false labels.
Besides, we also append all annotated images to the original datasets and obtain two larger augmented datasets.
The last two rows of Table 1 list the two augmented datasets and their underlying noise levels ( 19% and 21% ).
We report their performance in Section 5.3.
[Table]
For comparison, we also construct 10 uniform label-ﬂipping datasets under the same noise levels.
For convenience, we will use Blue Noise to denote the synthetic noise and Red Noise for the real-world image search noise.
Table 1 summarizes the datasets.
The test set in each dataset ( i.e. Mini-ImageNet and Stanford) is shared across all training conditions such that their results are comparable.
Red Mini-ImageNet is smaller because we ran out of noisy images for some common classes like “carton” and “hotdog”.
For common classes, it becomes more difﬁcult to get noisy images.
On average, we can get only one noisy label after labeling every 22 images.
The size difference in Mini-ImageNet may not be a problem for our study as they have similar test performance and we also verify on Stanford Cars whose size is the same for Blue and Red noise. [Figure] t an f o r d C a r s
[Figure]
text search image
[Figure] I [Figure] age
[Figure]
t
True positive
Blue Noise (symmetric label-flipping)
[Figure] Fig. 1 illustrates the true positive, blue, and red noisy images. For reference, Fig.
11 in the Appendix shows noisy training images on the WebVision benchmark (Li et al., 2017a) in which the true labels are not provided.
There are two noticeable differences between the blue and red noise:
• Real-world noisy images are more visually or semantically relevant to the true positive images.
• Real-world noisy images may come outside the ﬁxed set of classes in the dataset.
For example, the noisy images of “ladybug” include “ﬂy” and other bugs that do not belong to any of the class in Mini-ImageNet.
To understand their differences, we will compare Blue and Red noise in the rest of this paper.
M ETHODS
This paper evaluates robust deep learning methods on the introduced benchmark.
We select six methods from four directions that deal with noisy training data: (a) regularization, (b) label/prediction correction, (c) example weighting, and (d) vicinal risk minimization.
These methods are selected because they (i) represent a reasonable coverage of recent work, (ii) are competitive to the state-ofthe-art on the common CIFAR-100 dataset with synthetic noise.
Our study seeks the answer to the following questions:
How does their performance differ on synthetic versus real-world noise?
What is their real performance gap when each method is extensively tuned for every noise level?
As the noise levels span across a wide range from 0% to 80%, we ﬁnd the hyperparameters of robust
DNNs are important.
By extensively searching hyperparameters for every noise level, these methods can be very competitive to the state-of-the-art on the commonly used CIFAR-100 with synthetic noise.
See Table 3 in the Appendix.
To answer the second question, we need to train a formidable number of experiments, e.g. 920 experiments on a single dataset!
Daunting as it may seem, the experiments are, however, necessary to ensure the improvement stems from the methodology as opposed to favorable hyperparameter settings.
To brieﬂy introduce these methods, consider a classiﬁcation problem with training set D
= { x 1 , y 1 ) , · · · , ( x n , y n ) } , where x i denotes the i th training image and y i ∈
[1 , m ] is an integer-valued noisy label over m possible classes.
Let g s ( x i ; w ) denote the prediction of our DNN, parameterized by w ∈ R d .
In vanilla training, we optimize the following objective:
[Equation]
where (cid:96) ( y i ,g s ( x i , w )) , or (cid:96) i for short, is the cross-entropy loss with Softmax.
θ is the decay parameter on the l 2 norm of the model parameters.
Weight decay and dropout are two classical regularization methods.
In Weight Decay : we tune θ in { e − 5 , e − 4 , e − 3 , e − 2 } and set its default value to e − 4 which is the best value found on the ILSVRC dataset (Deng et al., 2009).
For Dropout (Srivastava et al., 2014), following (Arpit et al., 2017), we apply a large dropout ratio for noisy data and tune its keep probability in { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } .
By default, we disable dropout in training following the advice in (Kornblith et al., 2019).
We select two methods for label/prediction correction.
Reed (Reed et al., 2014) is a method for correcting the loss with the learned label.
The soft version is used for its better performance.
Let softmax ( g s ( x i ; w )) =
[ q i 1 , . . .
, q im ] denote the prediction for the i th image.
Reed replaces the loss in Equation 1 with:
[Equation]
where 1 condition is 1 if the condition is true, 0 otherwise.
Reed weights the cross-entropy loss computed over the noisy label (ﬁrst term) and the learned label (second term).
We tune the hyperparameter β in { 0 . 95 , 0 .
S-model (Goldberger & Ben-Reuven, 2017) introduces a convenient way to append a new layer to a DNN to learn noise transformation so as to “correct” the predictions.
Let z i denote the unknown true label for the i th image, and q i and ˆ q
i denote the original and the learned prediction.
It estimates the prediction over the true label by the learned conditional probability:
[Equation]
This is implemented as a label transition layer parameterized by B ∈ R m × m .
We have ˆ q i = softmax ( B ) q i , where the softmax is applied over B i, : , ∀ i ∈ [1 , m ] .
According to the paper, we initialize B = log((1 − (cid:15) )
I + (cid:15) × 1 m − 1 J ) , where I and J are the identity and the all-one matrix; (cid:15) is a small constant set to e − 6 .
MentorNet (Jiang et al., 2018) is a competitive example-weighting method, which aims to assign smaller weights to noisy examples.
It introduces the learnable latent weight variable v for every training example and adds a regularization term over the weight variable:
[Equation]
When w is ﬁxed, solving Equation 4 yields a weighting function that is monotonically decreasing with the example loss (cid:96) i :
[Equation]
where λ 1 and λ 2 are parameters.
We employ the predeﬁned MentorNet to compute the example weight in Equation 5 at the mini-batch level.
It tracks the moving average of the p -percentile of the loss inside every mini-batch and sets λ 1 and λ 2 accordingly.
Following the paper, we set the burn-in epoch to 10-20% of the total training epochs and tune the hyperparameter p -percentile in { 85% , 75% , 55% , 35% } .
Mixup (Zhang et al., 2018) is a simple and effective method for robust training.
It minimizes the vicinal risk (cid:96) (ˆ y i ,g s (ˆ x i , w )) calculated from:
[Equation]
where y i , y j ∈ R m are two one-hot label vectors and the pairs ( x i , y i ) and ( x j , y j ) are drawn at random from the same mini-batch.
The mixing weight λ is sampled from a conjugate prior Beta distribution λ ∼ Beta ( α, α ) for α > 0 .
Following the paper, we search the hyperparameter α in { 1 , 2 , 4 , 8 } for noisy training data.
For each method, we examine two training settings: (i) ﬁne-tuning from the ImageNet checkpoint and (ii) training from scratch.
For method comparison, Inception-ResNet-V2 (Szegedy et al., 2017) is used as the default network architectures.
For vanilla training, we also experiment with six other architectures: EfﬁcientNet-B5 (Tan & Le, 2019), MobileNet-V2 (Sandler et al., 2018),
ResNet-50 and ResNet-101 (He et al., 2016), Inception-V2 (Ioffe & Szegedy, 2015), and InceptionV3 (Szegedy et al., 2016).
The top-1 accuracy of these architectures on the ImageNet ILSVRCvalidation ranges from 71.6% to 83.6%.
We ﬁrst train our networks to get the best result on the clean training dataset and ﬁx the setting across all noise levels, e.g. learning rate schedule and maximum epochs to train.
See Section A.1 in the Appendix for the detailed implementation.
R ESULTS 5. V ANILLA T RAINING
Fig.
2 plots the training curve (gray) and the test curve (colored) on Blue and Red noisy benchmarks using vanilla training.
The ﬁrst two columns of Fig.
2 show the synthetic (in blue) and real-world noise (in red), respectively, where the x -axis is the training step.
The colored belt plots the 95% conﬁdence interval over 10 noise levels and the solid curve highlights the 40% noise level.
The ﬁrst row in each sub-ﬁgure is training from scratch and the second row is ﬁne-tuning.
Two classiﬁcation accuracies on the test set are compared.
The peak accuracy denotes the maximum test accuracy throughout the training.
The converged accuracy is the test accuracy after training has converged, which, for most methods, means the training accuracy reaches 100%.
See Fig.
2a for an example.
DNNs generalize much better on the red noise.
By comparing the width of Red and Blue belt in Fig.
2 under the same training condition (row-wise), we can see that the test accuracy’s standard deviation is considerably smaller on Red noise than on Blue noise.
This indicates a smaller difference in test accuracy between the clean and the noisy training data, suggesting that DNNs generalize better on Red noise.
For a clearer illustration, as an example, we plot networks trained from scratch on Mini-ImageNet in Fig. 3.
Speciﬁcally, Fig.
3a shows the training accuracy of the 0%, 40%, and 60% noise levels along with the training step.
Fig.
3b shows the difference in ﬁnal converged test accuracies, relative to the accuracy on the clean training data, under 10 noise levels.
As the training accuracy is perfect on all noise levels, the drop in the test accuracy can be regarded as an indicator of the generalization gap.
The blue curves in Fig 3b conﬁrm Zhang et al.
(2017)’s ﬁnding that DNNs generalize poorly as synthetic noise levels increase.
For example, the test accuracy of EfﬁcientNet will drop by 85% when Blue noise reaches 60%.
On the real-world noise, however, the gap is considerably smaller, e.g. the drop on 60% Red noise is only 23%.
This pattern holds for all architectures in our study.
See the curves for EfﬁcientNet, Inception-ResNet, and MobileNet in Fig.
3b.
Results in Fig.
2 and Fig.
3b suggest that DNNs generalize better on real-world noisy data.
This phenomenon is probably due to the two properties discussed in Section 3: (i) red noisy images are similar to clean training images, and hence bring less change to the training (ii) red noisy images are often sampled out of the training classes.
This may make them less confusing for the ﬁxed training classes.
DNNs may not learn patterns ﬁrst on the red noise.
The third column of Fig.
2 illustrates the relative drop between the peak and converged test accuracy, where the x -axis is the noise level and the y -axis computes the relative difference in percentage, i.e. (peak - converged)/peak accuracy.
We see that there is almost no drop on the clean data ( x = 0 ).
The drop starts to grow as the noise level increases.
This substantiates Arpit et al.
(2017)’s ﬁnding that DNNs learn patterns ﬁrst on noisy data.
Early stopping which terminates training at the peak accuracy is thus effective on Blue noise. [Figure]
[Figure] However, the drop on Red noise is signiﬁcantly smaller and even approaches zero on Stanford Cars.
See the red curves in the third column of Fig.
This suggests DNNs may not learn patterns ﬁrst on real-world noisy data, especially for the ﬁne-grained classiﬁcation task.
Our hypothesis is that Blue noise images are sampled uniformly from a ﬁxed number of classes, and the uniform errors can be mitigated in the DNN’s early training stage before it memorizes all noisy labels.
Real-world noisy images are sampled non-uniformly from an inﬁnite number of classes, making it difﬁcult for DNNs to identify meaningful patterns in the red noise.
ImageNet architectures generalize well on noisy data when the networks are ﬁne-tuned.
Comparing the ﬁrst and second rows in Fig.
2, we observe that the test accuracy for ﬁne-tuning is higher than that for training from scratch on both Red and Blue noise.
This is consistent with (Hendrycks et al., 2019) where they found pre-training improves model robustness on Blue noise.
Our results extend Hendrycks et al.
’s ﬁnding on Red noise.
Furthermore, the beneﬁt is more evident on the noisy data.
For example, on Stanford Cars, ﬁne-tuning improves training from scratch by less than 1% (91.2% vs. 90.5%) on the clean data.
At the 40% noise level, the difference grows to 28.9% (69.3% vs. 40.4%) on Blue noise and 4.8% (82.2% vs. 77.4%) on Red noise.
See Table 6 in the Appendix.
In Fig.
4, we compare the ﬁne-tuning performance using ImageNet architectures and compute the correlation coefﬁcient, where the y -axis is the peak accuracy and the x -axis is the top-1 accuracy of the architecture on ImageNet ILSVRC 2012 validation.
The bar plots the 95% conﬁdence interval across 10 noise levels, where the center dot marks the mean.
As it shows, there is a decent correlation between the ImageNet accuracy and the test accuracy on noisy data.
The Pearson correlation r is 0.897 on Mini-ImageNet and 0.875 on Stanford Cars.
The results suggest that, when networks are ﬁne-tuned, ImageNet architectures generalize well on noisy data.
This ﬁnding generalizes Kornblith et al.
(2019)’s recent ﬁnding to noisy data.
[Figure]
5. C OMPARISON OF THE ROBUST DEEP LEARNING METHODS
In Fig.
5, Fig. 6, Fig. 8, and Fig.
9, we compare the robust deep learning methods on Blue and
Red noise, where the x -axis shows the peak accuracy and its corresponding 95% conﬁdence interval over different hyperparameters.
We mainly compare the peak accuracy which, as shown in recent studies (Song et al., 2018; Arazo et al., 2019), is more challenging to improve.
We also list their converged accuracies in the Appendix for reference.
First of all, we observe big performance variances (or conﬁdence intervals) in most methods, suggesting that hyperparameters are important for robust learning methods.
The best hyperparameter usually varies for different noise levels, and our observation would be very different if the methods were not extensively tuned for each noise level.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table]
[Figure]
[Table] [Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Red noise is more difﬁcult for robust DNNs to improve.
Although robust DNNs are able to improve the performance of vanilla training across all noise levels and types, the improvement is noticeably smaller on Red noise.
For example, the average improvement of the best robust method
[Table]
[Figure]
Comparing methods, we ﬁnd that no single method performs the best across all noise levels and types.
Methods that work well on synthetic noise may not work as well on real-world noise, and vice versa.
To be speciﬁc, Dropout is effective for training from scratch on Stanford Cars and achieves the best accuracy in 20 trials.
Weight Decay mainly beneﬁts ﬁne-tuning but only to a small extent (6 best trials).
Reed achieves the best result in 10 trials, all of which are ﬁne-tuning on MiniImageNet.
S-Model yields marginal gains over the vanilla training.
Finally, MentorNet and Mixup achieve the best accuracy in 21 and 23 trials, respectively.
Unlike MentorNet, Mixup seems to be more effective on Red noise, suggesting that pair-wise image mixing is more effective than example weighting on real-world noise.
5. A UGMENTING D ATASET WITH N OISY E XAMPLES
[Figure]
In practice, a simple technique to improve performance is to add noisy or weakly-labeled examples to an existing clean dataset.
It is convenient because noisy examples can be automatically collected without any manual labeling effort.
Multiple studies have shown that noisy examples can be beneﬁcial for training e.g. (Krause et al., 2016; Liang et al., 2016; Mahajan et al., 2018).
In particular,
Guo et al.
(2018) found that adding WebVision noisy images to clean ImageNet can lead to a performance gain.
On the other hand, it may be disadvantageous if the true labels of all added examples are incorrect.
An interesting question that has yet been answered is what is the maximum noise level at which the added noisy data can be useful? .
Our benchmark allows for investigating this question using real-world noisy data in a controlled setting.
To do so, we add ∼ 30K and ∼ 25K additional images to the original training sets of MiniImageNet and Stanford Cars, respectively.
The sizes are selected to be 60% and 300% of the original datasets to examine different settings.
We control the noise level of the added images in 10 different levels from 0% to 80%.
Fig.
7 shows the peak test accuracy (the y -axis) across ten noise levels (the x -axis).
The dashed line represents the best accuracy obtained on the original dataset.
As it shows, the test accuracy generally decreases as the noise level grows.
Small noise is useful but large noise can hurt the performance.
The equilibrium occurs between 30% to 50%.
In all cases, it is useful if the noise level is under 30%.
When the noise level is below 30%, it also improves the full augmented datasets in Table 1, the accuracies of which are 0.770 (trained from scratch) and 0.
(ﬁne-tuned) on Mini-ImageNet, and 0.927 and 0.932 on Stanford Cars.
Note that the 30% threshold just represents the observation on our benchmark and should not be overinterpreted.
C ONCLUSION
In this paper, we established a benchmark for controlled real-world noise.
On the benchmark, we conducted a large-scale study to understand deep learning on noisy data across a variety of settings.
Our studies revealed a number of new ﬁndings, improving our understanding of deep learning on noisy data.
By comparing six robust deep learning methods, we found that real-world noise is more difﬁcult to improve and methods that work well on synthetic noise may not work as well on realworld noise, and vice versa.
This encourages future research to be also carried out on controlled real-world noise.
We hope our benchmark, as well as our ﬁndings, will facilitate deep learning research on real-world noisy data.
[Figure] [Equation]
[Figure]
mentornet
[Figure]
mentornet
[Figure]
mentornet
[Figure]
mentornet
[Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Figure]
A A PPENDIX
[Figure]
A. I MPLEMENTATION D ETAILS
This subsection presents the implementation details.
Architectures: Table 2 lists the parameter count and input image size for each network architecture examined.
We obtained their model checkpoints trained on the ImageNet 2012 dataset from
TensorFlow Slim 6 , EfﬁcienNet TPU 7 , and from the authors of (Kornblith et al., 2019).
The last two columns list the top-1 accuracy of our obtained models along with the accuracy reported in the original paper.
As shown, the top-1 accuracy of these architectures on the ImageNet ILSVRCvalidation ranges from 71.6% to 83.6%, making them suitable for our correlation study.
[Table]
Training from scratch (random initialization) : for vanilla training, we trained each architecture on the clean dataset (0% noise level) to ﬁnd the optimal training setting by grid search.
Our grid search consisted of 6 start learning rates of { 1 . 6 , 0 . 16 , 1 . 0 , 0 . 5 , 0 . 1 , 0 . 01 } and 3 learning rate decay epochs of { 1 , 2 , 3 } .
The exponential learning rate decay factor was ﬁxed to 0.975.
We trained each network to full convergence to ensure its training accuracy reached 1.0.
The maximum epoch to train was 200 on Mini-ImageNet (Red and Blue) and 300 epochs on Stanford Cars (Blue and Red), where the learning rate warmup (Goyal et al., 2017) was used in the ﬁrst 5 epochs.
The training was using
Nesterov momentum with a momentum parameter of 0.9 at a batch size of 64, taking an exponential moving average of the weights with a decay factor of 0.9999.
We had to reduce the batch size to 8 for EfﬁcientNet for its larger image input.
Following Kornblith et al.
(2019), our vanilla training was with batch normalization layers but without label smoothing, dropout, or auxiliary heads.
We employed the standard prepossessing in EfﬁcientNet 8 for data augmentation and evaluated on the central cropped images on the test set.
Training in this way, we obtained reasonable performance on the clean Stanford Cars test set.
For example, our Inception-ResNet-V2 got 90.8 (without dropout) and 92.4 (with dropout) versus 89.9 reported in (Kornblith et al., 2019).
Fine-tuning from ImageNet checkpoint: for ﬁne-tuning experiments, we initialized networks with
ImageNet-pretrained weights.
We used a similar training protocol for ﬁne-tuning as training from scratch.
The start learning rate was stable in ﬁne-tuning so we ﬁxed it to 0.01 and only searched the learning rate decay epochs in { 1 , 3 , 5 , 10 } .
Learning rate warmup was not used in ﬁne-tuning.
As ﬁne-tuning converges faster, we scaled down the maximum number of epoch to train by a factor of 2.
In this case, the training accuracy was still able to reach 1.0.
Training in this way, we obtained reasonable performance on the clean Stanford Cars test set.
For example, our Inception-ResNet-V got 92.4 versus 92.0 reported in (Kornblith et al., 2019) and our EfﬁcientNet-B5 got 93.8% versus 93.6% reported in (Tan & Le, 2019).
Robust deep learning method comparison: For method comparison, we used Inception-ResNet as the default network.
We ﬁxed the optimal setting found on the clean training set for all methods and all noise levels.
We found the hyperparameter for robust DNNs is important.
Therefore, we extensively searched the hyperparameter of each method for every noise level and every noise type, using the range discussed in the main manuscript.
Comparing 6 methods using all hyperparameters, 10 noise levels, 2 noise types, and 2 training conditions led to a total of 1,840 experiments on two datasets.
The mean and 95% conﬁdence interval over different hyperparameters were shown in
Fig.
5, Fig. 6, Fig. 8, and Fig.
9 in the main manuscript.
The best peak accuracy (along with the converged accuracy) could be found in Table 4 to Table 7.
For Dropout, as it converges slower, we added another 100 epochs to its maximum epochs to train.
A. C OMPARISON TO THE STATE - OF - THE - ART ON CIFAR-
This subsection shows that our examined robust learning methods are able to achieve the performance that is comparable to, or even better than, the state-of-the-art on the commonly used synthetic noisy dataset in the literature.
Following (Jiang et al., 2018; Shu et al., 2019; Arazo et al., 2019), the noisy CIFAR-100 data are of uniform label-ﬂipping noise, where the label of each image is independently changed to a uniform (incorrect) class with probability p , where p is the noise level and is set to 20%, 40%, 60%, and 80%.
The clean test images on CIFAR-100 are used for evaluation.
Table 3 shows the results where † marks our implementation of MentorNet (Jiang et al., 2018) 9 and
Mixup (Zhang et al., 2018) 10 under the best hyperparameter setting 11 .
First, the comparison between
Row 2 and Row 7 in the table shows extensive hyperparameter search leads to about a 2% gain over the published results in (Jiang et al., 2018).
Second, comparing Row 5 and 6 to Row 4, it shows that our examined methods are comparable to the state-of-the-art except for the 40% noise level.
Finally, comparing Row 7 and 8 to others, we ﬁnd that our examined methods are able to achieve the best result on this dataset.
[Table]
j agua r l ad y bug o r ange
Noisy Training Examples in the Webvision Dataset
A. T EXT - TO - IMAGE SEARCH ONLY RESULTS
In this section, we exclude the images brought by image-to-image search and reconduct our main experiments.
This data subset contains only the images from Google text-to-image search and hence is similar to the noisy datasets used in previous studies (Li et al., 2017a;
Krause et al., 2016; Chen & Gupta, 2015) (note that existing datasets do not have controlled noise). Our goal is to verify whether our ﬁndings still hold on this new data subset. We use dark red to denote the text-to-image search only noise and compare it with the red noise reported in the paper.
To be speciﬁc, the vanilla training and test curves are compared in Fig.
The generalization errors are compared in Fig. The ﬁnetuning models on different ImageNet architectures are compared in Fig. As we see in all cases, the text-to-image only noise (dark red) performs very similarly to the red noise studied in the paper. We can conﬁrm the our ﬁndings still hold.
That is (1) DNNs generalize much better on the dark red noise (Fig. 12 and Fig. 13).
(2) DNNs may not learn patterns ﬁrst on the dark red noise (Fig. 12). (3) ImageNet architectures generalize well on noisy data when the networks are ﬁne-tuned (Fig. 14). We did not rerun the method comparison given the limited time. The above results show that the red noise studied in our paper is consistent with the text-to-image only noise.
As red noise contains more diverse label noises, we keep it as our main datasets. [Figure] [Figure] [Figure]
A. N UMERICAL R ESULTS IN M ETHOD C OMPARISON
This subsection presents the numerical results for the best trial in Fig.
5, Fig. 6, Fig. [Table] and Fig.
9 in the main manuscript.
[Table] [Table]
R e d
N o i s e 68.9/67.
[Table] [Table]
[Table] [Table]