L EVERAGING INDUCTIVE BIAS OF NEURAL NETWORKS FOR LEARNING WITHOUT EXPLICIT HUMAN ANNOTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Image classiﬁcation problems today are often solved by ﬁrst collecting examples along with candidate labels, second obtaining clean labels from workers, and third training a large, overparameterized deep neural network on the clean examples.
The second, manual labeling step is often the most expensive one as it requires manually going through all examples.
In this paper we propose to i) skip the manual labeling step entirely, ii) directly train the deep neural network on the noisy candidate labels, and iii) early stop the training to avoid overﬁtting.
With this procedure we exploit an intriguing property of overparameterized neural networks: While they are capable of perfectly ﬁtting the noisy data, gradient descent ﬁts clean labels faster than noisy ones. Thus, training and early stopping on noisy labels resembles training on clean labels only.
Our results show that early stopping the training of standard deep networks (such as ResNet-18) on a subset of the Tiny Images dataset (which is obtained without any explicit human labels and only about half of the labels are correct), gives a signiﬁcantly higher test performance than when trained on the clean CIFAR-10 training dataset (which is obtained by labeling a subset of the Tiny Images dataset).
In addition, our results show that the noise generated through the label collection process is not nearly as adversarial for learning as the noise generated by randomly ﬂipping labels, which is the noise most prevalent in works demonstrating noise robustness of neural networks.
I NTRODUCTION
Much of the recent success in building machine learning systems for image classiﬁcation can be attributed to training deep neural networks on large, humanly annotated datasets, such as ImageNet (Deng et al., 2009) or CIFAR-10 (Torralba et al., 2008).
However, assembling such datasets is time-consuming and expensive: Both ImageNet and CIFAR-10 were constructed by ﬁrst searching the web for candidate images and second labeling the candidate images by workers to obtain clean labels instead of the noisy candidate ones.
The ﬁrst step already yields labeled examples, but the accuracy of those automatically collected, candidate labels is low: For example, only about half of the labels of the Tiny Images dataset constructed in the ﬁrst step of obtaining the CIFAR-10 dataset are correct (Torralba et al., 2008).
In the second step, clean labels are obtained by asking workers through a crowdsourcing platform for annotations and aggregating them.
In this paper we propose to train directly on the noisy candidate examples, effectively skipping the expensive human labeling step.
To make this work, we exploit an intriguing property of large, overparameterized neural networks: If trained with stochastic gradient descent or variants thereof, neural networks ﬁt clean labels signiﬁcantly faster than noisy ones.
This fact is well known, see for example the experiments by Zhang et al.
(2017, Fig.
1a) which demonstrated that overparameterized deep networks can ﬁt even randomly shufﬂed CIFAR-10 labels perfectly.
What is now well known is that this effect is sufﬁciently strong to enable training on candidate labels only.
Our idea is that, if neural networks ﬁt clean labels faster than noise, then training them on a set containing clean and wrong labels and early stopping the training resembles training on the clean labels only.
1. O UR CONTRIBUTIONS
We show that early stopping the training on candidate examples can achieve classiﬁcation performance higher than when training on clean labels, provided the total number of clean labels in the candidate training set is sufﬁciently large (throughout this paper, we refer to the dataset that consists of the examples with the associated, noisy candidate labels—as obtained in the ﬁrst step of the dataset construction process—as the candidate dataset/candidate training set).
This result questions the expensive practice of building clean humanly labeled training sets, and suggests that it can be better to collect larger, noisier datasets instead.
Speciﬁcally, we consider the CIFAR-10 classiﬁcation problem, where the goal is to classify 32x color images in 10 classes.
The clean training set consists of 5000 images per class and was obtained by labeling the images from the Tiny Images dataset with expert labelers.
Instead of training on the original, clean training set, we train on a new noisy training set consisting of candidate examples only. We constructed this candidate training set by picking the images from the Tiny Images dataset with the labels of the CIFAR-10 classes.
Only about half of the examples in this candidate dataset are correctly labeled.
We quantify performance on the new CIFAR-10.1 test set from Recht et al. We ensured by automatically and manually cleaning the training set that it contains no images that are similar to any of the images in the CIFAR-10.1 test set.
We trained the best performing networks for CIFAR-10, (ResNet (He et al., 2016), ShakeShake (Gastaldi, 2017), VGG (Simonyan & Zisserman, 2015), DenseNet (Huang et al., 2017) and others (Han et al., 2017; Liu et al., 2018)), and found that with early stopping the training we achieve signiﬁcantly higher test accuracy than when training on the original, clean examples.
That is possible, because even though the candidate dataset is very noisy, it contains more clean examples than the original CIFAR-10 training set, as additional experiments show.
More speciﬁcally, the best performing model trained on the original CIFAR-10 training set has 7% classiﬁcation error on the CIFAR-10.1 test set (Recht et al., 2019).
The best performing model trained on our candidate training set, in contrast, achieves the better error rate of only 5.85%, with early stopping training of the PyramidNet-110 model.
This is a lower (better) test accuracy (as measured on the CIFAR-10.1 dataset) than that achieved by any model trained on the clean CIFAR10 training set.
What is more, all models we tested have a signiﬁcant gap of training on the original clean and our noisy data. For example, the performance of ResNet-18 jumps from 85.2% to 92.15% when training and early stopping on the noisy dataset instead of the clean one.
Early stopping is critical to achieve the best performance.
By keeping track of the performance of a clean subset of the noisy training set, we show that the clean labels are ﬁtted signiﬁcantly faster than the noisy labels, and a good point to stop is when most of the clean labels are ﬁtted well by the network, because at this point most of the wrong labels have not been ﬁtted yet.
We also study the difference between the “real” noise in the data obtained through the actual data collection progress (from sources such as search engines, like Tiny Images) and artiﬁcially generated noise through randomly ﬂipping the labels of a clean dataset, which is often used in the literature as a proxy for the former.
We show that the “real” noise is more structured and therefore it is easier to ﬁt and less harmful to the classiﬁcation performance in contrast to artiﬁcially generated noise, which is harder to ﬁt and potentially much more adversarial to performance. Finally, we show that our results continue to hold for other datasets.
Speciﬁcally, we consider classifying invertebrate animals from training on i) a candidate dataset we collected from Flicker only through keyword search es and without subsequent labeling and from ii) a subset of the ImageNet.
While for this experiment training on the candidate dataset does not improve over training on the clean set, early stopped gradient descent achieves performance close to that obtained when training on the clean set.
1. R ELATED WORK
While our work focuses on exploiting the fact that large neural networks ﬁt clean labels faster than noise, a large number of related works have shown that neural networks are robust to label noise, both in theory and practice (Rolnick et al., 2017), and have proposed methods to make deep networks even more robust to label noise (Sukhbaatar & Fergus, 2014; Jindal et al., 2016).
Several other papers have suggested loss function adjustments and re-weighting techniques for noise robust training (Zhang &
Sabuncu, 2018; Ren et al., 2018; Tanaka et al., 2018).
Guan et al.
(2018) has demonstrated that the classiﬁcation performance of deep networks remains almost constant when training on partly randomly perturbed MNIST (handwritten digits) training examples.
A number of recent works have offered explanations for why deep neural networks ﬁt structure in data before ﬁtting noise or other complex patterns: Arpit et al.
(2017) has shown that loss sensitivity of clean examples is different to noisy examples, Ma et al.
(2018) demonstrated that DNNs model low-dimensional subspaces that match the underlying data distribution during the early stages of training but needs to increase the dimensionality of the subspaces to ﬁt the noise later in training, and ﬁnally Li et al.
Arora et al.
(2019) have attributed this in the over-parameterized case with the low-rank structure of the Jacobian at initialization.
There are few works that have exploited the fact that clean examples are ﬁtted faster than noise, with three noteable exceptions being the paper by Shen & Sanghavi (2019) which achieved noise-robustness by proposing a iterative retraining scheme, Sun et al.
(2018) which proposed systematic early stopping, and Song et al.
(2018) which trains networks with large learning rates and uses the resulting loss to separate clean from misslabeled examples.
Li et al.
(2019) provided the perhaps ﬁrst theoretical justiﬁcation for early stopping to achieve robustness, by showing that gradient descent with early stopping is provably robust to label noise for overparameterized networks.
Next we note that, we have used the Tiny Images dataset as a source for obtaining (noisy) candidate examples; other works have used it as a source for augmenting the clean CIFAR-10 and CIFARdatasets (Carmon et al., 2019; Laine & Aila, 2016) for a semi-supervised learning setting.
Finally we note that most of the works on learning from noisy labels have used artiﬁcially generated noise on clean, baseline datasets for motivation, experimentation and as performance metrics.
As discussed later in this paper, we ﬁnd that the performance degradation resulting from overﬁtting the noisy data is signiﬁcantly worse on such artiﬁcially generated noise, whereas overﬁtting to the true label noise is not nearly as detrimental.
The reason is that the true label noise found in the Tiny
Images dataset is highly structured and not adversarial.
D ATASET COLLECTION AND PROBLEM SETUP
We construct a candidate training dataset for the CIFAR-10 classiﬁcation problem, without any explicit human labeling effort.
Our noisy training set is a subset of the 80 million Tiny Images dataset (Torralba et al., 2008).
The Tiny Images dataset consists of 32x32 images obtained by searching for nearly 80 , 000 query keywords or labels in the image search engines Google, Flickr, and others.
This yields examples of the form (keyword, image), but the associated labels (keywords) are very noisy, speciﬁcally only about 44% are correct (Torralba et al., 2008, Sec.
III-B), and often an example image is completely unrelated to the label (keyword).
The goal of the CIFAR-10 classiﬁcation problem is to classify 32x32 color images into the ten classes { airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck } .
The original, clean, training set for the CIFAR-10 dataset was obtained by ﬁrst extracting candidates from the noisy examples of the Tiny Images dataset for each of the ten classes based on the ﬁnely grained tinyimages keywords (for example, keywords for the class dog are: puppy, pekingese, chihuahua, etc., see (Recht et al., 2019, Table 6) for a complete list), and then labeling all of the so-obtained images by human experts (Krizhevsky, 2009).
This gives a clean training set consisting of 5000 examples per class.
We construct a new and larger (by a factor of about 10) noisy training set by simply taking the subset of the Tiny Images for the respective class according to the same ﬁnely grained Tiny Images keywords, without any further human labeling.
This resembles a data collection process without any explicit human labeling or cleaning of the data.
In our experiments throughout this paper, test performance is measured on the CIFAR-10.1 test set from Recht et al.
To ensure that our noisy training set does not overlap with the CIFAR10.1 test set in any way, we carefully removed all test images as well as images that are similar to test images from the noisy training set.
Both is necessary because the extracted subset of the Tiny Images dataset contains many images that are similar to test images up to slight differences in color scale, contrast, original resolution, croppings, and watermarks.
See Appendix A on details and [Figure] examples of this non-trivial data cleaning process.
We work with the CIFAR-10.1 test set instead of the original CIFAR-10 test set because it contains fewer images (2000 instead of 10000), and thus made our manual data cleaning process faster.
Both datasets share the same distributional properties, and Recht et al.
(2019, Figure 1) found the test performance on both sets to be linearly related.
After removing the similar images, our noisy training set consists of the number of examples indicated in the table below.
About half of the labels in the noisy training set are wrong, see Figurefor an illustration.
Also note that they are often completely unrelated and out of distribution—for example a screenshot, a picture of a person, and a wheel are all assigned the label “dog”.
[Figure]
T RAINING ON A LARGE NOISY DATASET CAN BE BETTER THAN TRAINING ON A CLEAN ONE
In this section, we show that the performance of a deep network trained on clean examples can be matched or even bested if the same network is trained with early stopping on noisy candidate examples—despite the many wrong labels—as long as there are sufﬁciently many clean labels in the noisy dataset.
Our results show that this is due to neural networks ﬁtting clean labels signiﬁcantly faster than noise; speciﬁcally we observe that a subset of our training set that has clean labels is ﬁtted during the early stages of training and much faster than the overall training set.
Thus, when stopped early, the network is at a state similar to that obtained when only trained on the clean labels in the ﬁrst place.
This effect enables achieving classiﬁcation performance by training on very noisy, potentially automatically collected candidate datasets that is on par with the performance obtained by training on very clean, expert annotated datasets.
Figure 2 depicts the training and test accuracy of training with stochactic gradient descent on our candidate dataset as well as on the clean CIFAR-10 training dataset for ResNet-18 with default parameters, taken from the Pytorch GitHub repository (PyTorch).
The results show that when trained on the noisy dataset, the network achieves signiﬁcantly better test performance than when trained on the clean dataset.
Moreover, the best performance is achieved when the network is early stopped at about 30 epochs.
At this optimal early stopping point, essentially all the clean examples but only very few of the noisy examples are ﬁtted.
This follows by noting that at about 30 epochs, the network achieves almost 100% training accuracy on a clean subset of the noisy dataset, but only about 65% overall test accuracy at a false-label-rate of about 50%.
As test set we used CIFAR-10.1, and the clean subset of the training examples is the original CIFAR-10 test set, which is by construction a
[Figure] subset of the candidate training set. In practice, the optimal stopping time can be obtained in a data driven way by monitoring the test performance on a small, clean subset of the data, and stopping once close to 100% training accuracy is achieved on the clean subset of the data.
ResNet is sufﬁciently overparameterized to perfectly ﬁt the training data, even the mislabeled examples.
However, even a network with fewer parameters, such as the DenseNet-110 (k=12) ﬁts the clean data before the network reaches its capacity to ﬁt the noisy labels and the training accuracy plateaus at about 80%, see Figure 2.
We observe this effect across a large number of standard and state-of-the-art models (see Appendix B).
In Table 1, we provide the main results for the models we have tested on both the original clean CIFAR-10 training set and our candidate training set.
The reported accuracies correspond to the best ones achieved during training for each setup.
We note that there is a signiﬁcant improvement in test performance across all models.
We also observe that the number of parameters does not seem to be directly related to the performance increase of the models.
[Table]
Clean
[Table]
Noisy
[Table]
[Table]
[Figure]
H OW CLEAN DO THE LABELS HAVE TO BE ?
In the previous section we found that learning on the noisy data gives signiﬁcantly better performance than training on the clean data.
Next we demonstrate that this ﬁnding relies on the total number of clean labels that are available—if that number grows sufﬁciently fast with the noise level, then performance becomes increasingly better when training on the noisy training set.
We perform the following experiment to measure performance on increasingly larger and noisier datasets.
We start with the original CIFAR-10 training set and gradually replace a fraction of it with r = 1 , 2 , 3 , 5 , 10 times more images from the Tiny Images dataset.
For example, if the fraction is 0 . 5 and r = 2 , that means that the new, noisy dataset consists of 25.000 images of the original clean CIFAR-10 training set and of 50.000 noisy images from our noisy training set.
Increasing the fraction for some ﬁxed value of r > 1 thus increase both the dataset size as well as the noise level.
We did this experiment both in the regime of very few examples (500 per class) and many examples (5000 per class).
Figure 3 depicts the results.
For r = 1 , the total dataset size remains constant but becomes increasingly noisier.
Thus the effective number of clean examples in the noisy dataset becomes smaller as the fraction increases.
Non-surprisingly, we observe that the test accuracy drops as the noise increases.
The curve for r = 5 is more interesting: If we substitute one clean label with 5 noisy ones, then the test performance increases even though the noise level increases, because the total number of clean examples also grows sufﬁciently fast.
T RUE VERSUS SYNTHETIC LABEL NOISE
As mentioned in the introduction, a large number of papers has shown that neural networks ﬁt randomly corrupted labels slower than clean ones and/or that neural network are robust to label noise (Zhang et al., 2017; Rolnick et al., 2017;
Guan et al., 2018; Ma et al., 2018; Li et al., 2019).
The experiments in those works are exclusively for noisy labels generated by randomly ﬂipping the clean labels, which can be considered as in-distribution noise.
As we see from Figure 1, the ‘true’ label noise generated through the image collection process is very different in that it contains lots of out-of-distribution images as well as structured noise.
For example, the keyword ‘Mouser’ used to query images for the cat class by CIFAR-10 yields mostly toy guns and listing images of an electronics distribution company and only a few cat related images.
Similarly, the keyword ‘Tomcat’ surprisingly yields only few cat related images as well.
Most of the images are related to Java server implementations, the F-14 jet nicknamed Tomcat, and a species of insects.
Such structured, “real” noise is easier to ﬁt and less harmful to classiﬁcation performance.
To demonstrate this, we take the original CIFAR-10 training set and perturb 45% of the labels by uni-
[Figure]
trainingtest
[Figure]
epoch
CIFAR-10 shufﬂed labels formly ﬂipping them.
We then train on the so-obtained artiﬁcially perturbed dataset and a subset of our noisy dataset consisting of the same number of examples (5000 per class).
This process ensures that the label error probability is approximately the same, but the distribution of the errors is very different.
The results in Figure 4 show that the data from our noisy dataset is ﬁtted signiﬁcantly faster because the noise is more structured.
Moreover, the results show that early stopping is important for both setups, but training until overﬁtting is not nearly as detrimental to the test performance for training on the candidate dataset than on the noisy one.  E XTENSIONS TO A SUB - SET OF I MAGE N ET The ﬁnding that it is possible to build a well-performing classiﬁer by training and early stopping on a noisy candidate dataset is not unique to the CIFAR-10 classiﬁcation problem. In order to demonstrate this, we train a classiﬁer for a subset of the popular ImageNet Deng et al.
(2009) dataset.
We randomly choose 15 classes out of the 61 classes belonging to the family class of invertebrates (any animal lacking a backbone).
In order to construct a noisy candidate set, we use the synsets of these classes as search keywords to obtain candidate images from Flickr.
In order to avoid any overlaps between the original ImageNet dataset and the candidate dataset, we restrict the search date to a 5-year period, from 2014-07-12 to 2019-07-12.
We also search in 4-month intervals at a time and restrict the maximum number of images from each interval in order to prevent any bias that might be a result of mass uploads from institutional and professional accounts.
We do not ﬁlter or further label any of the images obtained from the Flickr searches, only resize them to 256x256, which is the size of the images in the ImageNet.
The resulting candidate dataset consists of around 173k images across 15 classes.
Note that this is an extremly noisy dataset, the average accuracy of image search results from the internet is considered to be around 10% Torralba et al.
In Figure 5, we show the training and test accuracy of training with stochastic gradient descent on our candidate dataset obtained from Flickr as well as the subset of ImageNet training set corresponding to the chosen 15 classes for the ResNet-18 model with default parameters.
The results show that when trained on the candidate dataset, the model achieves a comparable test performance to when trained on the clean dataset.
Moreover, there is a clear early stopping point (about 30 epochs) during the training on the candidate dataset where the best test performance is achieved.
Our manual inspections of the candidate dataset reveal that the dataset is very noisy with falsely labeled images accounting to much more than half of all the images for each class.
Moreover, a large part of the falsely labeled images comes from uploads from few, but highly active accounts.
This also hinders the independence of the noise distribution across different classes as there are few prominent sources of the noise, which in turn makes the noise more adversarial.
This partly explains the high variance and lower test performance during the early steps of training on the candidate dataset.
We note that in practice using more speciﬁc search keywords instead of the ImageNet synsets can yield much better results.
[Figure]
R EGULARIZATION WITH EARLY STOPPING : A PPEALING TO LINEAR MODELS
For understanding why highly overparameterized neural networks ﬁt structure faster than noise, it is helpful to appeal to linear models.
Speciﬁcally, a number of recent works show that highly overparameterized models behave like an associated linear model around the initialization, which in turn offers an explanation why neural networks ﬁt structure faster than no structure (Li et al., 2019;
Du et al., 2019; 2018; Arora et al., 2019).
While the regime where those results hold is extremely overparameterized and thus might not be the regime in which those models actually operate (Chizat et al., 2018), it can provide some intuition on the dynamics of ﬁtting clean versus noisy data.
In this section, we ﬁrst brieﬂy comment on those recent theoretical results and explain how they provide intuition on why deep networks ﬁt structured data faster than noise, and second, inspired by those explanations for linear models, we demonstrate that early stopping the training of non-deep overparameterized linear models enables learning from noisy examples as well.
7. E XISTING THEORY FOR OVERPARAMETERIZED MODELS
Consider a binary classiﬁcation problem, and let f W : R d → R be a neural network with weights
W ∈ R N , mapping a d -dimensional features space to a label.
The output label associated with example x is interpreted as belonging to class 1 if f W ( x ) is positive and as belonging to class − 1 if f W ( x ) is negative.
Let { ( y 1 , x 1 ) , . . .
, ( y n , x n ) } be a set of n many training examples.
The network is trained by minimizing the empirical loss
[Equation]
with gradient descent, where y =
[ y 1 , . . .
, y n ] are the labels of the examples and f W ( X ) =
[ f W ( x 1 ) , . . .
, f W ( x n )] are the predictions of the model.
Let J ( W ) ∈
R n × N be the Jacobian of the mapping from weights to outputs f : R N → R n , and let J ∈ R n × N be a speciﬁc matrix that obeys
[Equation]
, where expectation is with respect to the random initialization W 0 .
For two-layer networks, a number of very recent works (Li et al., 2019; Du et al., 2019; 2018; Arora et al., 2019) have shown that gradient descent in close proximity around the initialization is well approximated by gradient descent applied to the associated linear loss
[Equation]
From this, it follows that (see (Li et al., 2019; Arora et al., 2019)) that the prediction error after t iteration of gradient descent with stepsize η (provided that ηt is not too large) is given by
[Equation]
[Figure]
iteration clean dataset where v i are the eigenvectors and σ 2 i are the eigenvalues of the matrix JJ T , and η ≤ 1 /σ 2 max .
Equation (1) shows that the directions associated with large eigenvectors are ﬁtted signiﬁcantly faster by gradient descent than directions associated with small eigenvalues.
Thus labels vectors y that are well aligned with eigenvectors corresponding to large eigenvalues are ﬁtted signiﬁcantly faster than those corresponding to small eigenvalues.
The eigenvalues and eigenvectors are determined by the structure of the network.
It turns out that for neural network the structure is such that the projection of correctly labeled data (i.e., (cid:104) v i , y (cid:105) , for y a set of correct labels) is much more aligned with directions associated with large eigenvalues of JJ T , than the projection of falsely labeled data (i.e., (cid:104) v i , y (cid:105) , for y a set of falsely labeled examples).
This suggests that for highly linear overparameterized models we expect to see similar dynamics in that for certain model structure is ﬁtted faster than noise—a property that we demonstrate in the next section.
7. N ON - DEEP MODELS
Finally, we demonstrate that even with non-deep overparameterized models, clean labels are learned faster than noisy ones, and thus early stopping enables learning from (noisy) candidate examples.
We consider a random feature model ﬁtted with gradient descent and early stopped.
The reason for considering a random feature model is that such models can reach performances that are reasonably close to that of a neural network if the number of random features, and thus the size of the model, is sufﬁciently large.
We consider a random-feature model as proposed in Coates et al.
We ﬁrst extract for each example random features through a one-layer convolutional network G with 4000 random ﬁlters, each of size 6 × 6 .
For each image x , this gives a feature vector G ( x ) of size m = 72000 .
Consider a training set with N examples, and let X ∈ R N × m be the feature matrix associated with the training set, and let Y ∈ R N × K be the matrix containing the labels.
Speciﬁcally, Y has a one in the k -th position of the i -th row if the i -th training example belongs to the k -th class, k ∈ { 1 , . . .
, K } .
In the training phase, we ﬁt a linear model by minimizing the least-squares loss
[Equation]
with gradient descent.
Here, the matrix Z ∈ R m × K speciﬁes the model, along with the random featurizer (i.e., the random convolutional model).
Given a new example x , the model then assigns the label arg max k G ( x ) Z .
We trained the model above on two datasets: a clean dataset consisting of 500 examples for each class of the CIFAR-10 training set, and a candidate dataset consisting of 1500 examples for each class from our candidate dataset.
We measure classiﬁcation performance as done throughout on the
CIFAR-10.1 test set.
The results, shown in Figure 6, show that on the clean dataset, this model achieves essentially 100% training accuracy.
On the noisy dataset, similarly as for the deep networks, early stopping improves performance in that the best performance is achieved at about 70, iterations of gradient descent, long before the best ﬁt of the model is achieved.
A D ETAILS ON THE DATA - CLEANING PROCESS
In order to make sure that the CIFAR-10.1 test set has no overlap with our noisy dataset, we constructed our noisy dataset by ﬁrst extracting all images with the respective keywords from the Tiny
Images dataset, and then removing all images that are similar to the images in the CIFAR-10.1 test set as follows:
We ﬁrst identiﬁed 100 closest images for each individual image in the CIFAR-10.
test set in (cid:96) 2 -distance, and then removed equivalent and similar images after manual visual inspection of these candidates.
We consider two images similar, if they share the whole or a considerable portion of the same underlying main object, secondary object, or a speciﬁc background (see Figure 7 for examples).
Performing this step manually is necessary because even images that are relatively far in (cid:96) 2 -distance are often extremely similar, while images relatively close can be signiﬁcantly different (see Figurefor examples).
We also note that it is too pessimistic to cutoff the images by using a threshold together with a closeness measure because of this wide variety of differences across images that actually contain the same object.
Figure 8 demonstrates an instance of this problem for an image in the ‘bird’ class.
There are many other such instances with highly different distance values, which necessitates manual inspection for this part.
In the following ﬁgures (9–11), we showcase some instances that further demonstrate the need for expert labeling for this similar image identiﬁcation task.
We note that in addition to the provided similar images, all the test images that are shown here also have identical copies in our training set that can be found without any need for manual labor.
This is expected because the CIFAR-10.1 test set is compiled mostly from Tiny Images and it holds true for most of the images in this set.
[Equation]
images that are considered similar for the following reasons: a) Same primary object with different contrasts; b) Different primary objects in the same background c)
Same secondary object with different primary objects d) Same object in different poses.
[Figure]
[Figure] [Figure]
[Figure]
B L EARNING CURVES FOR DEEP LEARNING MODELS
In this section, we display additional learning curves for the models omitted in Section 3 of our main document.
[Figure]
trainingtest
[Figure]
C E RROR ANALYSIS
In this section we take a closer look at the accuracy obtained by training on the candidate dataset.
Figure 13 shows the accuracy per class for training ResNet-18 on the original CIFAR-10 training set and training on the candidate images with early stopping.
The plots show that the model trained on the candidate examples has better performance in each class than the model trained on the original training set.
[Figure]