Workshop version: How hard are computer vision datasets? Calibrating dataset difficulty to viewing time
Abstract
Humans outperform object recognizers despite the fact that models perform well on current datasets.
Numerous efforts exist to make more challenging datasets by scaling up on the web, exploring distribution shift, or adding controls for biases.
The difficulty of each image in each dataset is not independently evaluated, nor is the concept of dataset difficulty as a whole currently well defined.
We develop a new dataset difficulty metric based on how long humans must view an image in order to classify a target object.
Images whose objects can be recognized in 17ms are considered to be easier than those which require seconds of viewing time.
Using 133,588 judgments on two major datasets, ImageNet and ObjectNet, we determine the distribution of image difficulties in those datasets, which we find varies wildly, but significantly undersamples hard images.
Rather than hoping that distribution shift will lead to hard datasets, we should explicitly measure their difficulty.
Analyzing model performance guided by image difficulty reveals that models tend to have lower performance and a larger generalization gap on harder images.
We release a dataset of difficulty judgments as a complementary metric to raw performance and other behavioral/neural metrics.
Such experiments with humans allow us to create a metric for progress in object recognition datasets.
This metric can be used to both test the biological validity of models in a novel way, and develop tools to fill out the missing class of hard examples as datasets are being gathered.
Introduction
Numerous efforts exist to build better evaluations for object recognizers.
Broadly, these fall into four categories.
Those that probe distribution shift, like ImageNetV2 [1].
Those that add scale like OpenImages [2].
Those that explicitly attempt to make images more difficult for models by adversarially selecting them, like ImageNet-A [3] or adding artificial corruptions, like ImageNet-C [4].
And those that attempt to explicitly control for biases like ObjectNet [5].
These are responses to the fact that performance on standard benchmarks does not translate well to real-world conditions; 90% accuracy for one class in ImageNet does not mean 90% accuracy in the wild.
In all four cases, these efforts have no objective guide, no metric that evaluates their progress towards generalization.
We measure an orthogonal quantity – how difficult images in these datasets are for humans.
Distribution shift and bias control won’t on their own address this problem if datasets are easy compared to what humans are capable of recognizing.
And while scale helps, if datasets are heavily skewed [Figure] toward images that are easy for humans, the statistics of performance on such datasets may hide the real underlying performance trends of models on harder images.
Our contributions are: (1) a dataset of 133,588 human object recognition judgments as a function of viewing time for 4,771 images from ImageNet and ObjectNet, (2) the distribution of image difficulties for ImageNet and ObjectNet relative to what humans can recognize, shown in fig.
1, (3) a new analysis of model performance as a of image difficulty
(4) a new metric for validating models’
Experiment
We performed an experiment with human subjects on Mechanical Turk and in the lab in order to determine the minimum amount of viewing time required before subjects could identify an object present in an image.
See fig. 2 for the experiment overview.
Additional procedures and experiment details can be found in the appendix.
We considered an image as recognized at some viewing time when half of the participants could classify it.
Chance on the 1-out-of-50 task is 2%.
Typical images by minimum presentation time are shown in fig.
Images that are quickly recognized by humans, easy images, are most similar to those seen in current datasets, while more difficult images include occlusion or difficult lighting.
An overview of accuracy as a function of viewing time online and in the lab is shown in fig.
Both experiments broadly agree with one another.
In lab, the performance on short timings was significantly worse.
We believe this is largely due to slow monitors which display the image for significantly longer than 17ms, roughly twice as long.
At the high end, subjects in lab were [-nearly
-]nearly
[Figure]
[Figure]
MAE (ViT-B/16)
CLIP (ViT-L/14)
100% accurate, 10% higher than online.
We believe this likely has to do with how distracted subjects were.
In what follows, we focus on the Mechanical Turk experiments due to their scale.
How hard are today’s object recognition datasets?
We use minimum viewing time required for reliable recognition as a proxy for image difficulty.
Datasets today are not gathered to control for difficulty and, indeed, when plotting the difficulty of images in ImageNet and ObjectNet, we find that the difficulty curve for these datasets is highly skewed; see fig.
Rather than plotting four bins, one for each viewing time, we plot a more fine-grained quantity: the total number of incorrect responses out of the 28 presentations of each image (7 participants at 4 timings).
Images with few incorrect responses are easy: all participants at all timings could recognize them, even the short timings.
Images with many incorrect responses are hard: few participants could recognize the images at only a few longer timings.
What we can learn about models from hard images?
Machine accuracy varies as a function of the minimum viewing time required to recognize the object in an image.
Most models, but not all, see a significant performance dropoff between the easy images and the hard images, see fig.
See the appendix for extensive results for dozens of models broken down by image difficulty.
Note that this understates human performance as it shows the Mechanical Turk results.
In-lab, even for the hardest images, humans have nearly perfect performance.
These results also show that the gap between ImageNet and ObjectNet performance increases as image difficulty increases.
Likely, many phenomena such as distribution shift are much more acute for harder rather than easier images.
Image difficulty can tease apart differences between recognition models that would otherwise be lost because of the skewed underlying difficulty distributions in current datasets.
In fig. 5, we plot numerous object recognition models contrasting their performance on the easiest and hardest images.
Rather than computing the absolute performance of models, that would naturally favor larger models with larger training sets, we measure the gap in performance between ImageNet and ObjectNet.
Explaining image difficulty and testing how similar models are to humans.
If models are to not just perform well, but to also process images in ways that are similar to the human visual system, then they should contain a proxy for difficulty.
We investigated three quantities computed from models that could be used to explain the difficulty judgments: c-score [6], prediction depth [7], and adversarial robustness [8].
See the appendix for plots of the correlation between these metrics and difficulty.
This analysis revealed that images that require more viewing time for humans are harder for networks in several ways.
They are learned much later in the training process, are predicted by
As model size increases plain ResNets improve on easy images, but increase their gap on harder images.
Extrapolating the performance of models shows that many current model families are likely to stop or radically slow down performance improvements on hard images, a less optimistic story than aggregate accuracy.
CLIP stands out. Larger CLIP models tend toward only a small gap. later model layers and require much smaller perturbation to successfully adversarially attack.
This underscores that human minimum viewing time has many practical consequences for how networks process images.
Logistic regression including all three metrics classifies images into the four difficulty subsets with 48.5% accuracy, far above chance, but leaving plenty of room to do more.
New metrics for understanding how networks process images could be validated against this data.
Conclusion and related work
As a field, we have measured our progress by models’ performance on tests created by splitting a random subset of images from large-scale web scraped datasets, labeled by a consensus among human annotators [9].
More recently, researchers have realized that there is significant value to testing models’ abilities to recognize objects out of domain [1, 3, 4, 5].
Meding 2022 et al.
[10] investigated differences in model performance by removing images found to be either “trivial” or "impossible for models”.
They also found humans could accurately classify images into these two categories.
Geirhos
et al.
[11] studied human and machine performance under limited presentation time with test set images made more difficult by adding image corruptions.
Rather than focusing on scaling, distribution shift, or control for biases alone, we should also focus on dataset difficulty explicitly.
Today’s datasets skew toward being too easy by undersampling hard images.
ObjectNet was designed for distribution shift and bias control and was not collected from the web, yet its distribution of image difficulties is remarkably similar to that of ImageNet.
By focusing on ways to measure dataset difficulty as datasets are collected, we can better calibrate the entire community, and create the resources needed to push object recognition forward.
In addition to just creating better datasets, understanding performance as a function of difficulty reveals radically different scaling curves for different models and approaches.
It can also provide subsets of images that highlight different types of processing for neuroscientific or behavioral experiments.
While we performed extensive experiments to validate the approach presented here, measuring the difficulty of any one dataset is easy and cheap.
Doing so with a model is free, but noisy, limited by how well the combination of c-score, prediction depth, and adversarial robustness explain the viewing time results.
And of course, the validity of these metrics may go down with significant distribution shift.
On the other hand, measuring dataset difficulty by sampling several hundred images and running an experiment on Mechanical Turk, only costs on the order of low hundreds of dollars per dataset and can be carried out quickly.
The more critical the dataset, and the cost of object recognition failures, the more important doing so is.
Acknowledgments and Disclosure of Funding
We would like to thank Ko Kar, Jim DiCarlo, Dan Yamins, and Martin Schrimpf for their helpful feedback and discussion about our experiments.
We would also like to thank David Lu for his contributions to our early experiments and directions.
This work was supported by the Center for Brains, Minds and Machines, NSF STC award 1231216, the NSF award 2124052, the MIT CSAIL Systems that Learn Initiative, the MIT CSAIL Machine
Learning Applications Initiative, the MIT-IBM Watson AI Lab, the CBMM-Siemens Graduate
Fellowship, the DARPA Artificial Social Intelligence for Successful Teams (ASIST) program, the
DARPA Knowledge Management at Scale and Speed (KMASS) program, the United States Air
Force Research Laboratory and United States Air Force Artificial Intelligence Accelerator under
Cooperative Agreement Number FA8750-19-2-1000, the Air Force Office of Scientific Research (AFOSR) under award number FA9550-21-1-0014, and the Office of Naval Research under award number N00014-20-1-2589, and award number N00014- 20-1-2643.
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government.
The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
A Appendix A.1 Example images A.2 Dataset license
We release or presentation time judgments dataset under the Creative Commons Attribution 4. license.
No personally identifiable data was collected from any subject.
A.3 Image cropping procedure
We draw a bounding box around the object (we use existing bounding boxes for the ImageNet validation set and collect our own bounding boxes for ObjectNet from MTurk).
We initialize the cropping box to be the bounding box.
If the cropping box does not form a square, we extend the shorter side of the rectangular cropping box to form a square.
If the image is not large enough to extend the shorter side of the cropping box, we pad it with black pixels to form a square.
We crop using the cropping box for the image.
The cropped image will be a square.
We resize the cropped image to be 224x224 pixels.
A.4 Mask generation
The masks were generated following the procedure used by [35].
Specifically, a Fourier transform was applied to each image to obtain the magnitude and phase components.
Then, a random array with elements sampled uniformly from [0, 1] was added to the image phase component after which the magnitude and phase components were recombined via an inverse Fourier transform to produce the mask.
Each image was paired with its particular phase-scrambled mask in the experiments.
A.5 Experiment Procedure and Payment
Participants both in the lab and on Mechanical Turk were presented with a document informing them of the purpose, privacy, and risks associated with the experiment and soliciting their consent to participate (see fig. 7).
Participants were then instructed as to how to carry out the experiment and were shown an example video as well as the list of image classes for their review before beginning.
They were informed that they would not need to memorize the classes as the classes would be shown to them after each video.
Participants were also encouraged to take breaks should they feel fatigued or otherwise uncomfortable.
Example instructions are shown in [-ﬁg. 
-]fig.
After giving consent and reading the experiment overview.
participants then completed two calibration steps for to estimate the size of their monitor and their distance from the screen for us to then size the videos appropriately to 8 degrees of visual angle.
First, the participants are shown an image of a credit card and are asked to use a card of their own to adjust a slider to change the size of the card on the screen to the size of their card.
Since credit cards are the same size around the world, this allows us to measure the pixel-to-inches ratio of the participant’s monitor.
Next, the participant completes a blind-spot test [33] that allows us to estimate the distance they are sitting from their screen.
Together, these two measurements are sufficient to compute the desired video eccentricity.
See fig. 9 for images of the calibration steps.
[Table] [Table] [Table] The estimated hourly wage for participants on Mechanical Turk and in the lab was $10/hr and $20/hr respectively with approximately $15,000 spent in total on participant compensation.
A.6 In-Lab Experiment Results
To corroborate our Amazon Mechanical Turk results, we selected 200 images shown to Turk workers to conduct the same experiment in a controlled laboratory setting.
12 individuals came to participate in the experiment in which they viewed and responded to all 200 images on our 144Hz refresh rate monitor with 1ms gray-to-gray time.
After conducting the experiment, 3 individuals had seen each image at each of the 4 presentation times.
When compared to the MTurk results for those same 200 images, the comparison is much as we would expect.
The In-Lab accuracy with shortest image duration (17ms) is less than on MTurk which can likely be contributed to the use of our new, high refresh-rate monitor in the controlled environment.
It is likely that MTurk workers’ personal computers differ in their graphics presentation abilities which may result in the image being visible for slightly greater than 17ms on some monitors.
On the other end, the in-lab experiments reported higher accuracy at the longest image duration (10s) which is also unsurprising as the in-lab participants completed the task in a controlled environment with no distractions and are likely more inclined to take the task seriously and stay focused.
The results show no significant differences in accuracy at the intermediate image durations.
See fig. 3 for a side-by-side comparison between MTurk and In-Lab results.
A.7 Dataset statistics
We collected 28 human responses for each of 5,000 images (2,500 from ImageNet and 2,500 from
ObjectNet).
After reviewing response, 229 images were removed due to either being unrecognizable, mislabeled, or having been seen by the same worker twice despite safeguards in place to disallow it.
Additional dataset statistics are listed in table 1. [Table]
A.8 Finetuned Models
Here we list details regarding training/finetuning procedures for the model results reported in the paper.
A.8.1 Model training procedure
Pretrained models weights were instantiated using publicly available model checkpoints, either using torchvision or found on the model’s source repository.
The models—with the exception of
CLIP—were then finetuned using subsets of the ImageNet training and validation sets containing only the 50 classes we chose to use in the psychophysics experiments.
The models were finetuned for 90 epochs with an SGD optimizer and initial learning rate of 0.1 with momentum value of 0.9 and
Model
ImageNet ObjectNet
ResNet
[Table]
ResNet
[Table] [Table]
[Table]
ResNet
[Table] [Table]
ResNet
[Table]
[Table]
[Table]
ResNet
[Table]
[Table] [Table] [Table] ResNet-50-20% 70.
ResNet88.
[Table] [Table]
ResNet
[Table]
[Table]
[Table]
ResNet
[Table]
ResNet-152-80% 83. ResNet-152-60% 81. ResNet
[Table]
[Table]
ResNet-152-20% 71.
CORNet-S 80. VOneNet-Resnet 83. VOneNet-CORNet-S 80. VGG80.
Noisy Student (EfficientNet-L2) 85.
DenseNet83.
MSDNet Classifier61.
MSDNet Classifier74.
MSDNet Classifier78. MSDNet Classifier78.
MSDNet Classifier79.
SimCLR ResNet 75.
SimCLR ResNet 81.
SimCLR ResNet 83.
CLIP-ViT-B/ 79.
CLIP-ViT-B/ 84.
CLIP-ViT-L/ 91.
CLIP-ViT-L/14@336px 91.
CLIP-ResNet69.
CLIP-ResNet74.
CLIP-ResNet-50x 76.
CLIP-ResNet-50x 79.
CLIP-ResNet-50x 86.
EfficientNet-S 82.
EfficientNet-M 80.
EfficientNet-L 85.
EfficientNet-S89.
EfficientNet-M90.
EfficientNet-L91.
ViT-T/ 58.
ViT-S/ 84.
ViT-B/ 86.
ViT-L/ 94.
MoCo-V 85.
SWSL-ResNext101-32x16d 95.
SWSL-ResNet 93.
MAE-ViT-B/ 89.
A.8.2 Model Performance
We evaluate our finetuned models on the same cropped images used in our psychophysics experiments.
See table 3 for model accuracy reports on the image difficulty reported in the paper and table 2 for model performance on the full ImageNet and ObjectNet subsets of the experiment images.
A.9 Metric calculation procedure
In this section, we go through the details in computing c-score, prediction depth, and adversarial robustness for our experiment images.
A.9.1 C-score
C-score [6] identifies individual image difficulty by characterizing the expected accuracy or a held-out image given training sets of varying size sampled from the data distribution.
In particular, c-score is the frequency of classifying an example correctly when it is omitted from the training set.
However, computing c-score for each image by brute force is computationally infeasible since we must train a separate model for each image.
Instead, we computed the learning speed proxy as recommended by the authors.
Learning speed measures the epoch at which an image is correctly classified by a model.
Intuitively, a training example that is consistent with the training set should be learned quickly because the gradient step for all consistent examples should be similar.
The authors found high
Spearman rank correlation between c-score and cumulative learning speed based proxies.
We trained a ResNet-50 [13] from scratch on ImageNet1k [9] for 90 epochs with an SGD optimizer and initial learning rate of 0.1 with momentum value of 0.9 and weight decay coefficient of 0.0001.
The learning rate decayed by a factor of 2 every 9 epochs and the batch size was 256.
The standard
ImageNet transforms were applied to all images, and the network was initialized randomly.
We then evaluated our experiment images at each epoch and used the average of correct predictions as an estimated c-score for each image. ??
shows the average c-scores for ImageNet and ObjectNet experiment images split by whether the ResNet-50 correctly predicted the image.
C-score serves as an efficient predictor for human recognition difficulty only for images classified by the model in both
ImageNet and ObjectNet.
C-scores for images misclassified by the model do not reveal information about the human recognition difficulty and remain consistently low across all difficulty subsets.
A.9.2 Prediction depth
Prediction depth [7] represents the number of hidden layers after which the network’s final prediction is already determined.
The authors showed that prediction depth is larger for examples that visually appear to be more difficult and is consistent between architectures and random seeds.
We trained a linear decoder at the end of each block of a ResNet-50 on the 50 experiment classes using the ImageNet training and validation set.
We used the same ResNet-50 used to calculate c-scores to ensure consistency of our results.
There are 16 convolutional layers in a ResNet-50; and each linear decoder follows a convolution layer and consists of a pooling layer, flatten layer, and fully-connected layer.
We use the same hyperparameters as appendix A.9.1 and only updated the weights of the linear decoder.
A prediction is defined to be made at depth L = l if the linear classifier after layer L = l − 1 is different from the network’s final prediction, but the classification of the linear decoder after every layer L ≥ l are equal to the final classification of the network.
Images classified by all decoders are said to be predicted at layer 0.
Note that prediction depth is independent of whether the final prediction is correct or not.
It measures the layer at which an image’s prediction converges.
Figure 10 shows the correlation between c-score and image viewing time difficulty.
A.9.3 Adversarial robustness
We measured an image’s distance to the decision boundary of a network using fast gradient sign method (FGSM) [8].
FGSM creates an modified example that maximizes the loss using the gradients of loss with respect to the input image:
[Equation]
where adv x is the modified image, x is the original image, y is the original input label, ϵ is a multiplier adjusted accordingly to control the size of modification step, θ is the model parameters, and J is the
[Figure]
loss function.
Note that gradients are taken with respect to the input image, and model parameters remain constant.
For an image classified by a model, we define its distance to the closest decision boundary of the model as the minimum ϵ needed for the model to misclassify the modified image. On the other hand, for an image misclassified by a model, we define its distance to the closest decision boundary of the model as the minimum ϵ needed for the model to classify the modified image.
We used the same ResNet-50 used to calculate c-scores to ensure consistency of our results.
We finetuned the ResNet-50 on the 50 experiment classes using the ImageNet training and validation set.
We used the same hyperparameters as appendix A.9.1 and only updated the weights of the final pooling, flatten, and fully-connected layer.
We used this finetuned ResNet-50 as the backbone for adversarial perturbation and correction.
While perturbing each classified image, we searched for the smallest ϵ , from 0 to 0.02 incrementing by 1.25e-5 and from 0.02 to 2.5 incrementing by 0.005, that would result in a misclassification.
We only applied only one gradient step when perturbing.
While correcting each misclassified image, we searched for the smallest ϵ , from 0 to 0.001 incrementing by 1.25e-6 and from 0.001 to 0. incrementing by 1.25e-5.
We applied two gradient steps when correcting because correction requires finer and more steps.
Note that the search range depends on the backbone model and the dataset.
One must choose them through manual trial-and-errors to yield interesting and significant results.
Recall that after removing images that were incorrectly annotated, incorrectly cropped, etc ??
, we reduced to 4,771 images from the original 5,000.
Of these, 3,296 and 1,475 images were classified and misclassified by the finetuned ResNet-50 respectively.
We were not able to find an ϵ for every image while perturbing and correcting in the corresponding search range.
We omitted these images in our analysis.
We were able to successfully perturb 2,815 out of 3,296 classified images and correct 1,114 out of 1, misclassified images.
We hypothesized that difficult images that are classified and misclassified would be closer and further from the decision boundary respectively.
fig. 10 confirms the prior hypothesis.
B Compute usage
Roughly four weeks of compute time on two machines with eight TITAN RTX were used to generate the results presented here, largely in computing c-score, prediction depth, adversarial robustness, and finetuning models. C Constructing a metric for image difficulty We propose two metrics:
Difficulty score which provides an exact ranking from most difficult to recognize to least difficult to recognize based on each response 2.
four minimum recognition time (MRT) subsets that quantify the minimum amount of time required for the majority of participants to reliably recognize an image.
Difficulty score is a value from 0 to 28 that represents the number of incorrect predictions given by participants in our experiment across all timings for a particular image.
Each image in our experiment was seen an equal number of times per timing and and only rarely were images that were recognizable at shorter timings also recognizable at longer timings.
This results in a low difficulty score indicating that an image is easy to recognize and a high difficulty score indicating that an image is hard to recognize.
These scores correlate well with the MRT difficulty subsets as shown in fig.
Difficulty score varies significantly by object class as well (see fig. 12).
C.1 Difficulty score distribution by object class