ImageNet-Cartoon and ImageNet-Drawing: Two domain shift datasets for ImageNet
Abstract
Benchmarking the robustness to distribution shifts traditionally relies on dataset collection which is typically laborious and expensive, in particular for datasets with a large number of classes like ImageNet.
An exception to this procedure is
ImageNet-C (Hendrycks & Dietterich, 2019), a dataset created by applying common real-world corruptions at different levels of intensity to the (clean) ImageNet images.
Inspired by this work, we introduce ImageNet-Cartoon and ImageNetDrawing, two datasets constructed by converting ImageNet images into cartoons and colored pencil drawings, using a GAN framework (Wang & Yu, 2020) and simple image processing (Lu et al., 2012), respectively. Code is available at https://github.com/oberman-lab/ imagenet-shift .
1. Introduction
The main challenge in testing the robustness to domain shifts lies in data availability.
Dataset collection is a laborious and expensive procedure.
Previously proposed datasets to test the robustness of ImageNet models include ImageNetv2 (Recht et al., 2019),
ImageNet-Sketch (Wang et al., 2019), ImageNet-A (Hendrycks et al., 2021b), ImageNetR (Hendrycks et al., 2021a), all of which required collecting new images.
The sole exception to this is ImageNetC (Hendrycks & Dietterich, 2019) that applies synthetic corruptions to the validation set in the original ImageNet dataset.
The transformation preserves the image label and therefore there is no need to collect new data.
The bulk of the work consisted in carefully deﬁning the corruptions so they reﬂect real-world transformations.
The corruptions proposed include brightness (variations in daylight intensity), gaussian noise (in low-lighting conditions) and defocus blur (when the image is out of focus), among others.
The two datasets we propose here, ImageNet-Cartoon and
ImageNet-Drawing, follow the same principle of applying label-preserving transformations as in
ImageNet-C.
We choose cartoons and colored drawings for two main reasons: (i) for the original ImageNet dataset the annotators were instructed to collect “photos only, no painting, no drawings, etc.”
(Deng, 2012) and thus cartoons and drawings constitute a natural domain shift; (ii) cartoons and sketches 1 are part of the common domain shifts included in the datasets used in the domain adaptation literature: DomainNet (Peng et al., 2019) includes cliparts and sketches, Ofﬁce-Home (Venkateswara et al., 2017) includes cliparts and PACS Li et al.
(2017) includes cartoons and sketches.
The images in ImageNet-Cartoon are obtained using the
GAN framework proposed by Wang & Yu (2020) which converts photo images into cartoons.
To generate the images in ImageNet-Drawing we follow the work of Lu et al.
(2012) that transforms real photos into colored pencil drawings using image processing alone.
In Table 1, we display the accuracies of PyTorch pre-trained ImageNet models with different architectures.
While it is reasonable to say that for humans the task difﬁculty remains essentially the same, there is on average an 18 and 45 percent points accuracy drop for the deep neural network models on ImageNetCartoon and ImageNet-Drawing, respectively, which highlights the how challenging these datasets are for current models.
2. Related Work
Deep neural networks and 3D graphics engines have been used to create new datasets as they allow us to generate new labeled images.
Nakkiran et al.
(2021) introduce CIFAR5m, a dataset of 6 million synthetic CIFAR-10-like images which are generated with the Denoising Diffusion generative model of Ho et al.
More recently, Li et al.
(2022) leverage a BigGAN
(Brock et al., 2019) and a VQGAN (Esser et al., 2021) to synthesize a new ImageNet bench- [Table] mark with pixel-wise labels.
More pertinent to our work, is the VisDA-C dataset (Peng et al., 2017) whose training set consists of 3D renderings of different objects, while the validation and test sets consist in real-images from two different sources.
However, it is limited to only 12 different classes and as pointed in (Li et al., 2022) “creating ImageNet-level of class and instance diversity via the graphics approach would require signiﬁcant 3D content acquisition efforts” [Table]
The aforementioned ImageNet-C (Hendrycks & Dietterich, 2019), where different synthetic corruptions at different intensity levels are applied to ImageNet images, is perhaps the closest work to ours.
Similarly, style transformations were considered in (Geirhos et al., 2019) leading to StylizedImageNet (SIN): the texture of each image is altered based on the style of a randomly selected painting through AdaIN style transfer (Huang & Belongie, 2017).
By exposing the model to SIN images during training the goal was to reduce the texture bias of Convolutional Neural Networks and force the model to recognize objects based on their shape.
However, the images in SIN, while partially retaining their shape content, have their overall appearance severely altered with a loss of semantic meaning and thus are not conducive to test the robustness to domain shift (see Figure 2 for an example).
3. Dataset synthesis
In this section we discuss the two methods used to transform the images in the validation set of the (original)
ImageNet dataset into cartoons and drawings, forming respectively our proposed ImageNet-Cartoon and ImageNet-Drawing.
Examples are shown in Figure 1 and additional ones in
Figure 5 in the Appendix.
3.1. ImageNet-Cartoon
Wang & Yu (2020) propose a GAN framework to cartoonize a real photo into a cartoon that we leverage here to generate our dataset ImageNet-Cartoon.
Their method relies on decomposing images into three representations: (i) surface representation capturing the smooth surface of the images, mimicking a ﬁrst rough draft drawn by cartoonists that is later retouched and ﬁlled with details; (ii) structure representation that emulates ﬂattened global content, sparse color blocks, and clear boundaries, thus capturing the sparse visual effects; (iii) texture representation, a gray-scale representation retaining the details and edges but independent of color and luminance.
Three independent modules are used to extract the above representations, while the GAN framework itself contains a generator G , a fully-convolutional
U-Net-like (Ronneberger et al., 2015) network, and two discriminators D s and D t , with PatchGAN (Isola et al., 2017) networks, that distinguish the surface and structure representations of the generator output and cartoon images.
In addition, a pre-trained VGG network (Simonyan & Zis- serman, 2014) is used to extract high-level features and to impose spatial constrains on the global content.
We refer the reader to Wang & Yu (2020) for additional details.
Here we make use of the pre-trained generator model G provided in the ofﬁcial Tensorﬂow implementation 2 to construct
ImageNet-Cartoon.
If x is an ImageNet image, then we simply store G ( x ) in our dataset.
Using a single Tesla P
PCIe GPU with 16 GB, we are able to generate ImageNetCartoon in under 48 minutes.
For comparison, downloading
Imagenet-Cartoon at an average speed of 10Mbps would take 53 minutes.
In the supplemental material, we provide the necessary code to produce ImageNet-Cartoon.
3.2. ImageNet-Drawing
Lu et al.
(2012) propose a multi-stage procedure to create colored pencil drawings from real photos.
First, a line drawing with strokes, s , is produced based on the convolution of kernels representing lines in 8 possible different directions with the gradient norm of the gray-scale version of the input image.
Then, tone adjustment is performed leveraging parametric histogram models learned based on statistics from a set of sketch examples producing the tone map j .
This is then combined with a predeﬁned drawing pattern h to produce the pencil texture rendering t .
Finally, the pencil stroke s is combined with the tonal texture t through pointwise multiplication generating the grayscale pencil sketch r = s · t .
The ﬁnal color pencil drawing is obtained by taking sketch r as the brightness layer (the Y channel in the
YUV color space) of the original image.
We illustrate the entire process in Figure 4.
For the implementation, we used (Daniel, 2018).
Perhaps a bit surprising, the choice of the drawing pattern h has a signiﬁcant impact in the accuracy of deep neural networks.
In Table 2, we report the accuracies on ImageNet-Drawing datasets generated with 4 different drawing patterns.
The latter are shown in Figure 3.
We choose the pattern that causes the most drop in accuracy to form the ImageNet-Drawing dataset. [Table]
4. Evaluation Protocol
ImageNet-Cartoon and ImageNet-Drawing constitute domain shifts with respect to the (original) ImageNet dataset.
Therefore they can be used to test the robustness to domain shifts with the metric being the standard classiﬁcation accuracy.
Our experiments show that there is an average of 18 and 45 percent points drop in accuracy for ImageNetCartoon and ImageNet-Drawing, respectively (see Table 1).
On the other hand, one would expect humans to maintain their accuracy.
This highlights the usefulness of the proposed datasets.
In addition, calibration metrics can be considered as well.
These are of particular importance as vision models are increasingly used in safety-critical applications and therefore it is important to have a precise estimation of the predictive uncertainty of the models.
The most common metric is the
Expected Calibration Error (ECE)
[Equation]
Let f ( x ) denote the output of the deep neural network for the input image x with associated label y , after applying the softmax layer.
The predicted class ˆ y is given by the most likely output and the associated score is taken to be the conﬁdence
[Equation]
Then, given a set of samples { x (1) , . .
, x ( n ) } , the ECE (Eq. 1) is calculated in two steps.
First the conﬁdence [Table] scores { ˆ c (1) , . .
, ˆ c ( n ) } of samples are partitioned into M bins { B m } Mm =1 of equal mass.
Second, the weighted average of the differences between the average conﬁdence conf( B m ) =
| Bm |
P i 2 B m ˆ c ( i ) and the accuracy acc
( Bm ) =
| Bm |
P i 2 B m 1 y
( i ) =
ˆ y ( i ) in each bin is computed as the ECE metric, where | B m | denotes the size of bin B m , 1 is the indicator function.
While it is common to use equally spaced bins, we emphasize that one should use equal mass bins as described above in order to mitigate the bias in the estimation (Roelofs et al., 2022).
In addition, Classwise-ECE (Kull et al., 2019) and Brier score (Brier, 1950) which measure classwise calibration can also be considered.
We show the results for the ECE error, as described above, using 15 bins in Table 3.
In general, all models are poorly calibrated, some more than other.
This is in agreement with Minderer et al.
(2021) whose results suggest that the architecture plays a major role in determining calibration properties.
In addition, we highlight the following: (i) the ViT-B/16 model has better calibration on ImageNet-Cartoon and Imagenet-Drawing than on ImageNet; (ii) the ConvNeXt-S model, despite achieving the best accuracy on ImageNet and ImageNet-Cartoon it has the worst calibration on those same datasets.
5. Conclusions
By leveraging existing work, we are able to create two new datasets, ImageNet-Cartoon and ImageNet-Drawing, that can be used to test the robustness of models to domain shift.
We show that current pre-trained ImageNet models fail to generalize to these datasets exhibiting anand 45 percent points decrease in accuracy on average for ImageNet-Cartoon and ImageNet-Drawing, respectively, in comparison to ImageNet. Acknowledgements Thanks to CIFAR for their support through the CIFAR AI
Chairs program.
TS thanks KC Tsiolis and Vikram Voleti for constructive feedback on the ﬁrst draft of this manuscript.