  ImageNet has been arguably the most popular image classification benchmark,
but it is also the one with a significant level of label noise. Recent studies
have shown that many samples contain multiple classes, despite being assumed to
be a single-label benchmark. They have thus proposed to turn ImageNet
evaluation into a multi-label task, with exhaustive multi-label annotations per
image. However, they have not fixed the training set, presumably because of a
formidable annotation cost. We argue that the mismatch between single-label
annotations and effectively multi-label images is equally, if not more,
problematic in the training setup, where random crops are applied. With the
single-label annotations, a random crop of an image may contain an entirely
different object from the ground truth, introducing noisy or even incorrect
supervision during training. We thus re-label the ImageNet training set with
multi-labels. We address the annotation cost barrier by letting a strong image
classifier, trained on an extra source of data, generate the multi-labels. We
utilize the pixel-wise multi-label predictions before the final pooling layer,
in order to exploit the additional location-specific supervision signals.
Training on the re-labeled samples results in improved model performances
across the board. ResNet-50 attains the top-1 classification accuracy of 78.9%
on ImageNet with our localized multi-labels, which can be further boosted to
80.2% with the CutMix regularization. We show that the models trained with
localized multi-labels also outperforms the baselines on transfer learning to
object detection and instance segmentation tasks, and various robustness
benchmarks. The re-labeled ImageNet training set, pre-trained weights, and the
source code are available at {https://github.com/naver-ai/relabel_imagenet}.
