Original: When does dough become a bagel? Analyzing the remaining mistakes on ImageNet
Revised: When does dough become a bagel? Analyzing the remaining mistakes on ImageNet

Original: Anonymous Authors

Original: Abstract
Revised: Abstract

Original: Image classiﬁcation accuracy on the ImageNet dataset has been a barometer for progress in computer vision over the last decade.
Revised: Image classiﬁcation accuracy on the ImageNet dataset has been a barometer for progress in computer vision over the last decade.

Original: Several recent papers have questioned the degree to which the benchmark remains useful to the community (Stock & Cissé, 2017; Beyer et al., 2020;
Revised: Several recent papers have questioned the degree to which the benchmark remains useful to the community (Stock & Cissé, 2017; Beyer et al., 2020;

Original: Shankar et al., 2020; Yun et al., 2021; Tsipras et al., 2020), yet innovations continue to contribute gains to performance, with today’s largest models achieving 90%+ top-1 accuracy.
Revised: Shankar et al., 2020; Yun et al., 2021; Tsipras et al., 2020), yet innovations continue to contribute gains to performance, with today’s largest models achieving 90%+ top-1 accuracy.

Original: To help contextualize progress on ImageNet and provide a more meaningful evaluation for today’s state-ofthe-art models, we manually review and categorize every remaining mistake that a few top models make and provide insights into the long-tail of errors on one of the most benchmarked datasets in computer vision.
Revised: To help contextualize progress on ImageNet and provide a more meaningful evaluation for today’s state-ofthe-art models, we manually review and categorize every remaining mistake that a few top models make and provide insights into the long-tail of errors on one of the most benchmarked datasets in computer vision.

Original: We focus on the multi-label subset evaluation of ImageNet, where today’s best models achieve upwards of 97% accuracy.
Revised: We focus on the multi-label subset evaluation of ImageNet, where today’s best models achieve upwards of 97% accuracy.

Original: Our analysis reveals that nearly half of the supposed mistakes are not mistakes at all, and we uncover new valid multi-labels, demonstrating that, without careful review, we are signiﬁcantly underestimating the performance of these models.
Revised: Our analysis reveals that nearly half of the supposed mistakes are not mistakes at all, and we uncover new valid multi-labels, demonstrating that, without careful review, we are signiﬁcantly underestimating the performance of these models.

Original: On the other hand, we also ﬁnd that today’s best models still make a signiﬁcant number of mistakes (40%) that are obviously wrong to human reviewers.
Revised: On the other hand, we also ﬁnd that today’s best models still make a signiﬁcant number of mistakes (40%) that are obviously wrong to human reviewers.

Original: To calibrate future progress on ImageNet, we provide an updated multi-label evaluation set, and we curate ImageNet-M ajor: a -example "major error" slice of the obvious mistakes made by today’s top models—a slice where models should achieve near perfection, but today are far from doing so.
Revised: To calibrate future progress on ImageNet, we provide an updated multi-label evaluation set, and we curate ImageNet-M ajor: a 68-example "major error" slice of the obvious mistakes made by today’s top models—a slice where models should achieve near perfection, but today are far from doing so.

Original: 1. Introduction
Revised: 1. Introduction

Original: Label: dough; Model: bagel. When does dough become a bagel?
Revised: [Table]

Original: Computer vision models often evaluate their performance on the ImageNet classiﬁcation dataset (Deng et al., 2009; Russakovsky et al., 2015) and many variants (Recht et al., 2019;
Revised: Computer vision models often evaluate their performance on the ImageNet classiﬁcation dataset (Deng et al., 2009; Russakovsky et al., 2015) and many variants (Recht et al., 2019;

Original: Hendrycks & Dietterich, 2019; Hendrycks et al., 2019; 2020; Wang et al., 2019), as a signal of capability for visual understanding.
Revised: Hendrycks & Dietterich, 2019; Hendrycks et al., 2019; 2020; Wang et al., 2019), as a signal of capability for visual understanding.

Original: As performance on the standard sets have reached diminishing returns to top-1 and top-5 accuracy, much recent work (Stock & Cissé, 2017; Beyer et al., 2020; Recht et al., 2019; Shankar et al., 2020; Tsipras et al., 2020; Kornblith et al., 2019) has focused on understanding what is left for the computer vision community to solve, and where the community should be driving toward.
Revised: As performance on the standard sets have reached diminishing returns to top-1 and top-5 accuracy, much recent work (Stock & Cissé, 2017; Beyer et al., 2020; Recht et al., 2019; Shankar et al., 2020; Tsipras et al., 2020; Kornblith et al., 2019) has focused on understanding what is left for the computer vision community to solve, and where the community should be driving toward.

Original: Prior studies of ImageNet errors have identiﬁed issues stemming from lack of multi-labels, label noise, under-speciﬁed classes, and more (Stock & Cissé, 2017; Shankar et al., 2020; Beyer et al., 2020;
Revised: Prior studies of ImageNet errors have identiﬁed issues stemming from lack of multi-labels, label noise, under-speciﬁed classes, and more (Stock & Cissé, 2017; Shankar et al., 2020; Beyer et al., 2020;

Original: Tsipras et al., 2020; Lee et al., 2017).
Revised: Tsipras et al., 2020; Lee et al., 2017).

Original: Label errors and label noise affect the evaluation of any model (Northcutt et al., 2021b), and ImageNet is no exception.
Revised: Label errors and label noise affect the evaluation of any model (Northcutt et al., 2021b), and ImageNet is no exception.

Original: Many studies above have spent effort to correct and improve these labels, showing that while ImageNet performance improvements are approaching diminishing returns, the dataset can remain useful to the community, but only if we collectively continue to shepherd it.
Revised: Many studies above have spent effort to correct and improve these labels, showing that while ImageNet performance improvements are approaching diminishing returns, the dataset can remain useful to the community, but only if we collectively continue to shepherd it.

Original: As the best models improve, however, it is becoming increasingly challenging to assess the often novel predictions these models make.
Revised: As the best models improve, however, it is becoming increasingly challenging to assess the often novel predictions these models make.

Original: For example, should we penalize models for being the ﬁrst to predict that a pre-baked bagel may be a bagel, as one of the models we review in this work does?
Revised: For example, should we penalize models for being the ﬁrst to predict that a pre-baked bagel may be a bagel, as one of the models we review in this work does?

Original: Machine learning models tend to make mistakes with varying severity and importance (a function of both the prediction as well as the label deﬁnitions), and prior work (Stock & Cissé, 2017; Beyer et al., 2020) has shown that nonexperts ﬁnd it challenging to determine the correctness of a model’s prediction on ImageNet.
Revised: Machine learning models tend to make mistakes with varying severity and importance (a function of both the prediction as well as the label deﬁnitions), and prior work (Stock & Cissé, 2017; Beyer et al., 2020) has shown that nonexperts ﬁnd it challenging to determine the correctness of a model’s prediction on ImageNet.

Original: Our own experience, highlighted by the doughy-bagel, is that many of the remaining mistakes these top models make are quite reasonable and probably should not be considered mistakes—understanding the severity and type of these remaining mistakes can help calibrate progress.
Revised: Our own experience, highlighted by the doughy-bagel, is that many of the remaining mistakes these top models make are quite reasonable and probably should not be considered mistakes—understanding the severity and type of these remaining mistakes can help calibrate progress.

Original: Indeed, to our knowledge, there has not been an expertreview, categorization, and severity assessment of the remaining long-tailed mistakes, which becomes particularly important at these margins.
Revised: Indeed, to our knowledge, there has not been an expertreview, categorization, and severity assessment of the remaining long-tailed mistakes, which becomes particularly important at these margins.

Original: Our experience working with production teams on deployed applications has suggested that manual triage and assessing individual failures provides a useful indicator of model performance that aggregate measures fail to capture.
Revised: Our experience working with production teams on deployed applications has suggested that manual triage and assessing individual failures provides a useful indicator of model performance that aggregate measures fail to capture.

Original: Thus, in this work we attempt to analyze (as expert reviewers)
Revised: Thus, in this work we attempt to analyze (as expert reviewers)

Original: every remaining mistake that a few state-of-the-art models make to better understand (a) which of the remaining mistakes remain egregious errors, (b) what error category they might fall in, and (c) what evaluations might capture the most important long-tail failures.
Revised: every remaining mistake that a few state-of-the-art models make to better understand (a) which of the remaining mistakes remain egregious errors, (b) what error category they might fall in, and (c) what evaluations might capture the most important long-tail failures.

Original: In this paper we analyze the ImageNet multi-label validation subsets (Shankar et al., 2020), in which expert labelers were used to assess the correctness of model predictions through the year 2020, and on which a 1000-image human-evaluated subset provides a direct comparison to expert human performance.
Revised: In this paper we analyze the ImageNet multi-label validation subsets (Shankar et al., 2020), in which expert labelers were used to assess the correctness of model predictions through the year 2020, and on which a 1000-image human-evaluated subset provides a direct comparison to expert human performance.

Original: By analyzing the mistakes of two large 2022-era
Revised: By analyzing the mistakes of two large 2022-era

Original: ImageNet models, we found that:
Revised: ImageNet models, we found that:

Original: • Nearly half of each model’s mistakes were deemed correct under a careful, expert multi-label reevaluation, halving the error rate.
Revised: • Nearly half of each model’s mistakes were deemed correct under a careful, expert multi-label reevaluation, halving the error rate.

Original: Had we not analyzed the models’ mistakes, we would be severely underestimating the models’ actual performance.
Revised: Had we not analyzed the models’ mistakes, we would be severely underestimating the models’ actual performance.

Original: • Approximately 40% of the remaining mistakes can be classiﬁed as ‘major’ errors : errors that most humans would likely not make, suggesting that many of the long-tailed mistakes aren’t simply label noise, but legitimate mistakes that leave room for improvement.
Revised: • Approximately 40% of the remaining mistakes can be classiﬁed as ‘major’ errors : errors that most humans would likely not make, suggesting that many of the long-tailed mistakes aren’t simply label noise, but legitimate mistakes that leave room for improvement.

Original: What do these lessons portend for the future of ImageNet evaluation?
Revised: What do these lessons portend for the future of ImageNet evaluation?

Original: Top-1 will become increasingly noisy as our best models get better (though we have not yet completely saturated top-1).
Revised: Top-1 will become increasingly noisy as our best models get better (though we have not yet completely saturated top-1).

Original: Our work shows that multi-label accuracy, while better at capturing "true" errors compared to top-1, suffers from a lack of a comprehensive, accurately labeled large evaluation set, which is expensive to procure and challenging to maintain.
Revised: Our work shows that multi-label accuracy, while better at capturing "true" errors compared to top-1, suffers from a lack of a comprehensive, accurately labeled large evaluation set, which is expensive to procure and challenging to maintain.

Original: We therefore propose ImageNet-M, a 68-example evaluation split composed of "major" mistakes that several topperforming models make; we believe this subset is one that future image classiﬁcation models should achieve near perfect accuracy on, and provides three clear beneﬁts: (1) we attempt to comprehensively-label all examples for multilabel annotations to prevent the need to review novel correct predictions, (2) we endeavor to maintain and provide a way for the public to add new correct predictions; (3) the evaluation set is small enough to encourage completeness and allow the community to inspect their own errors.
Revised: We therefore propose ImageNet-M, a 68-example evaluation split composed of "major" mistakes that several topperforming models make; we believe this subset is one that future image classiﬁcation models should achieve near perfect accuracy on, and provides three clear beneﬁts: (1) we attempt to comprehensively-label all examples for multilabel annotations to prevent the need to review novel correct predictions, (2) we endeavor to maintain and provide a way for the public to add new correct predictions; (3) the evaluation set is small enough to encourage completeness and allow the community to inspect their own errors.

Original: Dataset release.
Revised: Dataset release.

Original: To evaluate on the imagenet2012_multilabel and the ImageNet-M subset, please visit https://www.tensorflow.
Revised: To evaluate on the imagenet2012_multilabel and the ImageNet-M subset, please visit https://www.tensorflow.

Original: org/datasets/catalog/imagenet2012_ multilabel .
Revised: org/datasets/catalog/imagenet2012_ multilabel .

Original: We also include a notebook with example code for evaluating on these splits, including multi-label accuracy and the ImageNet-M split using pre-computed logits for the ViT-3B and Greedy Soups models used in the paper.
Revised: We also include a notebook with example code for evaluating on these splits, including multi-label accuracy and the ImageNet-M split using pre-computed logits for the ViT-3B and Greedy Soups models used in the paper.

Original: 2. Mistake analysis method and taxonomy
Revised: 2. Mistake analysis method and taxonomy

Original: To obtain an initial set remaining mistakes, we used a standard ViT (Dosovitskiy et al., 2021) model scaled to 3B parameters (ViT-3B) that was pre-trained on JFT-3B (Sun et al., 2017) and ﬁne-tuned on ImageNet-1K (Deng et al., 2009), achieving a top-1 accuracy of 89.5% (details in Appendix E).
Revised: To obtain an initial set remaining mistakes, we used a standard ViT (Dosovitskiy et al., 2021) model scaled to 3B parameters (ViT-3B) that was pre-trained on JFT-3B (Sun et al., 2017) and ﬁne-tuned on ImageNet-1K (Deng et al., 2009), achieving a top-1 accuracy of 89.5% (details in Appendix E).

Original: We also later reviewed mistakes made by the
Revised: We also later reviewed mistakes made by the

Original: Greedy Soups model (Wortsman et al., 2022).
Revised: Greedy Soups model (Wortsman et al., 2022).

Original: Using the imagenet2012_multilabel dataset (Shankar et al., 2020), we measured the initial multi-label accuracy (MLA) of the ViT-3B model to be 96.3%.
Revised: Using the imagenet2012_multilabel dataset (Shankar et al., 2020), we measured the initial multi-label accuracy (MLA) of the ViT-3B model to be 96.3%.

Original: We describe the procedure we followed to review mistakes using in Appendix B. and the severity and category deﬁnitions in Appendix B.2.
Revised: We describe the procedure we followed to review mistakes using in Appendix B. and the severity and category deﬁnitions in Appendix B.2.

Original: 3. Analyzing the remaining mistakes
Revised: 3. Analyzing the remaining mistakes

Original: After review of all original 676 mistakes (comprising both novel predictions and previously reviewed mistakes), we found that a total of 298 were either correct or unclear, or determined the original groundtruth incorrect or problematic.
Revised: After review of all original 676 mistakes (comprising both novel predictions and previously reviewed mistakes), we found that a total of 298 were either correct or unclear, or determined the original groundtruth incorrect or problematic.

Original: Our evaluation of the ViT-3B model on this re-labeled dataset is shown in Table 1, with the model making a total of 378 mistakes on the dataset.
Revised: Our evaluation of the ViT-3B model on this re-labeled dataset is shown in Table 1, with the model making a total of 378 mistakes on the dataset.

Original: In other words, approximately 44% of the initial mistakes made by this model were determined to be correct!
Revised: In other words, approximately 44% of the initial mistakes made by this model were determined to be correct!

Original: 3.1. Mistake category and severity
Revised: 3.1. Mistake category and severity

Original: Each of the 378 remaining mistakes was assigned both a mistake category and severity (Table 2) by the expert panel.
Revised: Each of the 378 remaining mistakes was assigned both a mistake category and severity (Table 2) by the expert panel.

Original: Category: 78.3% of errors were assessed to be ﬁne-grained in nature (either in-vocabulary or OOV), and 13.8% categorized as a spurious correlation.
Revised: Category: 78.3% of errors were assessed to be ﬁne-grained in nature (either in-vocabulary or OOV), and 13.8% categorized as a spurious correlation.

Original: To measure whether these categories are meaningful, we measured the "hierarchy distance" between the groundtruth label and the model’s prediction using the WordNet hierarchy (Miller, 1995).
Revised: To measure whether these categories are meaningful, we measured the "hierarchy distance" between the groundtruth label and the model’s prediction using the WordNet hierarchy (Miller, 1995).

Original: For example, a hierarchy distance of 1 means that the groundtruth and model prediction share the same parent; a distance of 2 means they share the same grandparent.
Revised: For example, a hierarchy distance of 1 means that the groundtruth and model prediction share the same parent; a distance of 2 means they share the same grandparent.

Original: We found that 80.9% of errors with a hierarchy distance of 1 were assessed as ﬁne-grained.
Revised: We found that 80.9% of errors with a hierarchy distance of 1 were assessed as ﬁne-grained.

Original: In contrast, 54.3% of errors with a hierarchy distance of 3 were ﬁne-grained, matching intuition that predictions close in the hierarchy are very likely to beAll
Revised: In contrast, 54.3% of errors with a hierarchy distance of 3 were ﬁne-grained, matching intuition that predictions close in the hierarchy are very likely to be

Original: [Table]
Revised: [Table]

Original: ﬁne-grained errors, and predictions far in the hierarchy are more likely to be spurious correlations, ﬁne-grained with out-of-vocabulary, or non-prototypical examples.
Revised: ﬁne-grained errors, and predictions far in the hierarchy are more likely to be spurious correlations, ﬁne-grained with out-of-vocabulary, or non-prototypical examples.

Original: We note that WordNet does not provide a perfect (automated) category function: "goblet" and "vase" are a hierarchy distance of 4 apart, and we encountered one model mistake for this pair that we nonetheless assessed as ﬁne-grained.
Revised: We note that WordNet does not provide a perfect (automated) category function: "goblet" and "vase" are a hierarchy distance of 4 apart, and we encountered one model mistake for this pair that we nonetheless assessed as ﬁne-grained.

Original: Severity: We determined that around 40% of errors were assessed to be "major" errors, indicating that this model still appears to make mistakes that a human familiar with the class deﬁnitions would not make, despite the fact that the model on average performs better than an expert human.
Revised: Severity: We determined that around 40% of errors were assessed to be "major" errors, indicating that this model still appears to make mistakes that a human familiar with the class deﬁnitions would not make, despite the fact that the model on average performs better than an expert human.

Original: We return to ‘major’ errors later in Section 4.1, as we believe that a subset of these errors can be a useful evaluation slice for future ImageNet benchmarking.
Revised: We return to ‘major’ errors later in Section 4.1, as we believe that a subset of these errors can be a useful evaluation slice for future ImageNet benchmarking.

Original: 3.2. Generalization to new models
Revised: 3.2. Generalization to new models

Original: As models produce higher top-1 accuracy, how do the types of mistakes they make and improve upon change?
Revised: As models produce higher top-1 accuracy, how do the types of mistakes they make and improve upon change?

Original: We use the Greedy Soups model that obtains 90.9% top-1 accuracy on ImageNet validation (Wortsman et al., 2022), measuring its MLA after our initial re-labeling at 98.1%, and yielding 341 total remaining (and partially unreviewed) errors.
Revised: We use the Greedy Soups model that obtains 90.9% top-1 accuracy on ImageNet validation (Wortsman et al., 2022), measuring its MLA after our initial re-labeling at 98.1%, and yielding 341 total remaining (and partially unreviewed) errors.

Original: The Soups model corrected 209 mistakes that the ViT-3B model made, while the model made 170 mistakes where the ViT-3B model was correct, yielding an overall accuracy improvement; 28 mistake examples were common with ViT3B but with a different prediction.
Revised: The Soups model corrected 209 mistakes that the ViT-3B model made, while the model made 170 mistakes where the ViT-3B model was correct, yielding an overall accuracy improvement; 28 mistake examples were common with ViT3B but with a different prediction.

Original: In total there werenovel predictions made by this model that needed to be reviewed; upon review using the same panel method, we found 46.5% (92/198) were problematic (10) or actually correct (82), showing with a second model that model predictions on mistakes need to be reviewed, and that the single label expected by top-1 is often insufﬁcient.
Revised: In total there werenovel predictions made by this model that needed to be reviewed; upon review using the same panel method, we found 46.5% (92/198) were problematic (10) or actually correct (82), showing with a second model that model predictions on mistakes need to be reviewed, and that the single label expected by top-1 is often insufﬁcient.

Original: The Soups model in the end made only 249 errors, for an MLA of 98.6%, a 0.5% absolute increase compared to unreviewed mistakes.
Revised: The Soups model in the end made only 249 errors, for an MLA of 98.6%, a 0.5% absolute increase compared to unreviewed mistakes.

Original: The categorization and severity of these errors is shown in
Revised: The categorization and severity of these errors is shown in

Original: Table 2.
Revised: Table 2.

Original: A chi-square test of independence shows that there is no signiﬁcant difference between either the categorization or severity (
Revised: A chi-square test of independence shows that there is no signiﬁcant difference between either the categorization or severity (

Original: χ 2 (3 , N = 629) = 2 .
Revised: χ 2 (3 , N = 629) = 2 .

Original: 41 , p = .
Revised: 41 , p = .

Original: or mistake severity ( χ 2 (1 , N = 629) = 1 . 61 , p = . 20 ).
Revised: or mistake severity ( χ 2 (1 , N = 629) = 1 . 61 , p = . 20 ).

Original: Additional results: We include additional results analyzing generalization of our mistake analysis to ImageNetV2, analyzing class confusions, comparing model performance to human performance, and analyzing model errors through the lens of training data in Appendix D.
Revised: Additional results: We include additional results analyzing generalization of our mistake analysis to ImageNetV2, analyzing class confusions, comparing model performance to human performance, and analyzing model errors through the lens of training data in Appendix D.

Original: 4. Recommendations and Discussion
Revised: 4. Recommendations and Discussion

Original: In this section we provide recommendations for future evaluation, starting with ImageNet-M: our curated multi-label evaluation set of "major mistakes" that we suggest should be reported on in addition to metrics such as top-1 and multi-label accuracy.
Revised: In this section we provide recommendations for future evaluation, starting with ImageNet-M: our curated multi-label evaluation set of "major mistakes" that we suggest should be reported on in addition to metrics such as top-1 and multi-label accuracy.

Original: 4.1. ImageNet-M: A "major mistakes" evaluation split
Revised: 4.1. ImageNet-M: A "major mistakes" evaluation split

Original: Studies over the last few years seeking to understand whether ImageNet remains an informative benchmark have typically concluded that aspects of ImageNet remain useful but top-1 accuracy less so.
Revised: Studies over the last few years seeking to understand whether ImageNet remains an informative benchmark have typically concluded that aspects of ImageNet remain useful but top-1 accuracy less so.

Original: These works encourage alternative related metrics for ImageNet such as multi-label accuracy (Beyer et al., 2020; Shankar et al., 2020) or object vs. organism breakdowns (Shankar et al., 2020).
Revised: These works encourage alternative related metrics for ImageNet such as multi-label accuracy (Beyer et al., 2020; Shankar et al., 2020) or object vs. organism breakdowns (Shankar et al., 2020).

Original: In many cases, stronger, more generalizable models often continue to incrementally but inevitably improve on these metrics.
Revised: In many cases, stronger, more generalizable models often continue to incrementally but inevitably improve on these metrics.

Original: With an emphasis on understanding which long-tail errors are unambiguously errors, we suggest that a benchmark focusing on the most egregious long-tail errors could provide a useful additional signal about whether the improvements are meaningful.
Revised: With an emphasis on understanding which long-tail errors are unambiguously errors, we suggest that a benchmark focusing on the most egregious long-tail errors could provide a useful additional signal about whether the improvements are meaningful.

Original: In particular, we desire a long-tailed benchmark where we believe that 100% accuracy is achievable.
Revised: In particular, we desire a long-tailed benchmark where we believe that 100% accuracy is achievable.

Original: Because the minor mistakes are more subject to interpretation and discussion than the major mistakes, we believe a benchmark focused on the latter will help the community judge what is meaningful improvement on ImageNet.
Revised: Because the minor mistakes are more subject to interpretation and discussion than the major mistakes, we believe a benchmark focused on the latter will help the community judge what is meaningful improvement on ImageNet.

Original: To that end, we leverage our expert-reviewed analysis to produce a small slice of the ImageNet multi-label set where (1) today’s best top-1 models are still more wrong than right, and (2) the mistakes are largely unambiguous to a human given a reasonable understanding of the ImageNet label set.
Revised: To that end, we leverage our expert-reviewed analysis to produce a small slice of the ImageNet multi-label set where (1) today’s best top-1 models are still more wrong than right, and (2) the mistakes are largely unambiguous to a human given a reasonable understanding of the ImageNet label set.

Original: We call this evaluation slice ImageNet-M ajor.
Revised: We call this evaluation slice ImageNet-M ajor.

Original: ImageNet-M example selection method.
Revised: ImageNet-M example selection method.

Original: The ViT-3B model made 155 "major" mistakes, for which we analyzed whether each example was labeled correctly for three additional models: (1) the Greedy Soups model, (2) a model pre-trained on Instagram data but ﬁne-tuned on ImageNet that achieves 85.4% top-1 (Mahajan et al., 2018), and (3) A zero-shot evaluation (Radford et al., 2021; Jia et al., 2021;
Revised: The ViT-3B model made 155 "major" mistakes, for which we analyzed whether each example was labeled correctly for three additional models: (1) the Greedy Soups model, (2) a model pre-trained on Instagram data but ﬁne-tuned on ImageNet that achieves 85.4% top-1 (Mahajan et al., 2018), and (3) A zero-shot evaluation (Radford et al., 2021; Jia et al., 2021;

Original: Pham et al., 2021) using a CoCa (Yu et al., 2022) model pretrained on JFT and noisy image-text data.
Revised: Pham et al., 2021) using a CoCa (Yu et al., 2022) model pretrained on JFT and noisy image-text data.

Original: In order to maximize prediction diversity, we purposefully selected models with varying pre-training data and training methodologies, including a zero-shot model that does not see ImageNet image-label associations directly (Gontijo-Lopes
Revised: In order to maximize prediction diversity, we purposefully selected models with varying pre-training data and training methodologies, including a zero-shot model that does not see ImageNet image-label associations directly (Gontijo-Lopes

Original: [Table]
Revised: [Table]

Original: [Table]

Original: et al., 2021).
Revised: et al., 2021).

Original: From this suite of four models, we assembled a subset where three or more of the models make a major mistake, yielding 68 such major mistakes.
Revised: From this suite of four models, we assembled a subset where three or more of the models make a major mistake, yielding 68 such major mistakes.

Original: This process is similar in spirit to ImageNet-A
Revised: This process is similar in spirit to ImageNet-A

Original: (Hendrycks et al., 2019), except we use four high-performing but diverse models, and we restrict the set to ImageNet images and the corresponding model prediction that were rated as "major" errors.
Revised: (Hendrycks et al., 2019), except we use four high-performing but diverse models, and we restrict the set to ImageNet images and the corresponding model prediction that were rated as "major" errors.

Original: We analyzed the predictions of all models on these examples (including any novel predictions made by these additional models) and veriﬁed that none of them were correct new multi-labels, and that any model’s mistakes were major mistakes.
Revised: We analyzed the predictions of all models on these examples (including any novel predictions made by these additional models) and veriﬁed that none of them were correct new multi-labels, and that any model’s mistakes were major mistakes.

Original: In addition, we attempted to comprehensively manually label additional labels that no model has yet predicted but would be correct, in an attempt to reduce the likelihood that future models are penalized for making novel but unreviewed correct predictions.
Revised: In addition, we attempted to comprehensively manually label additional labels that no model has yet predicted but would be correct, in an attempt to reduce the likelihood that future models are penalized for making novel but unreviewed correct predictions.

Original: We design the ImageNet-M 68-example subset as an additional split of the validation set that we believe has the following properties: (a) many top performing models trained in different ways make mistakes on this set; (b) the mistakes are all major mistakes as determined by expert-reviewers; (c) the example set is small enough to permit manual inspection by model evaluators; (d) strives to be comprehensively labeled with respect to the ImageNet label set; (e) in theory provides a subset that future models could achieve perfect accuracy on without worrying about underspeciﬁed class deﬁnitions.
Revised: We design the ImageNet-M 68-example subset as an additional split of the validation set that we believe has the following properties: (a) many top performing models trained in different ways make mistakes on this set; (b) the mistakes are all major mistakes as determined by expert-reviewers; (c) the example set is small enough to permit manual inspection by model evaluators; (d) strives to be comprehensively labeled with respect to the ImageNet label set; (e) in theory provides a subset that future models could achieve perfect accuracy on without worrying about underspeciﬁed class deﬁnitions.

Original: We anticipate stronger models will make correct unreviewed predictions on this slice, so we will endeavor to update the set of multi-labels as needed.
Revised: We anticipate stronger models will make correct unreviewed predictions on this slice, so we will endeavor to update the set of multi-labels as needed.

Original: Evaluation.
Revised: Evaluation.

Original: By construction, our ViT-3B model achieves 0% accuracy on ImageNet-M; the Instagram-pretrained model gets 9 of the 68 correct, while the Greedy Soups model gets 19 correct.
Revised: By construction, our ViT-3B model achieves 0% accuracy on ImageNet-M; the Instagram-pretrained model gets 9 of the 68 correct, while the Greedy Soups model gets 19 correct.

Original: The zero-shot model gets the best performance with 24 correct, even though the zero-shot model overall achieves lower multi-label accuracy (94.2%) than any of the other models.
Revised: The zero-shot model gets the best performance with 24 correct, even though the zero-shot model overall achieves lower multi-label accuracy (94.2%) than any of the other models.

Original: Because we use these four models to help choose the mistakes, these speciﬁc models comparatively will perform poorly on this benchmark.
Revised: Because we use these four models to help choose the mistakes, these speciﬁc models comparatively will perform poorly on this benchmark.

Original: Like with ImageNet-A
Revised: Like with ImageNet-A

Original: (Hendrycks et al., 2019), a dataset chosen using models may bias selection in such a way that future models may easily get very high accuracy on this subset, though we try to mitigate this effect by using multiple differently-trained models in our selection criteria.
Revised: (Hendrycks et al., 2019), a dataset chosen using models may bias selection in such a way that future models may easily get very high accuracy on this subset, though we try to mitigate this effect by using multiple differently-trained models in our selection criteria.

Original: How do models that were not used to select this dataset perform?
Revised: How do models that were not used to select this dataset perform?

Original: We evaluate the suite of 70 models from Shankar et al.
Revised: We evaluate the suite of 70 models from Shankar et al.

Original: (Shankar et al., 2020) on this dataset, in addition to four recent top models not directly used to help ﬁlter the ImageNet-M set: a
Revised: (Shankar et al., 2020) on this dataset, in addition to four recent top models not directly used to help ﬁlter the ImageNet-M set: a

Original: ViT-G/14 model (Zhai et al., 2021) (90.5% top-1), a BASIC model (Pham et al., 2021) ﬁne-tuned on ImageNet (90.7% top-1), an ALIGN model (Jia et al., 2021) ﬁne-tuned on
Revised: ViT-G/14 model (Zhai et al., 2021) (90.5% top-1), a BASIC model (Pham et al., 2021) ﬁne-tuned on ImageNet (90.7% top-1), an ALIGN model (Jia et al., 2021) ﬁne-tuned on

Original: ImageNet (88.1% top-1), and a CoCa model (Yu et al., 2022) ﬁne-tuned on ImageNet (91.0% top-1).
Revised: ImageNet (88.1% top-1), and a CoCa model (Yu et al., 2022) ﬁne-tuned on ImageNet (91.0% top-1).

Original: The plot shown here shows that most models as far back as AlexNet through ResNets get between 10-25 examples correct, but recent high accuracy models such as ViT-G/14, BASIC-FT, and CoCa-FT are starting to solve more of these ‘major’ mistakes: CoCa-FT gets 42 of the 68 examples correct.
Revised: The plot shown here shows that most models as far back as AlexNet through ResNets get between 10-25 examples correct, but recent high accuracy models such as ViT-G/14, BASIC-FT, and CoCa-FT are starting to solve more of these ‘major’ mistakes: CoCa-FT gets 42 of the 68 examples correct.

Original: We reviewed the mistakes made by these four models, which yielded a total of 5 novel predictions; 4 of them were veriﬁed to be wrong (and major), and 1 additional new valid prediction, for which we updated the label set accordingly.
Revised: We reviewed the mistakes made by these four models, which yielded a total of 5 novel predictions; 4 of them were veriﬁed to be wrong (and major), and 1 additional new valid prediction, for which we updated the label set accordingly.

Original: We note that future models may predict classes on these examples that are "minor" mistakes, since the deﬁnition of severity is linked to the (prediction, example) pair; should it be useful, the dataset slice can be augmented with a ‘minor_wrong_multi_label’ attribute to provide more ﬁne-grained signals.
Revised: We note that future models may predict classes on these examples that are "minor" mistakes, since the deﬁnition of severity is linked to the (prediction, example) pair; should it be useful, the dataset slice can be augmented with a ‘minor_wrong_multi_label’ attribute to provide more ﬁne-grained signals.

Original: Although we did not ﬁnd statistical evidence that stronger models solve major errors ﬁrst, we hope that progress on image classiﬁcation can be evaluated by whether improvements help reduce egregious mistakes before focusing on nebulous ones.
Revised: Although we did not ﬁnd statistical evidence that stronger models solve major errors ﬁrst, we hope that progress on image classiﬁcation can be evaluated by whether improvements help reduce egregious mistakes before focusing on nebulous ones.

Original: Overall, we encourage reporting on several slices, including ImageNet-M, to give a better sense of the strengths and weaknesses of various models.
Revised: Overall, we encourage reporting on several slices, including ImageNet-M, to give a better sense of the strengths and weaknesses of various models.

Original: For a more thorough discussion of the limitations of our analysis see
Revised: For a more thorough discussion of the limitations of our analysis see

Original: Appendix C.
Revised: Appendix C.

Original: A. Related Work
Revised: A. Related Work

Original: Multi-label annotations on ImageNet.
Revised: Multi-label annotations on ImageNet.

Original: As models continue to improve ImageNet top-1 accuracy, there has been an increased interest in evaluating ImageNet multi-label accuracy (Stock & Cissé, 2017; Beyer et al., 2020; Shankar et al., 2020;
Revised: As models continue to improve ImageNet top-1 accuracy, there has been an increased interest in evaluating ImageNet multi-label accuracy (Stock & Cissé, 2017; Beyer et al., 2020; Shankar et al., 2020;

Original: Yun et al., 2021;
Revised: Yun et al., 2021;

Original: Tsipras et al., 2020).
Revised: Tsipras et al., 2020).

Original: Stock et al.
Revised: Stock et al.

Original: (Stock & Cissé, 2017) use non-expert human studies and explanations (e.g., model criticisms) on predictions of a then-SOTA model on ImageNet, ﬁnding that machine accuracy is underestimated and advocating for multi-label evaluation.
Revised: (Stock & Cissé, 2017) use non-expert human studies and explanations (e.g., model criticisms) on predictions of a then-SOTA model on ImageNet, ﬁnding that machine accuracy is underestimated and advocating for multi-label evaluation.

Original: Beyer et.
Revised: Beyer et.

Original: al.
Revised: al.

Original: (Beyer et al., 2020) introduce a set of Reassessed Labels (ReaL) for the ImageNet validation set containing multi-label annotations.
Revised: (Beyer et al., 2020) introduce a set of Reassessed Labels (ReaL) for the ImageNet validation set containing multi-label annotations.

Original: The researchers ﬁrst collected proposal labels from model predictions using a testbed of 19 models, and then, in order to reduce the number of predictions to review, they narrowed the set of models down to the 6 models that had the highest precision and recall above 97% on a set of 256 images reviewed by 5 experts.
Revised: The researchers ﬁrst collected proposal labels from model predictions using a testbed of 19 models, and then, in order to reduce the number of predictions to review, they narrowed the set of models down to the 6 models that had the highest precision and recall above 97% on a set of 256 images reviewed by 5 experts.

Original: The top predictions from these 6 models were then reviewed by human annotators from a crowdsourcing platform.
Revised: The top predictions from these 6 models were then reviewed by human annotators from a crowdsourcing platform.

Original: In a similar vein, Shankar et.
Revised: In a similar vein, Shankar et.

Original: al.
Revised: al.

Original: (Shankar et al., 2020) provide a multi-label annotation dataset for 20,000 of the 50,
Revised: (Shankar et al., 2020) provide a multi-label annotation dataset for 20,000 of the 50,

Original: ImageNet validation images and ﬁnd that roughly 20% of images have more than one valid label.
Revised: ImageNet validation images and ﬁnd that roughly 20% of images have more than one valid label.

Original: To generate the annotations, the researchers ﬁrst collected predictions from a testbed of 70 models for each image and then reviewed the unique model predictions using a panel of three human experts.
Revised: To generate the annotations, the researchers ﬁrst collected predictions from a testbed of 70 models for each image and then reviewed the unique model predictions using a panel of three human experts.

Original: Additionally, human accuracy was evaluated on a subset of 1,000 images and a panel of 5 human experts reviewed human and model predictions on this subset.
Revised: Additionally, human accuracy was evaluated on a subset of 1,000 images and a panel of 5 human experts reviewed human and model predictions on this subset.

Original: Tsipras et al.
Revised: Tsipras et al.

Original: (Tsipras et al., 2020) also ﬁnd that 20% of images in the validation set contain objects from multiple classes, and identify sources of ambiguous label classes.
Revised: (Tsipras et al., 2020) also ﬁnd that 20% of images in the validation set contain objects from multiple classes, and identify sources of ambiguous label classes.

Original: Hooker et.
Revised: Hooker et.

Original: al (Hooker et al., 2019) use human evaluations to label a subset of examples from ImageNet, ﬁnding that they contained multiple labels 40–60% of the time.
Revised: al (Hooker et al., 2019) use human evaluations to label a subset of examples from ImageNet, ﬁnding that they contained multiple labels 40–60% of the time.

Original: More recently,
Revised: More recently,

Original: Yun et al.
Revised: Yun et al.

Original: (Yun et al., 2021) obtained pixel-level multi-label ground truths for the ImageNet training set using a machine annotator, and found that training a model with the dense annotations leads to small improvements in both top-1 and multi-label ImageNet accuracy.
Revised: (Yun et al., 2021) obtained pixel-level multi-label ground truths for the ImageNet training set using a machine annotator, and found that training a model with the dense annotations leads to small improvements in both top-1 and multi-label ImageNet accuracy.

Original: Our paper focuses on (and adds to) the multi-label evaluation of ImageNet using expert labelers, updating the dataset to collapse classes that are overlapping or subset relationships to better capture the remaining real errors.
Revised: Our paper focuses on (and adds to) the multi-label evaluation of ImageNet using expert labelers, updating the dataset to collapse classes that are overlapping or subset relationships to better capture the remaining real errors.

Original: We adopt the labeling methodology of Shankar et al.
Revised: We adopt the labeling methodology of Shankar et al.

Original: (Shankar et al., 2020)
Revised: (Shankar et al., 2020)

Original: and use a panel of human experts to determine the validity of novel predictions.
Revised: and use a panel of human experts to determine the validity of novel predictions.

Original: In contrast to prior work, we re-visit all remaining model mistakes using human expert review, and categorize them by type and severity.
Revised: In contrast to prior work, we re-visit all remaining model mistakes using human expert review, and categorize them by type and severity.

Original: ImageNet Label Error.
Revised: ImageNet Label Error.

Original: Multi-label annotation datasets for
Revised: Multi-label annotation datasets for

Original: ImageNet (including our updated annotations) identify a set of images that have no correct ground truth label (i.e. label errors).
Revised: ImageNet (including our updated annotations) identify a set of images that have no correct ground truth label (i.e. label errors).

Original: Van Horn et.
Revised: Van Horn et.

Original: al (Van Horn et al., 2015) used experts from the Cornell Lab of Ornithology to estimate that at least 4% of the bird images are misclassiﬁed, and more recently
Revised: al (Van Horn et al., 2015) used experts from the Cornell Lab of Ornithology to estimate that at least 4% of the bird images are misclassiﬁed, and more recently

Original: Northcutt et.
Revised: Northcutt et.

Original: al (Northcutt et al., 2021b) used MTURK workers to review algorithmically identiﬁed potential errors and found a label error rate of 5.83% in ImageNet.
Revised: al (Northcutt et al., 2021b) used MTURK workers to review algorithmically identiﬁed potential errors and found a label error rate of 5.83% in ImageNet.

Original: Lee et.
Revised: Lee et.

Original: al (Lee et al., 2017) sample 400 ImageNet mistakes and perform a categorization, similarly ﬁnding signiﬁcant label error and label ambiguity.
Revised: al (Lee et al., 2017) sample 400 ImageNet mistakes and perform a categorization, similarly ﬁnding signiﬁcant label error and label ambiguity.

Original: Mistake analysis.
Revised: Mistake analysis.

Original: Recent work has sought to understand or analyze model mistakes on ImageNet, mainly focusing on similarities or differences between mistake sets of independently trained classiﬁers.
Revised: Recent work has sought to understand or analyze model mistakes on ImageNet, mainly focusing on similarities or differences between mistake sets of independently trained classiﬁers.

Original: For example, Mania et al.
Revised: For example, Mania et al.

Original: (Mania et al., 2019) found that predictions of different models on
Revised: (Mania et al., 2019) found that predictions of different models on

Original: ImageNet are more similar than one would expect if the models were making mistakes independently.
Revised: ImageNet are more similar than one would expect if the models were making mistakes independently.

Original: Geirhos et al.
Revised: Geirhos et al.

Original: (Geirhos et al., 2020) studied a 16-class ImageNet classiﬁcation task and similarly found that CNNs make remarkably consistent mistakes with one another but CNNs and humans have an error consistency that is only slightly above what can be expected from chance.
Revised: (Geirhos et al., 2020) studied a 16-class ImageNet classiﬁcation task and similarly found that CNNs make remarkably consistent mistakes with one another but CNNs and humans have an error consistency that is only slightly above what can be expected from chance.

Original: In follow up work, Mania et al.
Revised: In follow up work, Mania et al.

Original: (Mania & Sra, 2020) studied the dominance probabilities for pairs of ImageNet models, which capture the probability that a higher accuracy model will make a mistake on a particular image that a lower accuracy model correctly classiﬁes.
Revised: (Mania & Sra, 2020) studied the dominance probabilities for pairs of ImageNet models, which capture the probability that a higher accuracy model will make a mistake on a particular image that a lower accuracy model correctly classiﬁes.

Original: Their empirical analysis of dominance probabilities on ImageNet implies that the mistakes of higher accuracy models are typically subsets of the mistakes of lower accuracy models.
Revised: Their empirical analysis of dominance probabilities on ImageNet implies that the mistakes of higher accuracy models are typically subsets of the mistakes of lower accuracy models.

Original: However, Gontijo-Lopes et al.
Revised: However, Gontijo-Lopes et al.

Original: (Gontijo-Lopes et al., 2021) and Andreassen et al.
Revised: (Gontijo-Lopes et al., 2021) and Andreassen et al.

Original: (Andreassen et al., 2021) found that training on larger and more diverse datasets as well as zero-shot evaluation can lead to models with more mistake diversity, which in turn can be used to build more accurate ensembles.
Revised: (Andreassen et al., 2021) found that training on larger and more diverse datasets as well as zero-shot evaluation can lead to models with more mistake diversity, which in turn can be used to build more accurate ensembles.

Original: Similarly, Nguyen et al.
Revised: Similarly, Nguyen et al.

Original: (Nguyen et al., 2021) found systematic differences in the errors between wide and deep ResNets on ImageNet.
Revised: (Nguyen et al., 2021) found systematic differences in the errors between wide and deep ResNets on ImageNet.

Original: Ciregan et al.
Revised: Ciregan et al.

Original: (Ciregan et al., 2012) demonstrated that powerful models on MNIST allowed for remaining error analysis in the single-label context, highlighting the ambiguity in many of those remaining errors.
Revised: (Ciregan et al., 2012) demonstrated that powerful models on MNIST allowed for remaining error analysis in the single-label context, highlighting the ambiguity in many of those remaining errors.

Original: In contrast to this prior work, we take a step towards fully calibrating SOTA model evaluations by exhaustively and visually reviewing every remaining model mistake using manual expert review in the multi-label context ; we believe the remaining errors identiﬁed to be legitimately incorrect and tempered by severity and category ratings that we hope prove useful to future evaluations.
Revised: In contrast to this prior work, we take a step towards fully calibrating SOTA model evaluations by exhaustively and visually reviewing every remaining model mistake using manual expert review in the multi-label context ; we believe the remaining errors identiﬁed to be legitimately incorrect and tempered by severity and category ratings that we hope prove useful to future evaluations.

Original: B. Methods B.1. Panel Review
Revised: B. Methods B.1. Panel Review

Original: To exhaustively and accurately assess every remaining mistake, we formed a panel of ﬁve reviewers and followed a process similar to (Shankar et al., 2020) to evaluate the predictions made by this model on the 676 mistakes — we avoided using non-expert crowd-source platforms speciﬁcally because the remaining mistakes are often difﬁcult to assess by non-expert annotators (Tsipras et al., 2020; Beyer et al., 2020).
Revised: To exhaustively and accurately assess every remaining mistake, we formed a panel of ﬁve reviewers and followed a process similar to (Shankar et al., 2020) to evaluate the predictions made by this model on the 676 mistakes — we avoided using non-expert crowd-source platforms speciﬁcally because the remaining mistakes are often difﬁcult to assess by non-expert annotators (Tsipras et al., 2020; Beyer et al., 2020).

Original: For every mistake, the panel determined: (1)
Revised: For every mistake, the panel determined: (1)

Original: Did the model make a mistake?
Revised: Did the model make a mistake?

Original: (2) Was the original ground truth annotation correct?
Revised: (2) Was the original ground truth annotation correct?

Original: (3) If the model made a mistake, what is the category, type, and severity of the mistake?
Revised: (3) If the model made a mistake, what is the category, type, and severity of the mistake?

Original: The imagenet2012_multilabel dataset contains a ﬁeld for every image indicating which classes a large previous suite of models predicted that were determined to be incorrect (wrong_multi_labels).
Revised: The imagenet2012_multilabel dataset contains a ﬁeld for every image indicating which classes a large previous suite of models predicted that were determined to be incorrect (wrong_multi_labels).

Original: Of these 676 initial mistakes, 221 were novel: they were not reviewed in the original multi-label annotation process since none of the models evaluated made the same prediction.
Revised: Of these 676 initial mistakes, 221 were novel: they were not reviewed in the original multi-label annotation process since none of the models evaluated made the same prediction.

Original: Each member of the panel reviewed all 221 novel mistakes.
Revised: Each member of the panel reviewed all 221 novel mistakes.

Original: Similar to (Shankar et al., 2020), we built a review tool that allowed each panelist to see a) the predicted class, b) the predicted top softmax score, c) the set of ground-truth labels, d) the set of previously incorrect labels, and e) the image.
Revised: Similar to (Shankar et al., 2020), we built a review tool that allowed each panelist to see a) the predicted class, b) the predicted top softmax score, c) the set of ground-truth labels, d) the set of previously incorrect labels, and e) the image.

Original: We also employed the labeling guide produced by the authors of (Shankar et al., 2020) when investigating the deﬁnition of a class, and a tool to iterate through the images of every ImageNet validation example for that class, using the validation images to help deﬁne the class boundaries (rather than the gloss or lay deﬁnition of the class).
Revised: We also employed the labeling guide produced by the authors of (Shankar et al., 2020) when investigating the deﬁnition of a class, and a tool to iterate through the images of every ImageNet validation example for that class, using the validation images to help deﬁne the class boundaries (rather than the gloss or lay deﬁnition of the class).

Original: See
Revised: See

Original: Appendix F for screenshots of the review tools.
Revised: Appendix F for screenshots of the review tools.

Original: In addition, we collapsed a small number of classes for which previous work has identiﬁed exhibited extreme overlap (Northcutt et al., 2021a), such as ‘missile’ and ‘projectile missile’.
Revised: In addition, we collapsed a small number of classes for which previous work has identiﬁed exhibited extreme overlap (Northcutt et al., 2021a), such as ‘missile’ and ‘projectile missile’.

Original: In
Revised: In

Original: Appendix G we provide these new collapsed class mappings.
Revised: Appendix G we provide these new collapsed class mappings.

Original: We also used Google Image Search to help provide context to some assessments; in one interesting but not isolated case, a prediction of a taxi cab (with no obvious taxi cab indicators beyond yellow color) was present in the image; we determined the prediction to be correctly a taxi cab and not just a standard vehicle by identifying a landmark bridge in the background in order to localize the city, and a subsequent image search for taxis in that city yielded the images of the same taxi model and license plate design, validating the model’s actually correct prediction.
Revised: We also used Google Image Search to help provide context to some assessments; in one interesting but not isolated case, a prediction of a taxi cab (with no obvious taxi cab indicators beyond yellow color) was present in the image; we determined the prediction to be correctly a taxi cab and not just a standard vehicle by identifying a landmark bridge in the background in order to localize the city, and a subsequent image search for taxis in that city yielded the images of the same taxi model and license plate design, validating the model’s actually correct prediction.

Original: Each panelist rated whether these novel mistakes had a mislabeled ground truth, or whether the prediction should be added to the set of correct, unclear, or wrong multi-labels.
Revised: Each panelist rated whether these novel mistakes had a mislabeled ground truth, or whether the prediction should be added to the set of correct, unclear, or wrong multi-labels.

Original: As a group, we reviewed any image where there was no unanimous agreement, allowing those in the minority to make their case and change minds, or highlight potential oversights.
Revised: As a group, we reviewed any image where there was no unanimous agreement, allowing those in the minority to make their case and change minds, or highlight potential oversights.

Original: The use of a panel and discussion was important: in a non-trivial number of cases, a single panelist found unique evidence for or against a prediction that no other panelist saw that led to a different outcome.
Revised: The use of a panel and discussion was important: in a non-trivial number of cases, a single panelist found unique evidence for or against a prediction that no other panelist saw that led to a different outcome.

Original: After locking in ﬁnal votes, we took the majority assessment, or used ‘unclear’ for a tie.
Revised: After locking in ﬁnal votes, we took the majority assessment, or used ‘unclear’ for a tie.

Original: After this re-assessment, 140 of the novel predictions were deemed correct (or the ground truth deemed incorrect), leaving 536 remaining mistakes to assess.
Revised: After this re-assessment, 140 of the novel predictions were deemed correct (or the ground truth deemed incorrect), leaving 536 remaining mistakes to assess.

Original: B.2. Mistake severity and category
Revised: B.2. Mistake severity and category

Original: The remaining mistakes then comprised images that had either previously been deemed wrong by the panel in Shankar et al.
Revised: The remaining mistakes then comprised images that had either previously been deemed wrong by the panel in Shankar et al.

Original: (Shankar et al., 2020), or deemed wrong by the current panel.
Revised: (Shankar et al., 2020), or deemed wrong by the current panel.

Original: We then began a review of the mistakes for the category and severity of mistakes.
Revised: We then began a review of the mistakes for the category and severity of mistakes.

Original: During this second phase review, we re-reviewed all images in detail, potentially overturning decisions made by the previous panel, where they missed the presence of an object that was in an example or the decision was inconsistent with the labeling guide or validation examples we relied on.
Revised: During this second phase review, we re-reviewed all images in detail, potentially overturning decisions made by the previous panel, where they missed the presence of an object that was in an example or the decision was inconsistent with the labeling guide or validation examples we relied on.

Original: Severity: We assessed each mistake’s severity with the assumption that not all mistakes are equal: some mistakes are extremely borderline, particularly because the ImageNet class deﬁnitions are imprecise or because the image itself provides ambiguous or incomplete information.
Revised: Severity: We assessed each mistake’s severity with the assumption that not all mistakes are equal: some mistakes are extremely borderline, particularly because the ImageNet class deﬁnitions are imprecise or because the image itself provides ambiguous or incomplete information.

Original: For any remaining mistake, we broke down the severity levels into major and minor mistakes.
Revised: For any remaining mistake, we broke down the severity levels into major and minor mistakes.

Original: A major mistake is a model prediction that a human who understands the class deﬁnitions would ﬁnd obviously incorrect.
Revised: A major mistake is a model prediction that a human who understands the class deﬁnitions would ﬁnd obviously incorrect.

Original: For example Figure 1(a) shows an example image where the prediction is jigsaw puzzle but the label is dough; although the pieces are somewhat jigsaw-puzzle-shaped, untrained humans are more likely to classify this as dough than jigsaw puzzle.
Revised: For example Figure 1(a) shows an example image where the prediction is jigsaw puzzle but the label is dough; although the pieces are somewhat jigsaw-puzzle-shaped, untrained humans are more likely to classify this as dough than jigsaw puzzle.

Original: A minor mistake is a model prediction that a human who understands the class deﬁnitions would probably ﬁnd to be incorrect, but in a more subtle way than a major mistake.
Revised: A minor mistake is a model prediction that a human who understands the class deﬁnitions would probably ﬁnd to be incorrect, but in a more subtle way than a major mistake.

Original: Some minor mistakes are so subtle that even expert-trained humans might debate their correctness.
Revised: Some minor mistakes are so subtle that even expert-trained humans might debate their correctness.

Original: We recognize that these severities are subject to the inﬂuences of the worldview of the panelists (and the web) and should be judged accordingly (see Section C).
Revised: We recognize that these severities are subject to the inﬂuences of the worldview of the panelists (and the web) and should be judged accordingly (see Section C).

Original: For transparency, we provide the panel assessments of the severity for all the mistakes, and in Figure 1 we provide some examples of the severities of various mistakes made by this model for the reader’s calibration.
Revised: For transparency, we provide the panel assessments of the severity for all the mistakes, and in Figure 1 we provide some examples of the severities of various mistakes made by this model for the reader’s calibration.

Original: In Appendix H we provide many additional examples of every severity and category.
Revised: In Appendix H we provide many additional examples of every severity and category.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Category:
Revised: Category:

Original: After reviewing many mistakes, we formulated four mistake types (Figure 2).
Revised: After reviewing many mistakes, we formulated four mistake types (Figure 2).

Original: (1) Fine-grained errors are where the model makes a mistake between two similar types of organisms or objects, one of which is a groundtruth label.
Revised: (1) Fine-grained errors are where the model makes a mistake between two similar types of organisms or objects, one of which is a groundtruth label.

Original: These mistakes often occur when the two confused classes are already similar (e.g. two dog breeds), or when either of them is very broad or ill-scoped (e.g. the “bake shop" class includes any baked good or bakery).
Revised: These mistakes often occur when the two confused classes are already similar (e g. two dog breeds), or when either of them is very broad or ill-scoped (e g. the “bake shop" class includes any baked good or bakery).

Original: (2) Fine-grained with out-of-vocabulary : there is an object in the image that is not in the ImageNet class hierarchy but is similar to a predicted class that is in ImageNet.
Revised: (2) Fine-grained with out-of-vocabulary : there is an object in the image that is not in the ImageNet class hierarchy but is similar to a predicted class that is in ImageNet.

Original: We separate out this category because it highlights the possible beneﬁts of training models to expect new classes to appear at test time and the importance of model uncertainty and calibration in the face of this ‘open world’.
Revised: We separate out this category because it highlights the possible beneﬁts of training models to expect new classes to appear at test time, and the importance of model uncertainty and calibration in the face of this ‘open world’.

Original: (3) Spurious correlations : Either (a) the predicted object is not plausibly in the image but surrounding cues may have been used, or (b) the predicted object does not match the groundtruth.
Revised: (3) Spurious correlations : Either (a) the predicted object is not plausibly in the image but surrounding cues may have been used, or (b) the predicted object does not match the groundtruth.

Original: In extreme cases, there is no clear indication of the predicted object; in more subtle cases, it is more clear the model is trying to predict the class of an object in the image
Revised: In extreme cases, there is no clear indication of the predicted object; in more subtle cases, it is more clear the model is trying to predict the class of an object in the image

Original: but the predicted object would not be considered either semantically or visually similar to the groundtruth class.
Revised: but the predicted object would not be considered either semantically or visually similar to the groundtruth class.

Original: (4) Non-prototypical labels : The predicted label is not present but the groundtruth object is a non-prototypical example of the class that bears resemblance to the predicted label.
Revised: (4) Non-prototypical labels : The predicted label is not present but the groundtruth object is a non-prototypical example of the class that bears resemblance to the predicted label.

Original: Non- prototypical mistakes are relatively rare, and capture the ‘long tail’ of examples for each class.
Revised: Non-prototypical mistakes are relatively rare, and capture the ‘long tail’ of examples for each class.

Original: These mistakes highlight the internal diversity of each class, and the difﬁculty of modeling the long within-class tail.
Revised: These mistakes highlight the internal diversity of each class and the difﬁculty of modeling the long within-class tail.

Original: C. Limitations of Analysis
Revised: C. Limitations of Analysis

Original: Much of our analysis on mistake categorization, severity, and data cleaning depends greatly on qualitative factors determined by the authors and experiment design, which we brieﬂy discuss here.
Revised: Much of our analysis on mistake categorization, severity, and data cleaning depends greatly on qualitative factors determined by the authors and experiment design, which we brieﬂy discuss here.

Original: Limitations due to mistake subset.
Revised: Limitations due to mistake subset.

Original: We only reviewed multi-label annotation examples where the ViT-3B model and Greedy Soups model we chose were incorrect; while the multi-label dataset itself has undergone review of mistakes from a suite of many models, we never review any validation image whose groundtruth might be wrong, if all models evaluated also make the incorrect groundtruth prediction.
Revised: We only reviewed multi-label annotation examples where the ViT-3B model and Greedy Soups model we chose were incorrect; while the multi-label dataset itself has undergone review of mistakes from a suite of many models, we never review any validation image whose groundtruth might be wrong, if all models evaluated also make the incorrect groundtruth prediction.

Original: We do partially review some of the mistakes made by a few other models, and build upon a dataset where mistakes made by many other models have already been reviewed, but most of the work here assesses these two speciﬁc model’s predictions.
Revised: We do partially review some of the mistakes made by a few other models, and build upon a dataset where mistakes made by many other models have already been reviewed, but most of the work here assesses these two speciﬁc model’s predictions.

Original: Deﬁnition of a mistake.
Revised: Deﬁnition of a mistake.

Original: We re-iterate that we used qualitative judgments to decide whether a prediction was a mistake, and if so, its severity and categorization.
Revised: We re-iterate that we used qualitative judgments to decide whether a prediction was a mistake, and if so, its severity and categorization.

Original: Our qualitative judgments are therefore based on a biased worldview (Torralba & Efros, 2011; Friedler et al., 2016; Baker; Paullada et al., 2021) comprising the ﬁve panelists; moreover, we are not world experts on dog or animal species, though we believe our assessments are at least as good or better than the original labeling process used for the validation set, given the research effort we made on each mistake.
Revised: Our qualitative judgments are therefore based on a biased worldview (Torralba & Efros, 2011; Friedler et al., 2016; Baker; Paullada et al., 2021) comprising the ﬁve panelists; moreover, we are not world experts on dog or animal species, though we believe our assessments are at least as good or better than the original labeling process used for the validation set, given the research effort we made on each mistake.

Original: As a mitigation against imbuing too biased a worldview on class deﬁnitions, we relied heavily on the validation data to deﬁne the boundaries of the class, even when those examples did not match up with our personal deﬁnitions of the class.
Revised: As a mitigation against imbuing too biased a worldview on class deﬁnitions, we relied heavily on the validation data to deﬁne the boundaries of the class, even when those examples did not match up with our personal deﬁnitions of the class.

Original: Moreover, updating (and evaluating) multi-labels potentially allows for different world-views to be expressed.
Revised: Moreover, updating (and evaluating) multi-labels potentially allows for different world-views to be expressed.

Original: D. Additional Results D.1. Out-of-distribution generalization
Revised: D. Additional Results D.1. Out-of-distribution generalization

Original: We evaluated the ViT-3B model on the ImageNetV2 multilabel subset which produced over 900 unreviewed errors.
Revised: We evaluated the ViT-3B model on the ImageNetV2 multilabel subset which produced over 900 unreviewed errors.

Original: To assess what aspects of our analysis generalize to other datasets, we sampled 100 of these errors using the same panel review system employed for ImageNetV1.
Revised: To assess what aspects of our analysis generalize to other datasets, we sampled 100 of these errors using the same panel review system employed for ImageNetV1.

Original: We discovered that 47 of the 100 ImageNetV2 predictions were either correct or had problematic labels, leaving 53 mistakes that we reviewed for category and severity.
Revised: We discovered that 47 of the 100 ImageNetV2 predictions were either correct or had problematic labels, leaving 53 mistakes that we reviewed for category and severity.

Original: For ImageNetV1, we had previously found that 44% (296/676) of mistakes were either problematic or correct and we found no statistically signiﬁcant difference between the two datasets in this regard ( χ 2 (1 , N = 776)
Revised: For ImageNetV1, we had previously found that 44% (296/676) of mistakes were either problematic or correct and we found no statistically signiﬁcant difference between the two datasets in this regard ( χ 2 (1 , N = 776)

Original: = 0 .
Revised: = 0 .

Original: 36 , p = .
Revised: 36 , p = .

Original: These proportions suggest that large models are frequently uncovering new correct multi-labels, suggesting that mistake analysis and label correction needs to be part of the lifecycle and maintenance of benchmark development of long-tailed errors to properly assess performance as a benchmark saturates.
Revised: These proportions suggest that large models are frequently uncovering new correct multi-labels, suggesting that mistake analysis and label correction needs to be part of the lifecycle and maintenance of benchmark development of long-tailed errors to properly assess performance as a benchmark saturates.

Original: Table 2 compares the category and severity breakdowns between the two datasets—overall the model is making similar types and severities of mistakes on both datasets.
Revised: Table 2 compares the category and severity breakdowns between the two datasets—overall the model is making similar types and severities of mistakes on both datasets.

Original: A chi-square test of independence shows that there is no signiﬁcant difference between either the mistake categorization χ 2 (3 , N = 434)
Revised: A chi-square test of independence shows that there is no signiﬁcant difference between either the mistake categorization χ 2 (3 , N = 434)

Original: = 0 .
Revised: = 0 .

Original: 97 , p = .
Revised: 97 , p = .

Original: or mistake severity χ 2 (1 , N = 434)
Revised: or mistake severity χ 2 (1 , N = 434)

Original: = 0 .
Revised: = 0 .

Original: 01 , p = .
Revised: 01 , p = .

Original: D.2. Analyzing class confusions.
Revised: D.2. Analyzing class confusions.

Original: Given that most failures were ﬁne-grained, we tried to identify any patterns present in the class confusions made by the model, but found no consistent pattern.
Revised: Given that most failures were ﬁne-grained, we tried to identify any patterns present in the class confusions made by the model, but found no consistent pattern.

Original: The inline ﬁgure shows the frequency of occurrence for confused class pairs.
Revised: The inline ﬁgure shows the frequency of occurrence for confused class pairs.

Original: The distribution is long-tailed in nature—the majority of class pairs occur exactly once or twice and only a handful of class pairs occur three or more times.
Revised: The distribution is long-tailed in nature—the majority of class pairs occur exactly once or twice and only a handful of class pairs occur three or more times.

Original: The long-tailed nature of the class confusions suggests that we will not be able to resolve a large fraction of model mistakes by focusing on cleaning up or adding additional data to only a small number of classes.
Revised: The long-tailed nature of the class confusions suggests that we will not be able to resolve a large fraction of model mistakes by focusing on cleaning up or adding additional data to only a small number of classes.

Original: D.3. Comparison to humans
Revised: D.3. Comparison to humans

Original: We compare the performance of the ViT-3B and
Revised: We compare the performance of the ViT-3B and

Original: Greedy
Revised: Greedy

Original: Soups models to the best human labeler from Shankar et al.
Revised: Soups models to the best human labeler from Shankar et al.

Original: (Shankar et al., 2020) 2 by evaluating on the subset of 1,000 ImageNet val images used to compute human accuracy in the prior work.
Revised: (Shankar et al., 2020) 2 by evaluating on the subset of 1,000 ImageNet val images used to compute human accuracy in the prior work.

Original: To fairly compare to the models, we re-compute the human accuracy scores using the original human predictions and our updated label set.
Revised: To fairly compare to the models, we re-compute the human accuracy scores using the original human predictions and our updated label set.

Original: Overall, the re-labeling did not signiﬁcantly change human accuracy; the best human labeler achieved 97.3% MLA on the original multi-labels and 97.4% MLA on the updated multi-labels.
Revised: Overall, the re-labeling did not signiﬁcantly change human accuracy; the best human labeler achieved 97.3% MLA on the original multi-labels and 97.4% MLA on the updated multi-labels.

Original: Table 3 compares the re-labeled MLA of the ViT-3B and
Revised: Table 3 compares the re-labeled MLA of the ViT-3B and

Original: Greedy Soup model to the best human for all ImageNet classes as well as the subset of ImageNet classes corresponding to objects and organisms.
Revised: Greedy Soup model to the best human for all ImageNet classes as well as the subset of ImageNet classes corresponding to objects and organisms.

Original: Similar to Shankar et al.
Revised: Similar to Shankar et al.

Original: (Shankar et al., 2020), we also ﬁnd that the performance of the models is more uniform across the object and organism classes, but humans do substantially better on the object classes than the organism classes.
Revised: (Shankar et al., 2020), we also ﬁnd that the performance of the models is more uniform across the object and organism classes, but humans do substantially better on the object classes than the organism classes.

Original: However, unlike prior work, current models outperform the best human when evaluated on all ImageNet classes (though humans still achieve slightly better performance on the object classes).
Revised: However, unlike prior work, current models outperform the best human when evaluated on all ImageNet classes (though humans still achieve slightly better performance on the object classes).

Original: D.4. Analyzing the Training Data
Revised: D.4. Analyzing the Training Data

Original: Finally, we investigate how much we can understand model validation set errors through the lens of the training data.
Revised: Finally, we investigate how much we can understand model validation set errors through the lens of the training data.

Original: To do so, we inspect the K = 10 nearest neighbor training images using JFT-pretrained (before ImageNet ﬁne tuning) embeddings for the ViT-3B model.
Revised: To do so, we inspect the K = 10 nearest neighbor training images using JFT-pretrained (before ImageNet ﬁne tuning) embeddings for the ViT-3B model.

Original: Doing this, we rediscover (originally documented in Sun et al.
Revised: Doing this, we rediscover (originally documented in Sun et al.

Original: (Sun et al., 2017) and Kolesnikov et al.
Revised: (Sun et al., 2017) and Kolesnikov et al.

Original: (Kolesnikov et al., 2020))
Revised: (Kolesnikov et al., 2020))

Original: that(1.59%)
Revised: that(1.59%)

Original: ImageNet validation images exist in the training set as exact duplicates .
Revised: ImageNet validation images exist in the training set as exact duplicates .

Original: Interestingly, we are the ﬁrst to notice that every single leaked image has a different label in the training set than the validation set .
Revised: Interestingly, we are the ﬁrst to notice that every single leaked image has a different label in the training set than the validation set .

Original: In Appendix J. we show removing these images has relatively little impact on the model’s performance, and detail a more pernicious leakage pattern of "near duplicates" in Appendix J.2 that is hard to fully quantify.
Revised: In Appendix J. we show removing these images has relatively little impact on the model’s performance, and detail a more pernicious leakage pattern of "near duplicates" in Appendix J.2 that is hard to fully quantify.

Original: Finally, we show how the training data sometimes explain spurious correlations in Appendix
Revised: Finally, we show how the training data sometimes explain spurious correlations in Appendix

Original: J.3.
Revised: J.3.

Original: E. ViT-3B model details
Revised: E. ViT-3B model details

Original: The ViT model we use in this work is based on a standard Vision Transformer (Dosovitskiy et al., 2021) model scaled to nearly 3 billion parameters, using a patch size of 14, 16 heads, 64 blocks, an MLP dimension of 8192 and a hidden dimension of 2048.
Revised: The ViT model we use in this work is based on a standard Vision Transformer (Dosovitskiy et al., 2021) model scaled to nearly 3 billion parameters, using a patch size of 14, 16 heads, 64 blocks, an MLP dimension of 8192 and a hidden dimension of 2048.

Original: The model is deﬁned and trained in Lingvo (Shen et al., 2019); we additionally employ GSPMD (Xu et al., 2021) for training.
Revised: The model is deﬁned and trained in Lingvo (Shen et al., 2019); we additionally employ GSPMD (Xu et al., 2021) for training.

Original: The model is pre-trained on JFT-3B (Sun et al., 2020) using training settings that optimize for performance on JFT-3B rather than for ﬁne-tuning on ImageNet; notably, we do not use the training recipe that helps few-shot transfer performance (Zhai et al., 2021).
Revised: The model is pre-trained on JFT-3B (Sun et al., 2020) using training settings that optimize for performance on JFT-3B rather than for ﬁne-tuning on ImageNet; notably, we do not use the training recipe that helps few-shot transfer performance (Zhai et al., 2021).

Original: For ﬁne-tuning on ImageNet, we use the
Revised: For ﬁne-tuning on ImageNet, we use the

Original: AdamW optimizer (beta1=0.9, beta2=0.999, epsilon=1e- [Figure] , weight_decay=0.3) with a cosine learning rate schedule (max learning rate of 3e-2, warmup of 2k steps, ﬁnal rate of 3e-4), a training batch size of 512, and ﬁne-tune for a total of 10 epochs.
Revised: AdamW optimizer (beta1=0.9, beta2=0.999, epsilon=1e8, weight_decay=0.3) with a cosine learning rate schedule (max learning rate of 3e-2, warmup of 2k steps, ﬁnal rate of 3e-4), a training batch size of 512, and ﬁne-tune for a total of 10 epochs.

Original: F. Review tools
Revised: F. Review tools

Original: We include screenshots of the reviewing tools we built to analyze model mistakes.
Revised: We include screenshots of the reviewing tools we built to analyze model mistakes.

Original: Figure 3 shows the UI for reviewing model predictions and Figure 4 shows the UI that displays the labeling guide and slide bar to browse images for a particular class.
Revised: Figure 3 shows the UI for reviewing model predictions and Figure 4 shows the UI that displays the labeling guide and slide bar to browse images for a particular class.

Original: [Table]
Revised: [Table]

Original: G. Collapsed mappings
Revised: G. Collapsed mappings

Original: We provide our collapsed class mappings that we employed unilaterally, based on determining that the classes exhibited signiﬁcant overlap based on the validation set images, or there was a strict superset relationship between two or more classes.
Revised: We provide our collapsed class mappings that we employed unilaterally, based on determining that the classes exhibited signiﬁcant overlap based on the validation set images, or there was a strict superset relationship between two or more classes.

Original: For example, a prediction of ‘eskimo dog’ for a ‘siberian husky’ label would be considered correct, whereas a prediction of ‘siberian husky’ for an ‘eskimo dog’ label might not.
Revised: For example, a prediction of ‘eskimo dog’ for a ‘siberian husky’ label would be considered correct, whereas a prediction of ‘siberian husky’ for an ‘eskimo dog’ label might not.

Original: All siberian huskies and malamutes are also eskimo dogs.
Revised: All siberian huskies and malamutes are also eskimo dogs.

Original: Sunglass and sunglasses are the same class (bidirectional).
Revised: Sunglass and sunglasses are the same class (bidirectional).

Original: Indian and African elephants are also tuskers.
Revised: Indian and African elephants are also tuskers.

Original: A coffee mug is also a cup.
Revised: A coffee mug is also a cup.

Original: Maillot and maillot, tanksuit are the same class (bidirectional).
Revised: Maillot and maillot, tanksuit are the same class (bidirectional).

Original: Missile and projectile missile are the same class (bidirectional).
Revised: Missile and projectile missile are the same class (bidirectional).

Original: Notebook computer and laptop are the same class (bidirectional).
Revised: Notebook computer and laptop are the same class (bidirectional).

Original: Monitor and screen are the same class (bidirectional).
Revised: Monitor and screen are the same class (bidirectional).

Original: A cassette player is also a tape player.
Revised: A cassette player is also a tape player.

Original: Weasel, polecats, black-footed ferrets, and minks are all the same class.
Revised: Weasel, polecats, black-footed ferrets, and minks are all the same class.

Original: All bathtubs are tubs, but not all tubs are bathtubs.
Revised: All bathtubs are tubs, but not all tubs are bathtubs.

Original: H. Mistake Examples by Severity
Revised: H. Mistake Examples by Severity

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: I. Mistake Examples by Category
Revised: I. Mistake Examples by Category

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: GT: chime Pred: maze
Revised: GT: chime Pred: maze

Original: Lack of context
Revised: Lack of context

Original: GT: street sign Pred: bakery
Revised: GT: street sign Pred: bakery

Original: Over-reliance on context
Revised: Over-reliance on context

Original: GT: dam, stone wall, valley Pred: cliff
Revised: GT: dam, stone wall, valley Pred: cliff

Original: Lack of context
Revised: Lack of context

Original: GT: dock, crane Pred: submarine
Revised: GT: dock, crane Pred: submarine

Original: Over-reliance on context
Revised: Over-reliance on context

Original: GT: hatchet, chain mail Pred: cuirass
Revised: GT: hatchet, chain mail Pred: cuirass

Original: Over-reliance on context
Revised: Over-reliance on context

Original: GT: candle, packet, combination lock, knot, desk
Revised: GT: candle, packet, combination lock, knot, desk

Original: Pred: carpenter’s kit Lack of context
Revised: Pred: carpenter’s kit Lack of context

Original: GT: doormat Pred: switch
Revised: GT: doormat Pred: switch

Original: Over-reliance on context
Revised: Over-reliance on context

Original: GT: lens cap, sock, theater curtain, screen, monitorPred:Band Aid Lack of context
Revised: GT: lens cap, sock, theater curtain, screen, monitorPred:Band Aid Lack of context

Original: GT: pop bottle, water bottle Pred: pot
Revised: GT: pop bottle, water bottle Pred: pot

Original: Over-reliance on context
Revised: Over-reliance on context

Original: GT: bonnet Pred: bookcase
Revised: GT: bonnet Pred: bookcase

Original: Lack of context
Revised: Lack of context

Original: GT: gondola, dock, lakeside Pred: boathouse
Revised: GT: gondola, dock, lakeside Pred: boathouse

Original: Over-reliance on context
Revised: Over-reliance on context

Original: GT: cradle, barn,
Revised: GT: cradle, barn,

Original: hay Pred: altar
Revised: hay Pred: altar

Original: Over-reliance on context
Revised: Over-reliance on context

Original: GT: paper [Figure] , washbasin Pred: dishwasher
Revised: GT: paper towel, washbasin Pred: dishwasher

Original: Over-reliance on context
Revised: Over-reliance on context

Original: GT: chain Pred: crane
Revised: GT: chain Pred: crane

Original: Lack of context
Revised: Lack of context

Original: GT: mouse, desk, monitor, screen Pred: desktop computer
Revised: GT: mouse, desk, monitor, screen Pred: desktop computer

Original: Over-reliance on context
Revised: Over-reliance on context

Original: GT: tobacco shop Pred: restaurant
Revised: GT: tobacco shop Pred: restaurant

Original: Over-reliance on context
Revised: Over-reliance on context

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: J. Analyzing the Training Data
Revised: J. Analyzing the Training Data

Original: In this section, we try to understand the model’s remaining mistakes by investigating the training data.
Revised: In this section, we try to understand the model’s remaining mistakes by investigating the training data.

Original: We do this through the lens of looking at the nearest training examples to the ViT-3B models remaining mistakes.
Revised: We do this through the lens of looking at the nearest training examples to the ViT-3B models remaining mistakes.

Original: To do this, we generate embeddings for every training and validation example using the pretrained ViT-3B models JFT checkpoint (before ﬁne-tuning on Imagenet), and for every error, we query the K=10 nearest neighbors using an exact nearest neighbor lookup.
Revised: To do this, we generate embeddings for every training and validation example using the pretrained ViT-3B models JFT checkpoint (before ﬁne-tuning on Imagenet), and for every error, we query the K=10 nearest neighbors using an exact nearest neighbor lookup.

Original: J.1. Validation Set Leakage.
Revised: J.1. Validation Set Leakage.

Original: One of the most interesting ﬁndings using nearest neighbors was rediscovering that 797 ImageNet validation images (1.594%) exist in the training set as exact duplicates (pixelwise L2 distance of 0), 34 of them more than once for a total of 831 duplicate training images.
Revised: One of the most interesting ﬁndings using nearest neighbors was rediscovering that 797 ImageNet validation images (1.594%) exist in the training set as exact duplicates (pixelwise L2 distance of 0), 34 of them more than once for a total of 831 duplicate training images.

Original: While this was previously documented in (Sun et al., 2017) and (Kolesnikov et al., 2020), we are the ﬁrst to notice that every single leaked sample has a different label in the train set than in the validation set, indicating the ImageNet authors did deduplicate within a class, but not across classes.
Revised: While this was previously documented in (Sun et al., 2017) and (Kolesnikov et al., 2020), we are the ﬁrst to notice that every single leaked sample has a different label in the train set than in the validation set, indicating the ImageNet authors did deduplicate within a class, but not across classes.

Original: Analyzing these duplicates we ﬁnd most of them represent challenging ﬁne grained image classes (e.g. two similar dog breeds), or images where multiple annotations are appropriate.
Revised: Analyzing these duplicates we ﬁnd most of them represent challenging ﬁne grained image classes (e.g. two similar dog breeds), or images where multiple annotations are appropriate.

Original: Additionally, in the Appendix J.2 we detail a second, harder to detect leak pattern we saw commonly in the training data with "near duplicates", images in the training set that are from the same photo-shoot or scene as a validation image, or are cropped or processed versions of a validation image.
Revised: Additionally, in the Appendix J.2 we detail a second, harder to detect leak pattern we saw commonly in the training data with "near duplicates", images in the training set that are from the same photo-shoot or scene as a validation image, or are cropped or processed versions of a validation image.

Original: This phenomenon, with similar discovery methodology, has also been observed on CIFAR (Barz & Denzler, 2020).
Revised: This phenomenon, with similar discovery methodology, has also been observed on CIFAR (Barz & Denzler, 2020).

Original: To understand the impact of this validation set leakage, we remove all the exact duplicates from the training set and retrain both a ResNet50 from scratch, and our JFT pretrained
Revised: To understand the impact of this validation set leakage, we remove all the exact duplicates from the training set and retrain both a ResNet50 from scratch, and our JFT pretrained

Original: ViT-3B. Results are shown in Table 4.
Revised: ViT-3B. Results are shown in Table 4.

Original: Unintuitively, when we remove all the leaked validation images from the training set and retrain, we see both Top1 Accuracy and Multi-label
Revised: Unintuitively, when we remove all the leaked validation images from the training set and retrain, we see both Top1 Accuracy and Multi-label

Original: Accuracy (MLA) actually stay the same or decrease overall, despite all leaked training images having different labels than their leaked validation image counterpart.
Revised: Accuracy (MLA) actually stay the same or decrease overall, despite all leaked training images having different labels than their leaked validation image counterpart.

Original: MLA accuracy both before and after deduplication are high, which leads us to believe that the training labels may be correct under multi-label evaluation.
Revised: MLA accuracy both before and after deduplication are high, which leads us to believe that the training labels may be correct under multi-label evaluation.

Original: To verify this, we ﬁnd 320 of the leaked validation images were in our 20k multi-label validation set, corresponding to 331 training images.
Revised: To verify this, we ﬁnd 320 of the leaked validation images were in our 20k multi-label validation set, corresponding to 331 training images.

Original: Using these multi-labels, we ﬁnd that 219/331 (66.2%) of these training images labels would have been correct under multilabel evaluation.
Revised: Using these multi-labels, we ﬁnd that 219/331 (66.2%) of these training images labels would have been correct under multilabel evaluation.

Original: Nevertheless, we do see an increase in both Top1 and MLA accuracies on the leaked subset of the data.
Revised: Nevertheless, we do see an increase in both Top1 and MLA accuracies on the leaked subset of the data.

Original: Finally, it seems the ViT-3B model is less sensitive to leaked validation images, which may be a function of the ﬁne tuning recipe we used (which was exactly the same as the ﬁne-tuning recipe on the origianl non-deduped data).
Revised: Finally, it seems the ViT-3B model is less sensitive to leaked validation images, which may be a function of the ﬁne tuning recipe we used (which was exactly the same as the ﬁne-tuning recipe on the origianl non-deduped data).

Original: J.2. Near Duplicates
Revised: J.2. Near Duplicates

Original: In addition to the 831 training examples that are exact duplicates of validation examples (all with different labels), there is a large collection of "near duplicates" in the Imagenet train set.
Revised: In addition to the 831 training examples that are exact duplicates of validation examples (all with different labels), there is a large collection of "near duplicates" in the Imagenet train set.

Original: "Near duplicates" are images that are either crops, augmentations, or resizes of validation images, or more unexpectedly, images from the same scene or photo shoot.
Revised: "Near duplicates" are images that are either crops, augmentations, or resizes of validation images, or more unexpectedly, images from the same scene or photo shoot.

Original: These images are often visually different but semantically the same, and as a result are much harder to detect with traditional embedding distance thresholding based deduplication.
Revised: These images are often visually different but semantically the same, and as a result are much harder to detect with traditional embedding distance thresholding based deduplication.

Original: Nevertheless, these can also leak test set information, introduce label noise (if the labels are different than the validation set), or if many training examples are from the same scene, reduce the effective dataset size of that class.
Revised: Nevertheless, these can also leak test set information, introduce label noise (if the labels are different than the validation set), or if many training examples are from the same scene, reduce the effective dataset size of that class.

Original: While we cannot provide an estimate of the prevalence of this problem, we ﬁnd it often while analyzing the K-Nearest
Revised: While we cannot provide an estimate of the prevalence of this problem, we ﬁnd it often while analyzing the K-Nearest

Original: Neighbors (in JFT embedding space) of validation images.
Revised: Neighbors (in JFT embedding space) of validation images.

Original: We show some of these examples in Figures 14 and 15.
Revised: We show some of these examples in Figures 14 and 15.

Original: While we just show dog and object examples here, we ﬁnd this also happens with reasonable frequency for human related classes (especially activity related classes like soccer, basketball, parallel bars, etc).
Revised: While we just show dog and object examples here, we ﬁnd this also happens with reasonable frequency for human related classes (especially activity related classes like soccer, basketball, parallel bars, etc).

Original: The existence of these duplicates raises an interesting question: How close can training examples be to your validation set before it becomes problematic?
Revised: The existence of these duplicates raises an interesting question: How close can training examples be to your validation set before it becomes problematic?

Original: J.3. Neighbors of Spurious Correlations
Revised: J.3. Neighbors of Spurious Correlations

Original: While ﬁne grained errors and OOV mistakes are often somewhat intuitive to a human, spurious correlations are harder to understand.
Revised: While ﬁne grained errors and OOV mistakes are often somewhat intuitive to a human, spurious correlations are harder to understand.

Original: To try to understand them in more detail, we look at the neighbors of several of the ViT-3B models major spurious correlations.
Revised: To try to understand them in more detail, we look at the neighbors of several of the ViT-3B models major spurious correlations.

Original: In Figure 16 we show two such examples, but ﬁnd this to be relatively common.
Revised: In Figure 16 we show two such examples, but ﬁnd this to be relatively common.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]