Original: I MPROVING AND A SSESSING A NOMALY D ETECTORS FOR L ARGE -S CALE S ETTINGS
Revised: I MPROVING AND A SSESSING A NOMALY D ETECTORS FOR L ARGE -S CALE S ETTINGS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Detecting out-of-distribution examples is important for safety-critical machine learning applications such as detecting novel biological phenomena and self-driving cars.
Revised: Detecting out-of-distribution examples is important for safety-critical machine learning applications such as detecting novel biological phenomena and self-driving cars.

Original: However, existing research mainly focuses on simple small-scale settings.
Revised: However, existing research mainly focuses on simple small-scale settings.

Original: To set the stage for more realistic out-of-distribution detection, we depart from small-scale settings and explore large-scale multiclass and multi-label settings with high-resolution images and thousands of classes.
Revised: To set the stage for more realistic out-of-distribution detection, we depart from small-scale settings and explore large-scale multiclass and multi-label settings with high-resolution images and thousands of classes.

Original: To make future work in real-world settings possible, we create new benchmarks for three large-scale settings.
Revised: To make future work in real-world settings possible, we create new benchmarks for three large-scale settings.

Original: To test
Revised: To test

Original: ImageNet multiclass anomaly detectors, we introduce a new dataset of anomalous species.
Revised: ImageNet multiclass anomaly detectors, we introduce a new dataset of anomalous species.

Original: We leverage ImageNet-22K to evaluate PASCAL VOC and COCO multilabel anomaly detectors.
Revised: We leverage ImageNet-21K to evaluate PASCAL VOC and COCO multilabel anomaly detectors.

Original: Third, we introduce a new benchmark for anomaly segmentation by introducing a segmentation benchmark with road anomalies.
Revised: Third, we introduce a new benchmark for anomaly segmentation by introducing a segmentation benchmark with road anomalies.

Original: We conduct extensive experiments in these more realistic settings for out-of-distribution detection and ﬁnd that a surprisingly simple detector based on the maximum logit outperforms prior methods in all the large-scale
Revised: We conduct extensive experiments in these more realistic settings for out-of-distribution detection and ﬁnd that a surprisingly simple detector based on the maximum logit outperforms prior methods in all the large-scale

Original: multi-class, multi-
Revised: multi-class, multi-

Original: label, and segmentation tasks, establishing a simple new baseline for future work.
Revised: label, and segmentation tasks, establishing a simple new baseline for future work.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Out-of-distribution  detection is a valuable tool for creating safe and robust applications of machine learning in the real world.
Revised: Out-of-distribution (OOD) detection is a valuable tool for developing safe and reliable machine learning (ML) systems.

Original: Detecting anomalous inputs allows systems to initiate a conservative fallback policy or defer to human judgment.
Revised: Detecting anomalous inputs allows systems to initiate a conservative fallback policy or defer to human judgment.

Revised: As an important component of ML Safety (Hendrycks et al., 2021),

Original: This isespecially important in safety-critical applications, such as self-driving cars and medical diagnosis.
Revised: OOD detection is important for safety-critical applications such as self-driving cars and detecting novel microorganisms.

Original: Accordingly, research on out-of-distribution detection has a rich history spanning several decades (Schölkopf et al., 1999; Breunig et al., 2000; Emmott et al., 2015).
Revised: Accordingly, research on out-of-distribution detection has a rich history spanning several decades (Schölkopf et al., 1999; Breunig et al., 2000; Emmott et al., 2015).

Original: Recent work leverages deep neural representations for out-of-distribution detection in complex domains, such as image data (Hendrycks & Gimpel, 2017; Lee et al., 2018a; Mohseni et al., 2020; Hendrycks et al., 2019b).
Revised: Recent work leverages deep neural representations for out-of-distribution detection in complex domains, such as image data (Hendrycks & Gimpel, 2017; Lee et al., 2018a; Mohseni et al., 2020; Hendrycks et al., 2019b).

Original: However, these works still primarily use small-scale datasets with low-resolution images and few classes.
Revised: However, these works still primarily use small-scale datasets with low-resolution images and few classes.

Original: As the community moves towards more realistic, large-scale settings, strong baselines and high-quality benchmarks are imperative for future progress.
Revised: As the community moves towards more realistic, large-scale settings, strong baselines and high-quality benchmarks are imperative for future progress.

Original: Large-scale datasets such as ImageNet (Deng et al., 2009) and Places365 (Zhou et al., 2017) present unique challenges not seen in small-scale settings, such as a plethora of ﬁne-grained object classes.
Revised: Large-scale datasets such as ImageNet (Deng et al., 2009) and Places365 (Zhou et al., 2017) present unique challenges not seen in small-scale settings, such as a plethora of ﬁne-grained object classes.

Original: We demonstrate that the maximum softmax probability (MSP) detector, a state-of-the-art method for small-scale problems, does not scale well to these challenging conditions.
Revised: We demonstrate that the maximum softmax probability (MSP) detector, a state-of-the-art method for small-scale problems, does not scale well to these challenging conditions.

Original: Through extensive experiments, we identify a detector based on the maximum logit (MaxLogit) that greatly outperforms the MSP and other strong baselines in large-scale multi-class anomaly segmentation.
Revised: Through extensive experiments, we identify a detector based on the maximum logit (MaxLogit) that greatly outperforms the MSP and other strong baselines in large-scale multi-class anomaly segmentation.

Original: To facilitate further research in this setting, we also collect a new out-of-distribution test dataset suitable for models trained on highly diverse datasets.
Revised: To facilitate further research in this setting, we also collect a new out-of-distribution test dataset suitable for models trained on highly diverse datasets.

Original: Shown in Figure 2, our Species dataset contains diverse, anomalous species that do not overlap with datasets such as ImageNet-22K, enabling a stricter evaluation methodology for ImageNet-22K models.
Revised: Shown in Figure 2, our Species dataset contains diverse, anomalous species that do not overlap ImageNet-21K which has approximately twenty two thousand classes. Species avoids data leakage and enables a stricter evaluation methodology for ImageNet-21K models.

Original: Using Species to conduct more controlled experiments without train-test overlap, we ﬁnd that contrary to prior claims (Fort et al.,  2021), Vision 
Revised: Using Species to conduct more controlled experiments without train-test overlap, we ﬁnd that contrary to prior claims (Fort et al., 2021; Koner et al., 2021), Vision Transformers

Original: Transformers (Dosovitskiy et al., 2021a) pre-trained on ImageNet-22K are not substantially better at out-of-distribution detection.
Revised: (Dosovitskiy et al., 2021a) pre-trained on ImageNet-21K are not substantially better at out-of-distribution detection.

Original: Moreover, in the common real-world case of multi-label data, the MSP detector cannot naturally be applied in the ﬁrst place, as it requires softmax probabilities.
Revised: Moreover, in the common real-world case of multi-label data, the MSP detector cannot naturally be applied in the ﬁrst place, as it requires softmax probabilities.

Original: To enable research into the multi-label setting for anomaly detection, we contribute a multi-label experimental setup and explore variousmethods on large-scale multi-label datasets. We ﬁnd that the MaxLogit detector from our investigation
Revised: To enable research into the multi-label setting for anomaly detection, we contribute a multi-label experimental setup and explore various

Original: [Figure]
Revised: [Figure]

Original:  into the large-scale multi-class setting generalizes well to multi-label data and again outperforms all other baselines.
Revised: methods on large-scale multi-label datasets. We ﬁnd that the MaxLogit detector from our investigation into the large-scale multi-class setting generalizes well to multi-label data and again outperforms all other baselines.

Original: In addition to focusing on small-scale datasets, most existing benchmarks for anomaly detection treat entire images as anomalies.
Revised: In addition to focusing on small-scale datasets, most existing benchmarks for anomaly detection treat entire images as anomalies.

Original: In practice, an image could be anomalous in localized regions while being in-distribution elsewhere.
Revised: In practice, an image could be anomalous in localized regions while being in-distribution elsewhere.

Original: Knowing which regions of an image are anomalous could allow for safer handling of unfamiliar objects in the case of self-driving cars.
Revised: Knowing which regions of an image are anomalous could allow for safer handling of unfamiliar objects in the case of self-driving cars.

Original: Creating a benchmark for this task is difﬁcult, though, as simply cutting and pasting anomalous objects into images introduces various unnatural giveaway cues such as edge effects, mismatched orientation, and lighting, all of which trivialize the task of anomaly segmentation (Blum et al., 2019).
Revised: Creating a benchmark for this task is difﬁcult, though, as simply cutting and pasting anomalous objects into images introduces various unnatural giveaway cues such as edge effects, mismatched orientation, and lighting, all of which trivialize the task of anomaly segmentation (Blum et al., 2019).

Original: To overcome these issues, we utilize a simulated driving environment to create the novel StreetHazards dataset for anomaly segmentation.
Revised: To overcome these issues, we utilize a simulated driving environment to create the novel StreetHazards dataset for anomaly segmentation.

Original: Using the Unreal Engine and the open-source CARLA simulation environment (Dosovitskiy et al., 2017), we insert a diverse array of foreign objects into driving scenes and re-render the scenes with these novel objects.
Revised: Using the Unreal Engine and the open-source CARLA simulation environment (Dosovitskiy et al., 2017), we insert a diverse array of foreign objects into driving scenes and re-render the scenes with these novel objects.

Original: This enables integration of the foreign objects into their surrounding context with correct lighting and orientation, sidestepping giveaway cues.
Revised: This enables integration of the foreign objects into their surrounding context with correct lighting and orientation, sidestepping giveaway cues.

Original: To complement the StreetHazards dataset, we convert the BDD100K semantic segmentation dataset (Yu et al., 2018) into an anomaly segmentation dataset, which we call BDD-Anomaly.
Revised: To complement the StreetHazards dataset, we convert the BDD100K semantic segmentation dataset (Yu et al., 2018) into an anomaly segmentation dataset, which we call BDD-Anomaly.

Original: By leveraging the large scale of BDD100K, we reserve infrequent object classes to be anomalies.
Revised: By leveraging the large scale of BDD100K, we reserve infrequent object classes to be anomalies.

Original: We combine this dataset with StreetHazards to form the Combined Anomalous Object Segmentation (CAOS) benchmark.
Revised: We combine this dataset with StreetHazards to form the Combined Anomalous Object Segmentation (CAOS) benchmark.

Original: The CAOS benchmark improves over previous evaluations for anomaly segmentation in driving scenes by evaluating detectors on realistic and diverse anomalies.
Revised: The CAOS benchmark improves over previous evaluations for anomaly segmentation in driving scenes by evaluating detectors on realistic and diverse anomalies.

Original: We evaluate several baselines on the CAOS benchmark and discuss problems with porting existing approaches from earlier formulations of out-of-distribution detection.
Revised: We evaluate several baselines on the CAOS benchmark and discuss problems with porting existing approaches from earlier formulations of out-of-distribution detection.

Original: Despite its simplicity, we ﬁnd that the MaxLogit detector outperforms all baselines on Species, our multi-class benchmark, and CAOS.
Revised: Despite its simplicity, we ﬁnd that the MaxLogit detector outperforms all baselines on Species, our multi-class benchmark, and CAOS.

Original: In each of these three settings, we discuss why MaxLogit provides superior performance, and we show that these gains are hidden if one looks at small-scale problems alone.
Revised: In each of these three settings, we discuss why MaxLogit provides superior performance, and we show that these gains are hidden if one looks at small-scale problems alone.

Original: The code for our experiments and the Species and CAOS datasets are available at [anonymized] .
Revised: The code for our experiments and the Species and CAOS datasets are available at [anonymized] .

Original: Our new baseline combined with Species and CAOS benchmarks pave the way for future research on large-scale out-of-distribution detection.
Revised: Our new baseline combined with Species and CAOS benchmarks pave the way for future research on large-scale out-of-distribution detection.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Multi-Class Out-of-Distribution Detection.
Revised: Multi-Class Out-of-Distribution Detection.

Original: A recent line of work leverages deep neural representations from multi-class classiﬁers to perform out-of-distribution (OOD) detection on highdimensional data, including images, text, and speech data.
Revised: A recent line of work leverages deep neural representations from multi-class classiﬁers to perform out-of-distribution (OOD) detection on highdimensional data, including images, text, and speech data.

Original: Hendrycks & Gimpel (2017) formulate the task and propose the simple baseline of using the maximum softmax probability of the classiﬁer on an input to gauge whether the input is out-of-distribution.
Revised: Hendrycks & Gimpel (2017) formulate the task and propose the simple baseline of using the maximum softmax probability of the classiﬁer on an input to gauge whether the input is out-of-distribution.

Original: In particular, they formulate the task as distinguishing between examples from an in-distribution dataset and various OOD datasets.
Revised: In particular, they formulate the task as distinguishing between examples from an in-distribution dataset and various OOD datasets.

Original: Importantly, entire images are treated as out-of-distribution.
Revised: Importantly, entire images are treated as out-of-distribution.

Original: Continuing this line of work, Lee et al.
Revised: Continuing this line of work, Lee et al.

Original: (2018a) propose to improve the neural representation of the classiﬁer to better separate OOD examples.
Revised: (2018a) propose to improve the neural representation of the classiﬁer to better separate OOD examples.

Original: They use generative adversarial networks to produce neardistribution examples and induce uniform posteriors on these synthetic OOD examples.
Revised: They use generative adversarial networks to produce neardistribution examples and induce uniform posteriors on these synthetic OOD examples.

Original: Hendrycks et al.
Revised: Hendrycks et al.

Original: (2019b) observe that outliers are often easy to obtain in large quantity from diverse, realistic datasets and demonstrate that OOD detectors trained on these outliers generalize to unseen classes of anomalies.
Revised: (2019b) observe that outliers are often easy to obtain in large quantity from diverse, realistic datasets and demonstrate that OOD detectors trained on these outliers generalize to unseen classes of anomalies.

Original: Other work investigates improving the anomaly detectors themselves given a ﬁxed classiﬁer (DeVries & Taylor, 2018; Liang et al., 2018).
Revised: Other work investigates improving the anomaly detectors themselves given a ﬁxed classiﬁer (DeVries & Taylor, 2018; Liang et al., 2018).

Original: However, as Hendrycks et al.
Revised: However, as Hendrycks et al.

Original: (2019b) observe, many of these works tune hyperparameters on a particular type of anomaly that is also seen at test time, so their evaluation setting is more lenient.
Revised: (2019b) observe, many of these works tune hyperparameters on a particular type of anomaly that is also seen at test time, so their evaluation setting is more lenient.

Original: In this paper, all anomalies seen at test time come from entirely unseen categories and are not tuned on in any way.
Revised: In this paper, all anomalies seen at test time come from entirely unseen categories and are not tuned on in any way.

Original: Hence, we do not compare to techniques such as ODIN (Liang et al., 2018).
Revised: Hence, we do not compare to techniques such as ODIN (Liang et al., 2018).

Original: Additionally, in a point of departure from prior work, we focus primarily on large-scale images and datasets with many classes.
Revised: Additionally, in a point of departure from prior work, we focus primarily on large-scale images and datasets with many classes.

Original: Recent work has suggested that stronger representations from Vision Transformers pre-trained on
Revised: Recent work has suggested that stronger representations from Vision Transformers pre-trained on

Original: ImageNet-22K can make out-of-distribution detection trivial (Fort et al., 2021; Koner et al., 2021).
Revised: ImageNet-21K can make out-of-distribution detection trivial (Fort et al., 2021; Koner et al., 2021).

Original: They evaluate models on detecting CIFAR-10 when ﬁne-tuned on CIFAR-100 or vice versa, using models pretrained on ImageNet-22K.
Revised: They evaluate models on detecting CIFAR-10 when ﬁne-tuned on CIFAR-100 or vice versa, using models pretrained on ImageNet-21K.

Original: However, over 1,000 classes in ImageNet-22K overlap with
Revised: However, over 1,000 classes in ImageNet-21K overlap with

Original: CIFAR-10, so it is still unclear how Vision Transformers perform at detecting entirely unseen OOD categories.
Revised: CIFAR-10, so it is still unclear how Vision Transformers perform at detecting entirely unseen OOD categories.

Original: We create a new OOD test dataset of anomalous species to investigate how well Vision
Revised: We create a new OOD test dataset of anomalous species to investigate how well Vision

Original: Transformers perform in controlled OOD detection settings without data leakage and overlap.
Revised: Transformers perform in controlled OOD detection settings without data leakage and overlap.

Original: We ﬁnd that Vision Transformers pre-trained on ImageNet-22K are far from solving OOD detection in large-scale settings.
Revised: We ﬁnd that Vision Transformers pre-trained on ImageNet-21K are far from solving OOD detection in large-scale settings.

Original: [Table]
Revised: [Table]

Original: Anomaly Segmentation.
Revised: Anomaly Segmentation.

Original: Several prior works explore segmenting anomalous image regions.
Revised: Several prior works explore segmenting anomalous image regions.

Original: One line of work uses the WildDash dataset (Zendel et al., 2018), which contains numerous annotated driving scenes in conditions such as snow, fog, and rain.
Revised: One line of work uses the WildDash dataset (Zendel et al., 2018), which contains numerous annotated driving scenes in conditions such as snow, fog, and rain.

Original: The WildDash test set contains ﬁfteen “negative images” from different domains for which the goal is to mark the entire image as out-ofdistribution.
Revised: The WildDash test set contains ﬁfteen “negative images” from different domains for which the goal is to mark the entire image as out-ofdistribution.

Original: Thus, while the task is segmentation, the anomalies do not exist as objects within an otherwise in-distribution scene.
Revised: Thus, while the task is segmentation, the anomalies do not exist as objects within an otherwise in-distribution scene.

Original: This setting is similar to that explored by Hendrycks & Gimpel (2017), in which whole images from other datasets serve as out-of-distribution examples.
Revised: This setting is similar to that explored by Hendrycks & Gimpel (2017), in which whole images from other datasets serve as out-of-distribution examples.

Original: To approach anomaly segmentation on WildDash, Krešo et al.
Revised: To approach anomaly segmentation on WildDash, Krešo et al.

Original: (2018) train on multiple semantic segmentation domains and treat regions of images from the WildDash driving dataset as out-ofdistribution if they are segmented as regions from different domains, i.e. indoor classes.
Revised: (2018) train on multiple semantic segmentation domains and treat regions of images from the WildDash driving dataset as out-ofdistribution if they are segmented as regions from different domains, i.e. indoor classes.

Original: Bevandi´c et al.
Revised: Bevandi´c et al.

Original: (2018) use ILSVRC 2012 images and train their network to segment the entirety of these images as out-of-distribution.
Revised: (2018) use ILSVRC 2012 images and train their network to segment the entirety of these images as out-of-distribution.

Original: In medical anomaly segmentation and product fault detection, anomalies are regions of otherwise in-distribution images.
Revised: In medical anomaly segmentation and product fault detection, anomalies are regions of otherwise in-distribution images.

Original: Baur et al.
Revised: Baur et al.

Original: (2019) segment anomalous regions in brain MRIs using pixelwise reconstruction loss.
Revised: (2019) segment anomalous regions in brain MRIs using pixelwise reconstruction loss.

Original: Similarly, Haselmann et al.
Revised: Similarly, Haselmann et al.

Original: (2018) perform product fault detection using pixel-wise reconstruction loss and introduce an expansive dataset for segmentation of product faults.
Revised: (2018) perform product fault detection using pixel-wise reconstruction loss and introduce an expansive dataset for segmentation of product faults.

Original: In these relatively simple domains, reconstruction-based approaches work well.
Revised: In these relatively simple domains, reconstruction-based approaches work well.

Original: In contrast to medical anomaly segmentation and fault detection, we consider complex images from street scenes.
Revised: In contrast to medical anomaly segmentation and fault detection, we consider complex images from street scenes.

Original: These images have high variability in scene layout and lighting, and hence are less amenable to reconstruction-based techniques.
Revised: These images have high variability in scene layout and lighting, and hence are less amenable to reconstruction-based techniques.

Original: The two works closest to our own are the Lost and Found (Pinggera et al., 2016) and Fishyscapes (Blum et al., 2019) datasets.
Revised: The two works closest to our own are the Lost and Found (Pinggera et al., 2016) and Fishyscapes (Blum et al., 2019) datasets.

Original: The Lost and Found dataset consists of real images in a driving environment with small road hazards.
Revised: The Lost and Found dataset consists of real images in a driving environment with small road hazards.

Original: The images were collected to mirror the Cityscapes dataset (Cordts et al., 2016) but are only collected from one city and so have less diversity.
Revised: The images were collected to mirror the Cityscapes dataset (Cordts et al., 2016) but are only collected from one city and so have less diversity.

Original: The dataset contains 35 unique anomalous objects, and methods are allowed to train on many of these.
Revised: The dataset contains 35 unique anomalous objects, and methods are allowed to train on many of these.

Original: For
Revised: For

Original: Lost and Found, only nine unique objects are truly unseen at test time.
Revised: Lost and Found, only nine unique objects are truly unseen at test time.

Original: Crucially, this is a different evaluation setting from our own, where anomalous objects are not revealed at training time, so their dataset is not directly comparable.
Revised: Crucially, this is a different evaluation setting from our own, where anomalous objects are not revealed at training time, so their dataset is not directly comparable.

Original: Nevertheless, the BDD-Anomaly dataset ﬁlls several gaps in Lost and Found.
Revised: Nevertheless, the BDD-Anomaly dataset ﬁlls several gaps in Lost and Found.

Original: First, the images are more diverse, because they are sourced from a more recent and comprehensive semantic segmentation dataset.
Revised: First, the images are more diverse, because they are sourced from a more recent and comprehensive semantic segmentation dataset.

Original: Second, the anomalies are not restricted to small, sparse road hazards.
Revised: Second, the anomalies are not restricted to small, sparse road hazards.

Original: Concretely, anomalous regions in Lost and Found take up 0.11% of the image on average, whereas anomalous regions in the BDD-Anomaly dataset are larger and ﬁll 0.83% of the image on average.
Revised: Concretely, anomalous regions in Lost and Found take up 0.11% of the image on average, whereas anomalous regions in the BDD-Anomaly dataset are larger and ﬁll 0.83% of the image on average.

Original: Finally, although the BDD-Anomaly dataset treats three categories as anomalous, compared to Lost and Found it has far more unique anomalous objects.
Revised: Finally, although the BDD-Anomaly dataset treats three categories as anomalous, compared to Lost and Found it has far more unique anomalous objects.

Original: The Fishyscapes benchmark for anomaly segmentation consists of cut-and-paste anomalies from out-of-distribution domains.
Revised: The Fishyscapes benchmark for anomaly segmentation consists of cut-and-paste anomalies from out-of-distribution domains.

Original: This is problematic, because the anomalies stand out as clearly unnatural in context.
Revised: This is problematic, because the anomalies stand out as clearly unnatural in context.

Original: For instance, the orientation of anomalous objects is unnatural, and the lighting of the cut-and-paste patch differs from the lighting in the original image, providing an unnatural cue to anomaly detectors that would not exist for real anomalies.
Revised: For instance, the orientation of anomalous objects is unnatural, and the lighting of the cut-and-paste patch differs from the lighting in the original image, providing an unnatural cue to anomaly detectors that would not exist for real anomalies.

Original: Figure 7 shows an example of these inconsistencies.
Revised: Figure 7 shows an example of these inconsistencies.

Original: Techniques for detecting image manipulation (Zhou et al., 2018; Johnson & Farid, 2005) are competent at detecting artiﬁcial image elements of this kind.
Revised: Techniques for detecting image manipulation (Zhou et al., 2018; Johnson & Farid, 2005) are competent at detecting artiﬁcial image elements of this kind.

Original: Our StreetHazards dataset overcomes these issues by leveraging a simulated driving environment to naturally insert anomalous 3D models into a scene rather than overlaying 2D images.
Revised: Our StreetHazards dataset overcomes these issues by leveraging a simulated driving environment to naturally insert anomalous 3D models into a scene rather than overlaying 2D images.

Original: These anomalies are integrated into the scene with proper lighting and orientation, mimicking real-world anomalies and making them signiﬁcantly more difﬁcult to detect.
Revised: These anomalies are integrated into the scene with proper lighting and orientation, mimicking real-world anomalies and making them signiﬁcantly more difﬁcult to detect.

Original: A i r p l a n e
Revised: A i r p l a n e

Original: A u t o m o b il e B i r d C a t D ee r D
Revised: A u t o m o b il e B i r d C a t D ee r D

Original: o g F r o g
Revised: o g F r o g

Original: H
Revised: H

Original: o r s e S h
Revised: o r s e S h

Original: i p T
Revised: i p T

Original: r u c k
Revised: r u c k

Original: C
Revised: C

Original: I F
Revised: I F

Original: A
Revised: A

Original: RC l a ss i f
Revised: RC l a ss i f

Original: i e r
Revised: i e r

Original: C o n f i d e n c e ( % )
Revised: C o n f i d e n c e ( % )

Original: T e n c h
Revised: T e n c h

Original: G o l d f i s h
Revised: G o l d f i s h

Original: I r i s h
Revised: I r i s h

Original: T e rr i e r
Revised: T e rr i e r

Original: N o r f o l k T e
Revised: N o r f o l k T e

Original: rr i e r
Revised: rr i e r

Original: N o r w
Revised: N o r w

Original: i
Revised: i

Original: c h T e
Revised: c h T e

Original: rr i e r
Revised: rr i e r

Original: C a n d l e
Revised: C a n d l e

Original: I m a g e N e t
Revised: I m a g e N e t

Original: C l a ss i f
Revised: C l a ss i f

Original: i e r
Revised: i e r

Original: C o n f i d e n c e ( % ) 
Revised: C o n f i d e n c e ( % ) Scale)

Original:  M ULTI -C LASS P REDICTION FOR OOD D ETECTION
Revised:  M ULTI -C LASS P REDICTION FOR OOD D ETECTION

Original: Problem with existing baselines.
Revised: Problem with existing baselines.

Original: Existing baselines for anomaly detection can work well in smallscale settings.
Revised: Existing baselines for anomaly detection can work well in small-scale settings.

Original: However, in more realistic settings image classiﬁcation networks are often tasked with distinguishing hundreds or thousands of classes, possibly with subtle differences.
Revised: However, in more realistic settings image classiﬁcation networks are often tasked with distinguishing hundreds or thousands of classes, possibly with subtle differences.

Original: This is problematic for  maximum softmax probability (MSP) baseline (Hendrycks & Gimpel, 2017), which uses the negative maximum softmax probability as the anomaly score, or − max k  p ( y = k | x )  .
Revised: This is problematic for the maximum softmax probability (MSP) baseline (Hendrycks & Gimpel, 2017), which uses the negative maximum softmax probability as the anomaly score, or − max k exp f ( x ) k / (cid:80) i exp f ( x ) i = − max k ˆ p ( y = k | x ) , where f ( x ) is the unnormalized logits of classiﬁer f on input x .

Original: Classiﬁers tend to have higher conﬁdence on in-distribution examples than out-of-distribution examples, enabling OOD detection.
Revised: Classiﬁers tend to have higher conﬁdence on in-distribution examples than out-of-distribution examples, enabling OOD detection.

Original: Assuming single-model evaluation and no access to other anomalies or test-time adaptation, the MSP attains state-of-the-art anomaly detection performance in small-scale settings.
Revised: Assuming single-model evaluation and no access to other anomalies or test-time adaptation, the MSP attains state-of-the-art anomaly detection performance in small-scale settings.

Original: However, we show that the MSP is problematic for realistic in-distribution datasets with many classes, such as ImageNet and Places365 (Zhou et al., 2017).
Revised: However, we show that the MSP is problematic for realistic in-distribution datasets with many classes, such as ImageNet and Places365 (Zhou et al., 2017).

Original: Probability mass can be dispersed among visually similar classes, as shown in Figure 3.
Revised: Probability mass can be dispersed among visually similar classes, as shown in Figure 3.

Original: Consequently, a classiﬁer may produce a low conﬁdence prediction for an in-distribution image, not because the image is unfamiliar, but because the object’s exact class is difﬁcult to determine.
Revised: Consequently, a classiﬁer may produce a low conﬁdence prediction for an in-distribution image, not because the image is unfamiliar, but because the object’s exact class is difﬁcult to determine.

Original: To circumvent this problem, we propose using the negative of the maximum unnormalized logit for an anomaly score, which we call MaxLogit.
Revised: To circumvent this problem, we propose using the negative of the maximum unnormalized logit for an anomaly score − max k f ( x ) k , which we call MaxLogit.

Original: Since the logits are unnormalized, they are not affected by the number of classes and can serve as a better baseline for large-scale out-of-distribution detection.
Revised: Since the logits are unnormalized, they are not affected by the number of classes and can serve as a better baseline for large-scale out-of-distribution detection.

Original: The Species Out-Of-Distribution Dataset.
Revised: The Species Out-Of-Distribution Dataset.

Original: To enable controlled experiments and high-quality evaluations of anomaly detectors in large-scale settings, we create the Species dataset, a new outof-distribution test dataset that has no overlapping classes with ImageNet-22K.
Revised: To enable controlled experiments and high-quality evaluations of anomaly detectors in large-scale settings, we create the Species dataset, a new outof-distribution test dataset that has no overlapping classes with ImageNet-21K.

Original: The Species dataset is comprised of images scraped from the iNaturalist website and contains hundreds of anomalous species grouped into seven high-level categories: Plants, Microorganisms, Amphibians, Protozoa,
Revised: The Species dataset is comprised of images scraped from the iNaturalist website and contains hundreds of anomalous species grouped into seven high-level categories: Plants, Microorganisms, Amphibians, Protozoa,

Original: Fungi, Arachnids, and Insects.
Revised: Fungi, Arachnids, and Insects.

Original: Example images from the Species dataset are in Figure 2.
Revised: Example images from the Species dataset are in Figure 2.

Original: Setup.
Revised: Setup.

Original: To evaluate the MSP baseline out-of-distribution detector and the MaxLogit detector, we use ImageNet-22K as the in-distribution dataset D in .
Revised: To evaluate the MSP baseline out-of-distribution detector and the MaxLogit detector, we use ImageNet-21K as the in-distribution dataset D in .

Original: To obtain representations for anomaly detection, we use models trained on ImageNet-21K-P, a cleaned version of ImageNet-22K with a train/val split (Ridnik et al., 2021a).
Revised: To obtain representations for anomaly detection, we use models trained on ImageNet-21K-P, a cleaned version of ImageNet-21K with a train/val split (Ridnik et al., 2021a).

Original: We evaluate a TResNet-M, ViT-B-16, and Mixer-B-16 (Ridnik et al., 2021b;
Revised: We evaluate a TResNet-M, ViT-B-16, and Mixer-B-16 (Ridnik et al., 2021b;

Original: Dosovitskiy et al., 2021b;
Revised: Dosovitskiy et al., 2021b;

Original: Tolstikhin et al., 2021), and the validation split is used for obtaining in-distribution scores.
Revised: Tolstikhin et al., 2021), and the validation split is used for obtaining in-distribution scores.

Original: For out-of-distribution test datasets D out , we use categories from the Species dataset, all of which are unseen during training.
Revised: For out-of-distribution test datasets D out , we use categories from the Species dataset, all of which are unseen during training.

Original: Results for these experiments are in Table 2.
Revised: Results for these experiments are in Table 2.

Original: Wealso use ImageNet-1K and Places365 as in-distribution datasets D in , for which we use pretrained
Revised: We

Original: [Table]
Revised: [Table]

Original: D in
Revised: D in

Original: D testout
Revised: D testout

Original: [Table]
Revised: [Table]

Original: I m a g e N e tK - P
Revised: I m a g e N e tK - P

Original: Plants
Revised: Plants

Revised: [Table]

Revised: Microorganisms

Original: [Table]
Revised: [Table]

Revised: also use ImageNet-1K and Places365 as in-distribution datasets D in , for which we use pretrained

Original: ResNet-50 models and use several out-of-distribution test datasets D 
Revised: ResNet-50 models and use several out-of-distribution test datasets D out .

Revised: Full results with ImageNet and Places365 as in-distribution are in the Appendix.

Original: Metrics.
Revised: Metrics.

Original: To evaluate out-of-distribution detectors in large-scale settings, we use three standard metrics of detection performance: area under the ROC curve (AUROC), false positive rate at 95% recall (FPR95) [Table] and area under the precision-recall curve (AUPR).
Revised: To evaluate out-of-distribution detectors in large-scale settings, we use three standard metrics of detection performance: area under the ROC curve (AUROC), false positive rate at 95% recall (FPR95), and area under the precision-recall curve (AUPR).

Original: The AUROC and AUPR are important metrics, because they give a holistic measure of performance when the cutoff for detecting anomalies is not a priori obvious or when we want to represent the performance of a detection method across several different cutoffs.
Revised: The AUROC and AUPR are important metrics, because they give a holistic measure of performance when the cutoff for detecting anomalies is not a priori obvious or when we want to represent the performance of a detection method across several different cutoffs.

Original: The AUROC can be thought of as the probability that an anomalous example is given a higher score than an ordinary example.
Revised: The AUROC can be thought of as the probability that an anomalous example is given a higher score than an ordinary example.

Original: Thus, a higher score is better, and an uninformative detector has a AUROC of 50%.
Revised: Thus, a higher score is better, and an uninformative detector has a AUROC of 50%.

Original: AUPR provides a metric more attuned to class imbalances, which is relevant in anomaly and failure detection, when the number of anomalies or failures may be relatively small.
Revised: AUPR provides a metric more attuned to class imbalances, which is relevant in anomaly and failure detection, when the number of anomalies or failures may be relatively small.

Original: Last, the FPR metric consists of measuring the false positive rate at 95%.
Revised: Last, the FPR metric consists of measuring the false positive rate at 95%.

Original: Since these measures are correlated, we occasionally solely present the AUROC for brevity and to preserve space.
Revised: Since these measures are correlated, we occasionally solely present the AUROC for brevity and to preserve space.

Original: Results.
Revised: Results.

Original: Results on Species are shown in Table 2.
Revised: Results on Species are shown in Table 2.

Original: Results with ImageNet-1K and Places365 as in-distribution datasets are in Table 1.
Revised: Results with ImageNet-1K and Places365 as in-distribution datasets are in Table 1.

Original: We ﬁnd that the proposed MaxLogit method outperforms the maximum softmax probability baseline on all out-of-distribution test datasets D out .
Revised: We ﬁnd that the proposed MaxLogit method outperforms the maximum softmax probability baseline on all out-of-distribution test datasets D out .

Original: This holds true for all three models trained on ImageNet-22K.
Revised: This holds true for all three models trained on ImageNet-21K.

Original: The MSP baseline is not much better than random and is has similar performance for all three model classes.
Revised: The MSP baseline is not much better than random and is has similar performance for all three model classes.

Original: This suggests that contrary to recent claims, (Fort et al., 2021) simply scaling up Vision Transformers does not make OOD detection trivial.
Revised: This suggests that contrary to recent claims, (Fort et al., 2021) simply scaling up Vision Transformers does not make OOD detection trivial.

Original:  M ULTI -L ABEL P REDICTION FOR OOD D ETECTION
Revised:  M ULTI -L ABEL P REDICTION FOR OOD D ETECTION

Original: Current work on out-of-distribution detection primarily considers multi-class or unsupervised settings.
Revised: Current work on out-of-distribution detection primarily considers multi-class or unsupervised settings.

Original: Yet as classiﬁers become more useful in realistic settings, the multi-label formulation becomes increasingly natural.
Revised: Yet as classiﬁers become more useful in realistic settings, the multi-label formulation becomes increasingly natural.

Original: To investigate out-of-distribution detection in multi-label settings, we provide a baseline and evaluation setup.
Revised: To investigate out-of-distribution detection in multi-label settings, we provide a baseline and evaluation setup.

Original: Setup.
Revised: Setup.

Original: For multi-label classiﬁcation we use PASCAL VOC (Everingham et al., 2009) and MSCOCO (Lin et al., 2014) as in-distribution data.
Revised: For multi-label classiﬁcation we use PASCAL VOC (Everingham et al., 2009) and MSCOCO (Lin et al., 2014) as in-distribution data.

Original: To evaluate anomaly detectors for these in-distribution datasets, we use 20 out-of-distribution classes from ImageNet-22K.
Revised: To evaluate anomaly detectors for these in-distribution datasets, we use 20 out-of-distribution classes from ImageNet-21K.

Original: These classes have no overlap with ImageNet-1K, PASCAL VOC, or MS-COCO.
Revised: These classes have no overlap with ImageNet-1K, PASCAL VOC, or MS-COCO.

Original: The 20 classes are chosen not to overlap with
Revised: The 20 classes are chosen not to overlap with

Original: ImageNet-1K since the multi-label classiﬁers models are pre-trained on ImageNet-1K. We list the class WordNet IDs in the Appendix.
Revised: ImageNet-1K since the multi-label classiﬁers models are pre-trained on ImageNet-1K. We list the class WordNet IDs in the Appendix.

Original: Methods.
Revised: Methods.

Original: For our experiments, we use a ResNet-101 backbone architecture pre-trained on
Revised: For our experiments, we use a ResNet-101 backbone architecture pre-trained on

Original: ImageNet-1K. We replace the ﬁnal layer with 2 fully connected layers and apply the logistic sigmoid function for multi-label prediction.
Revised: ImageNet-1K. We replace the ﬁnal layer with 2 fully connected layers and apply the logistic sigmoid function for multi-label prediction.

Original: During training we freeze the batch normalization parameters due to an insufﬁcient number of images for proper mean and variance estimation.
Revised: During training we freeze the batch normalization parameters due to an insufﬁcient number of images for proper mean and variance estimation.

Original: We train each model for 50 epochs using the Adam optimizer (Kingma & Ba, 2014) with hyperparameter values 10 − 4 and10 − 5 for β 1 and β 2 respectively. For data augmentation we use standard resizing, random crops, and
Revised: We train each model for 50 epochs using the Adam optimizer (Kingma & Ba, 2014) with hyperparameter values 10 − 4 and

Original: [Table]
Revised: [Table]

Original:  random ﬂips to obtain images of size 256 × 256 × 3 .
Revised: 10 − 5 for β 1 and β 2 respectively. For data augmentation we use standard resizing, random crops, and random ﬂips to obtain images of size 256 × 256 × 3 .

Original: As a result of this training procedure, the mAP of the ResNet-101 on PASCAL VOC is 89.11% and 72.0% for MS-COCO.
Revised: As a result of this training procedure, the mAP of the ResNet-101 on PASCAL VOC is 89.11% and 72.0% for MS-COCO.

Original: As there has been little work on out-of-distribution detection in multilabel settings, we include comparisons to classic anomaly detectors for general settings.
Revised: As there has been little work on out-of-distribution detection in multilabel settings, we include comparisons to classic anomaly detectors for general settings.

Original: Isolation Forest, denoted by iForest, works by randomly partitioning the space into half spaces to form a decision tree.
Revised: Isolation Forest, denoted by iForest, works by randomly partitioning the space into half spaces to form a decision tree.

Original: The score is determined by how close a point is to the root of the tree.
Revised: The score is determined by how close a point is to the root of the tree.

Original: The local outlier factor (LOF) (Breunig et al., 2000) computes a local density ratio between every element and its neighbors.
Revised: The local outlier factor (LOF) (Breunig et al., 2000) computes a local density ratio between every element and its neighbors.

Original: We set the number of neighbors as 20 .
Revised: We set the number of neighbors as 20 .

Original: iForest and LOF are both computed on features from the penultimate layer of the networks.
Revised: iForest and LOF are both computed on features from the penultimate layer of the networks.

Original: MSP denotes a natural extension of the maximum softmax probability detector in the multi-label setting, obtained by taking the sigmoid of each output score f
Revised: MSP denotes a natural extension of the maximum softmax probability detector in the multi-label setting, obtained by taking the sigmoid of each output score f

Original: ( x ) i and computing − max i σ ( f ( x ) i ) .
Revised: ( x ) i and computing − max i σ ( f ( x ) i ) .

Original: Alternatively, one can average the logit values, denoted by LogitAvg.
Revised: Alternatively, one can average the logit values, denoted by LogitAvg.

Original: These serve as our baseline detectors for multi-label OOD detection.
Revised: These serve as our baseline detectors for multi-label OOD detection.

Original: We compare these baselines to the MaxLogit detector that we introduce in Section 3.
Revised: We compare these baselines to the MaxLogit detector that we introduce in Section 3.

Original: As in the multi-class case, the MaxLogit anomaly score for multi-label classiﬁcation is − max i f ( x ) i .
Revised: As in the multi-class case, the MaxLogit anomaly score for multi-label classiﬁcation is − max i f ( x ) i .

Original: Results.
Revised: Results.

Original: Results are shown in Table 3.
Revised: Results are shown in Table 3.

Original: We ﬁnd that MaxLogit obtains the highest performance in all cases.
Revised: We ﬁnd that MaxLogit obtains the highest performance in all cases.

Original: MaxLogit bears similarity to the MSP baseline (Hendrycks & Gimpel, 2017) but is naturally applicable to multi-label problems.
Revised: MaxLogit bears similarity to the MSP baseline (Hendrycks & Gimpel, 2017) but is naturally applicable to multi-label problems.

Original: These results establish the MaxLogit as an effective and natural baseline for large-scale multi-label problems.
Revised: These results establish the MaxLogit as an effective and natural baseline for large-scale multi-label problems.

Original: Further, the evaluation setup enables future work in out-of-distribution detection with multi-label datasets.
Revised: Further, the evaluation setup enables future work in out-of-distribution detection with multi-label datasets.

Original:  T HE CAOS B ENCHMARK
Revised:  T HE CAOS B ENCHMARK

Original: The Combined Anomalous Object Segmentation (CAOS) benchmark is comprised of two complementary datasets for evaluating anomaly segmentation systems on diverse, realistic anomalies.
Revised: The Combined Anomalous Object Segmentation (CAOS) benchmark is comprised of two complementary datasets for evaluating anomaly segmentation systems on diverse, realistic anomalies.

Original: First is the StreetHazards dataset, which leverages simulation to provide a large variety of anomalous objects realistically inserted into driving scenes.
Revised: First is the StreetHazards dataset, which leverages simulation to provide a large variety of anomalous objects realistically inserted into driving scenes.

Original: Second is the BDD-Anomaly dataset, which consists of real images taken from the BDD100K dataset (Yu et al., 2018).
Revised: Second is the BDD-Anomaly dataset, which consists of real images taken from the BDD100K dataset (Yu et al., 2018).

Original: StreetHazards contains a highly diverse array of anomalies; BDD-Anomaly contains anomalies in real-world images.
Revised: StreetHazards contains a highly diverse array of anomalies; BDD-Anomaly contains anomalies in real-world images.

Original: Together, these datasets allow researchers to judge techniques on their ability to segment diverse anomalies as well as anomalies in real images.
Revised: Together, these datasets allow researchers to judge techniques on their ability to segment diverse anomalies as well as anomalies in real images.

Original: All images have 720 × 1280 resolution.
Revised: All images have 720 × 1280 resolution.

Original: The StreetHazards Dataset.
Revised: The StreetHazards Dataset.

Original: StreetHazards is an anomaly segmentation dataset that leverages simulation to provide diverse, realistically-inserted anomalous objects.
Revised: StreetHazards is an anomaly segmentation dataset that leverages simulation to provide diverse, realistically-inserted anomalous objects.

Original: To create the StreetHazards dataset, we use the Unreal Engine along with the CARLA simulation environment (Dosovitskiy et al., 2017).
Revised: To create the StreetHazards dataset, we use the Unreal Engine along with the CARLA simulation environment (Dosovitskiy et al., 2017).

Original: From several months of development and testing including customization of the Unreal Engine and CARLA, we can insert foreign entities into a scene while having them be properly integrated.
Revised: From several months of development and testing including customization of the Unreal Engine and CARLA, we can insert foreign entities into a scene while having them be properly integrated.

Original: Unlike previous work, this avoids the issues of inconsistent chromatic aberration, inconsistent lighting, edge effects, and other simple cues that an object is anomalous.
Revised: Unlike previous work, this avoids the issues of inconsistent chromatic aberration, inconsistent lighting, edge effects, and other simple cues that an object is anomalous.

Original: Additionally, using a simulated environment allows us to dynamically insert diverse anomalous objects in any location and have them render properly with changes to lighting and weather including time of day, cloudy skies, and rain.
Revised: Additionally, using a simulated environment allows us to dynamically insert diverse anomalous objects in any location and have them render properly with changes to lighting and weather including time of day, cloudy skies, and rain.

Original: We use 3 towns from CARLA for training, from which we collect RGB images and their respective semantic segmentation maps to serve as training data for semantic segmentation models.
Revised: We use 3 towns from CARLA for training, from which we collect RGB images and their respective semantic segmentation maps to serve as training data for semantic segmentation models.

Original: We generate a validation set from the fourth town.
Revised: We generate a validation set from the fourth town.

Original: Finally, we reserve the ﬁfth and sixth town as our test set.
Revised: Finally, we reserve the ﬁfth and sixth town as our test set.

Original: We insert anomalies taken from the Digimation Model Bank Library and semantic ShapeNet(ShapeNetSem) (Savva et al., 2015) into the test set in order to evaluate methods for out-of-distribution
Revised: We insert anomalies taken from the Digimation Model Bank Library and semantic ShapeNet

Original: Examples and Predictions for Our StreetHazards Dataset
Revised: Examples and Predictions for Our StreetHazards Dataset

Original: I n p u t
Revised: I n p u t

Original: G T
Revised: G T

Original: M S P
Revised: M S P

Original: A u t o e n c o d e r
Revised: A u t o e n c o d e r

Original: M a x L o g i t  detection.
Revised: M a x L o g i t is the maximum softmax probability baseline ( a new the MaxLogit detector places lower scores on in-distribution image regions, including object outlines, while also doing a better job of highlighting anomalous objects. (ShapeNetSem) (Savva et al., 2015) into the test set in order to evaluate methods for out-of-distribution detection.

Original: In total, we use 250 unique anomaly models of diverse types.
Revised: In total, we use 250 unique anomaly models of diverse types.

Original: There are 12 classes used for training: background, road, street lines, trafﬁc signs, sidewalk, pedestrian, vehicle, building, wall, pole, fence, and vegetation.
Revised: There are 12 classes used for training: background, road, street lines, trafﬁc signs, sidewalk, pedestrian, vehicle, building, wall, pole, fence, and vegetation.

Original: The thirteenth class is the anomaly class that is only used at test time.
Revised: The thirteenth class is the anomaly class that is only used at test time.

Original: We collect 5,125 image and semantic segmentation ground truth pairs for training, 1,031 pairs without anomalies for validation, and 1,500 test pairs with anomalies.
Revised: We collect 5,125 image and semantic segmentation ground truth pairs for training, 1,031 pairs without anomalies for validation, and 1,500 test pairs with anomalies.

Original: The BDD-Anomaly Dataset.
Revised: The BDD-Anomaly Dataset.

Original: BDD-Anomaly is an anomaly segmentation dataset with real images in diverse conditions.
Revised: BDD-Anomaly is an anomaly segmentation dataset with real images in diverse conditions.

Original: We source BDD-Anomaly from BDD100K (Yu et al., 2018), a large-scale semantic segmentation dataset with diverse driving conditions.
Revised: We source BDD-Anomaly from BDD100K (Yu et al., 2018), a large-scale semantic segmentation dataset with diverse driving conditions.

Original: The original data consists inimages for training and 1000 for validation.
Revised: The original data consists in 7, images for training and 1,000 for validation.

Original: There are 18 original classes.
Revised: There are 18 original classes.

Original: We choose motorcycle , train , and bicycle as the anomalous object classes and remove all images with these objects from the training and validation sets.
Revised: We choose motorcycle , train , and bicycle as the anomalous object classes and remove all images with these objects from the training and validation sets.

Original: This yields 6,280 training pairs, 910 validation pairs without anomalies, and 810 testing pairs with anomalous objects.
Revised: This yields 6,280 training pairs, 910 validation pairs without anomalies, and 810 testing pairs with anomalous objects.

Original: 5. E XPERIMENTS
Revised: 5. E XPERIMENTS

Original: Evaluation.
Revised: Evaluation.

Original: In anomaly segmentation experiments, each pixel is treated as a prediction, resulting in many predictions to evaluate.
Revised: In anomaly segmentation experiments, each pixel is treated as a prediction, resulting in many predictions to evaluate.

Original: To ﬁt these in memory, we compute the metrics on each image and average over the images to obtain ﬁnal values.
Revised: To ﬁt these in memory, we compute the metrics on each image and average over the images to obtain ﬁnal values.

Original: Methods.
Revised: Methods.

Original: Our ﬁrst baseline is pixel-wise Maximum Softmax Probability (MSP).
Revised: Our ﬁrst baseline is pixel-wise Maximum Softmax Probability (MSP).

Original: Introduced by
Revised: Introduced by

Original: Hendrycks & Gimpel (2017) for multi-class out-of-distribution detection, we directly port this baseline to anomaly segmentation.
Revised: Hendrycks & Gimpel (2017) for multi-class out-of-distribution detection, we directly port this baseline to anomaly segmentation.

Original: Alternatively, the background class might serve as an anomaly detector, because it contains everything not in the other classes.
Revised: Alternatively, the background class might serve as an anomaly detector, because it contains everything not in the other classes.

Original: To test this hypothesis, “Background” uses the posterior probability of the background class as the anomaly score.
Revised: To test this hypothesis, “Background” uses the posterior probability of the background class as the anomaly score.

Original: The Dropout methodleverages MC Dropout (Gal & Ghahramani, 2016) to obtain an epistemic uncertainty estimate.
Revised: The Dropout method

Original: [Table]
Revised: [Table]

Revised: leverages MC Dropout (Gal & Ghahramani, 2016) to obtain an epistemic uncertainty estimate.

Original: Following Kendall et al.
Revised: Following Kendall et al.

Original: (2015), we compute the pixel-wise posterior variance over multiple dropout masks and average across all classes, which serves as the anomaly score.
Revised: (2015), we compute the pixel-wise posterior variance over multiple dropout masks and average across all classes, which serves as the anomaly score.

Original: We also experiment with an autoencoder baseline similar to Baur et al.
Revised: We also experiment with an autoencoder baseline similar to Baur et al.

Original: ; Haselmann et al.
Revised: ; Haselmann et al.

Original: (2018) where pixel-wise reconstruction loss is used as the anomaly score.
Revised: (2018) where pixel-wise reconstruction loss is used as the anomaly score.

Original: This method is called AE.
Revised: This method is called AE.

Original: The “Branch” method is a direct port of the conﬁdence branch detector from DeVries & Taylor (2018) to pixel-wise prediction.
Revised: The “Branch” method is a direct port of the conﬁdence branch detector from DeVries & Taylor (2018) to pixel-wise prediction.

Original: Finally, we use the MaxLogit method described in earlier sections.
Revised: Finally, we use the MaxLogit method described in earlier sections independently on each pixel.

Original: For all of the baselines except the autoencoder, we train a PSPNet (Zhao et al., 2017) decoder with a
Revised: For all of the baselines except the autoencoder, we train a PSPNet (Zhao et al., 2017) decoder with a

Original: ResNet-101 encoder (He et al., 2015) for 20 epochs.
Revised: ResNet-101 encoder (He et al., 2015) for 20 epochs.

Original: We train both the encoder and decoder using
Revised: We train both the encoder and decoder using

Original: SGD with momentum of 0.9, a learning rate of 2 × 10 − 2 , and learning rate decay of 10 − 4 .
Revised: SGD with momentum of 0.9, a learning rate of 2 × 10 − 2 , and learning rate decay of 10 − 4 .

Original: For AE, we use a 4-layer U-Net (Ronneberger et al., 2015) with a spatial latent code as in Baur et al.
Revised: For AE, we use a 4-layer U-Net (Ronneberger et al., 2015) with a spatial latent code as in Baur et al.

Original: The U-Net also uses batch norm and is trained for 10 epochs.
Revised: The U-Net also uses batch norm and is trained for 10 epochs.

Original: Results are in Table 4.
Revised: Results are in Table 4.

Original: [Figure]
Revised: [Figure]

Original: Results and Analysis.
Revised: Results and Analysis.

Original: MaxLogit outperforms all other methods across the board by a substantial margin.
Revised: MaxLogit outperforms all other methods across the board by a substantial margin.

Original: The intuitive baseline of using the posterior for the background class to detect anomalies performs poorly, which suggests that the background class may not align with rare visual features.
Revised: The intuitive baseline of using the posterior for the background class to detect anomalies performs poorly, which suggests that the background class may not align with rare visual features.

Original: Even though reconstruction-based scores succeed in product fault segmentation, we ﬁnd that the AE method performs poorly on the CAOS benchmark, which may be due to the more complex domain.
Revised: Even though reconstruction-based scores succeed in product fault segmentation, we ﬁnd that the AE method performs poorly on the CAOS benchmark, which may be due to the more complex domain.

Original: AUPR for all methods is low, indicating that the large class imbalance presents a serious challenge.
Revised: AUPR for all methods is low, indicating that the large class imbalance presents a serious challenge.

Original: However, the substantial improvements with the MaxLogit method suggest that progress on this task is possible and there is much room for improvement.
Revised: However, the substantial improvements with the MaxLogit method suggest that progress on this task is possible and there is much room for improvement.

Original: A comparison with other datasets is in Figure 5 (Pinggera et al., 2016;
Revised: A comparison with other datasets is in Figure 5 (Pinggera et al., 2016;

Original: Blum et al., 2019; Jung et al., 2021).
Revised: Blum et al., 2019; Jung et al., 2021).

Original: In Figure 4, we see that both MaxLogit and MSP have many false positives, as they assign high anomaly scores to semantic boundaries, a problem also observed in the recent works of (Blum et al., 2019;
Revised: In Figure 4, we see that both MaxLogit and MSP have many false positives, as they assign high anomaly scores to semantic boundaries, a problem also observed in the recent works of (Blum et al., 2019;

Original: Angus, 2019).
Revised: Angus, 2019).

Original: However, the problem is less severe with MaxLogit.
Revised: However, the problem is less severe with MaxLogit.

Original: A potential explanation for this is that even when the prediction conﬁdence dips at semantic boundaries, the maximum logit can remain the same in a ‘hand-off’ procedure between the classes.
Revised: A potential explanation for this is that even when the prediction conﬁdence dips at semantic boundaries, the maximum logit can remain the same in a ‘hand-off’ procedure between the classes.

Original: Thus, MaxLogit provides a natural mechanism to combat semantic boundary artifacts that could be further explored in future work.
Revised: Thus, MaxLogit provides a natural mechanism to combat semantic boundary artifacts that could be further explored in future work.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We scaled out-of-distribution detection to settings with thousands of classes and high-resolution images.
Revised: We scaled out-of-distribution detection to settings with thousands of classes and high-resolution images.

Original: We identiﬁed an issue faced by existing baselines when scaling to these settings and proposed the maximum logit detector as a natural solution.
Revised: We identiﬁed an issue faced by existing baselines when scaling to these settings and proposed the maximum logit detector as a natural solution.

Original: We introduced the Species dataset to enable more controlled experiments without class overlap and also investigated using multi-label classiﬁers for
Revised: We introduced the Species dataset to enable more controlled experiments without class overlap and also investigated using multi-label classiﬁers for

Original: OOD detection, establishing an experimental setup for this previously unexplored setting.
Revised: OOD detection, establishing an experimental setup for this previously unexplored setting.

Original: Finally, we introduced the CAOS benchmark for anomaly segmentation, consisting of diverse, naturally-integrated anomalous objects in driving scenes.
Revised: Finally, we introduced the CAOS benchmark for anomaly segmentation, consisting of diverse, naturally-integrated anomalous objects in driving scenes.

Original: Baseline methods on the CAOS benchmark substantially improve on random guessing but are still lacking, indicating potential for future work.
Revised: Baseline methods on the CAOS benchmark substantially improve on random guessing but are still lacking, indicating potential for future work.

Original: Interestingly, the MaxLogit detector also provides consistent and signiﬁcant gains in the multi-label and anomaly segmentation settings, thereby establishing it as a new baseline in place of the maximum softmax probability baseline on large-scale OOD detection problems.
Revised: Interestingly, the MaxLogit detector also provides consistent and signiﬁcant gains in the multi-label and anomaly segmentation settings, thereby establishing it as a new baseline in place of the maximum softmax prob- ability baseline on large-scale OOD detection problems.

Original: In all, we we hope that our contributions will enable further research on out-of-distribution detection for real-world safety-critical environments.
Revised: In all, we we hope that our contributions will enable further research on out-of-distribution detection for real-world safety-critical environments.

Original: B F ULL M ULTICLASS OOD D ETECTION R ESULTS
Revised: B F ULL M ULTICLASS OOD D ETECTION R ESULTS

Original: Datasets.
Revised: Datasets.

Original: To evaluate the MSP baseline out-of-distribution detector and the MaxLogit detector, we use the ImageNet-1K object recognition dataset and Places365 scene recognition dataset as in-distribution datasets D in .
Revised: To evaluate the MSP baseline out-of-distribution detector and the MaxLogit detector, we use the ImageNet-1K object recognition dataset and Places365 scene recognition dataset as in-distribution datasets D in .

Original: We use several out-of-distribution test datasets D out , all of which are unseen during training.
Revised: We use several out-of-distribution test datasets D out , all of which are unseen during training.

Original: The ﬁrst out-of-distribution dataset is Gaussian noise, where each example’s pixels are i.i.d.
Revised: The ﬁrst out-of-distribution dataset is Gaussian noise, where each example’s pixels are i.i.d.

Original: sampled from N (0 , 0 . 5) and clipped to be contained within [ − 1 , 1] .
Revised: sampled from N (0 , 0 . 5) and clipped to be contained within [ − 1 , 1] .

Original: Another type of test-time noise is Rademacher noise, in which each pixel is i.i.d.
Revised: Another type of test-time noise is Rademacher noise, in which each pixel is i.i.d.

Original: sampled from 2 · Bernoulli (0 . 5) − 1 , i.e. each pixel is 1 or − 1 with equal probability.
Revised: sampled from 2 · Bernoulli (0 . 5) − 1 , i.e. each pixel is 1 or − 1 with equal probability.

Original: Blob examples are more structured than noise; they are algorithmically generated blob images.
Revised: Blob examples are more structured than noise; they are algorithmically generated blob images.

Original: Meanwhile, Textures is a dataset consisting in images of describable textures (Cimpoi et al., 2014).
Revised: Meanwhile, Textures is a dataset consisting in images of describable textures (Cimpoi et al., 2014).

Original: When evaluating the ImageNet-1K detector, we use LSUN images, a scene recognition dataset (Yu et al., 2015).
Revised: When evaluating the ImageNet-1K detector, we use LSUN images, a scene recognition dataset (Yu et al., 2015).

Original: Our ﬁnal D out is Places69 , a scene classiﬁcation dataset that does not share classes with Places365.
Revised: Our ﬁnal D out is Places69 , a scene classiﬁcation dataset that does not share classes with Places365.

Original: In all, we evaluate against out-of-distribution examples spanning synthetic and realistic images.
Revised: In all, we evaluate against out-of-distribution examples spanning synthetic and realistic images.

Original: Overview of Other Detection Methods.
Revised: KL Matching Method. To verify our intuitions that led us to develop the MaxLogit detector, we developed a less convenient but similarly powerful technique applicable for the multiclass setting. Recall that some classes tend to be predicted with low conﬁdence and others high conﬁdence. The shape of predicted posterior distributions is often class dependent.

Revised: We capture the typical shape of each class’s posterior distribution and form posterior distribution templates for each class.

Revised: During test time, the network’s softmax posterior distribution is compared to these templates and an anomaly score is generated.

Revised: More concretely, we compute k

Revised: different distributions d k , one for each class.

Revised: We write d k = E x (cid:48)

Revised: ∼X val [ p ( y | x (cid:48) )]

Revised: where k = argmax k p ( y = k | x (cid:48) ) .

Revised: Then for a new test input x , we calculate the anomaly score min k KL [ p ( y | x ) (cid:107) d k ] rather than the MSP baseline − max k p ( y = k | x ) .

Revised: Note that we utilize the validation dataset, but our KL matching method does not require the validation dataset’s labels.

Revised: That said, our KL matching method is less convenient than our MaxLogit technique, and the two perform similarly.

Revised: Since this technique requires more data than MaxLogit, we opt to simply use the MaxLogit in the main paper.

Revised: Results.

Revised: Observe that the proposed MaxLogit method outperforms the maximum softmax probability baseline for all three metrics on both ImageNet and Places365.

Revised: These results were computed using a ResNet-50 trained on either ImageNet-1K or Places365.

Revised: In the case of Places365, the AUROC improvement is over 10%.

Revised: We note that the utility of the maximum logit could not be appreciated as easily in previous work’s small-scale settings.

Revised: For example, using the small-scale CIFAR-10 setup of

Revised: Hendrycks et al.

Revised: Hendrycks et al.

Revised: (2019a)

Revised: , the MSP attains an average AUROC of 90.08% while the maximum logit attains 90.22%, a minor 0.14% difference.

Revised: However, in a large-scale setting, the difference can be over 10% on individual D out datasets.

Revised: We are not claiming that utilizing the maximum logit is a mathematically innovative formulation, only that it serves as a consistently powerful baseline for large-scale settings that went unappreciated in small-scale settings.

Revised: In consequence, we suggest using the maximum logit as a new baseline for large-scale multi-class out-of-distribution detection.

Revised: Overview of Other Detection Methods.

Original: There are other techniques in out-of-distribution detection which require other assumptions such as more training data.
Revised: There are other techniques in out-of-distribution detection which require other assumptions such as more training data.

Original: For instance, Hendrycks et al.
Revised: For instance, Hendrycks et al.

Original: (2019a);
Revised: (2019a);

Original: Mohseni et al.
Revised: Mohseni et al.

Original: (2020) use additional training data labeled as out-of-distribution, and the MaxLogit technique can be naturally extended should such data be available.
Revised: (2020) use additional training data labeled as out-of-distribution, and the MaxLogit technique can be naturally extended should such data be available.

Original: Hendrycks et al.
Revised: Hendrycks et al.

Original: (2019c) use rotation prediction and self-supervised learning, but we found that scaling this to the ImageNet multiclass setting did not produce strong results.
Revised: (2019c) use rotation prediction and self-supervised learning, but we found that scaling this to the ImageNet multiclass setting did not produce strong results.

Original: The MSP baseline trained with auxiliary rotation prediction has an AUROC of 59.1%, and with MaxLogit it attains a 73.6% AUROC, over a 10% absolute improvement with MaxLogit.
Revised: The MSP baseline trained with auxiliary rotation prediction has an AUROC of 59.1%, and with MaxLogit it attains a 73.6% AUROC, over a 10% absolute improvement with MaxLogit.

Original: Nonetheless this technique did not straightforwardly scale, as the network is better without auxiliary rotation prediction.
Revised: Nonetheless this technique did not straightforwardly scale, as the network is better without auxiliary rotation prediction.

Original: Likewise, Lee et al.
Revised: Likewise, Lee et al.

Original: (2018b) propose to use Mahalanobis distances, but in scaling this to 1000 classes, we consistently encountered NaN errors due to high condition numbers.
Revised: (2018b) propose to use Mahalanobis distances, but in scaling this to 1000 classes, we consistently encountered NaN errors due to high condition numbers.

Original: This shows the importance of ensuring that out-of-distribution techniques can scale.
Revised: This shows the importance of ensuring that out-of-distribution techniques can scale.

Original: ODIN Liang et al.
Revised: ODIN Liang et al.

Original: (2018) assumes that, for each OOD example source, we can tune hyperparameters for detection.
Revised: (2018) assumes that, for each OOD example source, we can tune hyperparameters for detection.

Original: For this reason we do not evaluate with ODIN in the rest of the paper.
Revised: For this reason we do not evaluate with ODIN in the rest of the paper.

Original: However, for thoroughness, we evaluate it here.
Revised: However, for thoroughness, we evaluate it here.

Original: ODIN uses temperature scaling and adds an epsilon perturbation to the input in order to separate the softmax posteriors between in- and out-of-distribution images; we set these hyperparameters following DeVries & Taylor (2018).
Revised: ODIN uses temperature scaling and adds an epsilon perturbation to the input in order to separate the softmax posteriors between in- and out-of-distribution images; we set these hyperparameters following DeVries & Taylor (2018).

Original: Then, MaxLogit combined with
Revised: Then, MaxLogit combined with

Original: ODIN results in an FPR95 of 33.6, an AUROC of 88.8 and an AUPR of 51.3 on ImageNet.
Revised: ODIN results in an FPR95 of 33.6, an AUROC of 88.8 and an AUPR of 51.3 on ImageNet.

Original: On
Revised: On

Original: Places365, the FPR95 is 35.3, the AUROC is 86.5, and the AUPR is 24.2.
Revised: Places365, the FPR95 is 35.3, the AUROC is 86.5, and the AUPR is 24.2.

Original: Consequently, techniques built with different assumptions can integrate well with MaxLogit.
Revised: Consequently, techniques built with different assumptions can integrate well with MaxLogit.

Original: We do not train ImageNet-22K models from scratch with these methods due to limited compute.
Revised: We do not train ImageNet-21K models from scratch with these methods due to limited compute.

Original: C M ULTI -L ABEL O UT -O F -D ISTRIBUTION D ATASET L IST0.
Revised: C M ULTI -L ABEL O UT -O F -D ISTRIBUTION D ATASET L IST

Original: 1.0 False Positive Rate 0.

Original: T r u e
Revised: [Figure]

Original: P o s i t i v e R a t e MaxLogit LOF
Revised: [Figure]

Original: Random
Revised: [Figure]

Original: For multi-label classiﬁcation experiments, we choose the following classes from ImageNet-22K to serve as out-of-distribution data: dolphin (n02069412), deer (n02431122), bat (n02139199), rhino (n02392434), raccoon (n02508213), octopus (n01970164), giant clam (n01959492) (n01937909) Venus ﬂytrap (n12782915) (n12641413)
Revised: For multi-label classiﬁcation experiments, we choose the following classes from ImageNet-21K to serve as out-of-distribution data: dolphin (n02069412), deer (n02431122), bat (n02139199), rhino (n02392434), raccoon (n02508213), octopus (n01970164), giant clam (n01959492), leech (n01937909), Venus ﬂytrap (n12782915), cherry tree (n12641413), Japanese cherry blossoms (n12649317), red wood (n12285512), sunﬂower (n11978713), croissant (n07691650), stick cinnamon (n07814390), cotton (n12176953), rice (n12126084), sugar cane (n12132956), bamboo (n12147226), and tumeric (n12356395).

Original: Japanese cherry blossoms (n12649317) red wood (n12285512) sunﬂower (n11978713) (n07691650) stick cinnamon (n07814390) (n12176953) and [Equation]
Revised: These classes were hand-chosen so that they are distinct from VOC and COCO classes.

Original:  EGMENTATION
Revised: D OOD S EGMENTATION

Original: We cover methods used in the paper in more depth and the modiﬁcations necessary to make the methods work with OOD detection in semantic segmentation.
Revised: We cover methods used in the paper in more depth and the modiﬁcations necessary to make the methods work with OOD detection in semantic segmentation.

Original: We use f to denote the function typically a neural network, x is the input image, and y i,j is the prediction for pixel i, j .
Revised: We use f to denote the function typically a neural network, x is the input image, and y i,j is the prediction for pixel i, j .

Original: We will denote the output probability distribution per pixel as P and locations i, j as the location of the respective pixel in the output.
Revised: We will denote the output probability distribution per pixel as P and locations i, j as the location of the respective pixel in the output.

Original: f ( x ) i,j denotes the i th row and j ’th column of the output. 
Revised: f ( x ) i,j denotes the i th row and j ’th column of the output. [Figure]

Revised: lighting allows techniques anomaly

Revised: Johnson & Farid cutand anomalies the anomalies in

Revised: Street

Revised: Hazards dataset are naturally integrated their environment them

Original: Conﬁdence Estimation.
Revised: Conﬁdence Estimation.

Original: The method proposed in DeVries & Taylor (2018) works by training a conﬁdence branch added at the end of the neural network.
Revised: The method proposed in DeVries & Taylor (2018) works by training a conﬁdence branch added at the end of the neural network.

Original: We denote the network predictions as both P and ˆ c whereby every pixel is assigned a conﬁdence value.
Revised: We denote the network predictions as both P and ˆ c whereby every pixel is assigned a conﬁdence value.

Original: [Equation]
Revised: [Equation]

Original: The conﬁdence estimation denoted by c is given “hints” during training to guide what it is learning.
Revised: The conﬁdence estimation denoted by c is given “hints” during training to guide what it is learning.

Original: The B is a beta distribution and acts as a regularizer similar to dropout so that the network f does not exclusively rely on the true labels being present.
Revised: The B is a beta distribution and acts as a regularizer similar to dropout so that the network f does not exclusively rely on the true labels being present.

Original: The ﬁnal loss is modiﬁed to include the extra term below:
Revised: The ﬁnal loss is modiﬁed to include the extra term below:

Original: [Equation]
Revised: [Equation]

Original: The reasoning for L c is to encourage the network to output conﬁdent predictions.
Revised: The reasoning for L c is to encourage the network to output conﬁdent predictions.

Original: Finally λ is initialized to 0.1 and is updated by a “budget” parameter which is set to the default of 0.3.
Revised: Finally λ is initialized to 0.1 and is updated by a “budget” parameter which is set to the default of 0.3.

Original: The update equation:
Revised: The update equation:

Revised: [Equation]

Revised: [Table]

Original: [Figure]
Revised: [Equation]

Original: This adaptively adjusts the weighting between the two losses and experimentally the update is not sensitive to the budget parameter.
Revised: This adaptively adjusts the weighting between the two losses and experimentally the update is not sensitive to the budget parameter.

Original: Semantic Segmentation BDD Anomalies Dataset List.
Revised: Semantic Segmentation BDD Anomalies Dataset List.

Original: The BDD100K dataset containsinstances of the train class, 4296 instances of the motorcycle class, and 10229 instances of the bicycle class.[Figure] techniques detect the
Revised: The BDD100K dataset containsinstances of the train class, 4296 instances of the motorcycle class, and 10229 instances of the bicycle class.

Original: Johnson & and paste anomalies

Original: Street dataset naturally integrated their environment with proper lighting and orientation, making them more difﬁcult to detect.

Original: StreetHazards 3D Models Dataset List.
Revised: StreetHazards 3D Models Dataset List.

Original: For semantic segmentation experiments, we choose to use the following classes 3D models from Model Bank Library to serve as out-ofdistribution data:Meta-categories: Animals,
Revised: For semantic segmentation experiments, we choose to use the following classes 3D models from Model Bank Library to serve as out-of-distribution data:

Original:  Vehicles, Weapons, Appliances, Household items (furniture, and kitchen items), Electronics, Instruments, and miscellaneous.
Revised: Meta-categories: Animals, Vehicles, Weapons, Appliances, Household items (furniture, and kitchen items), Electronics, Instruments, and miscellaneous.

Original: The speciﬁc animals used are kangaroos, whales, dolphins, cows, lions, frogs, bats, insects, mongooses, scorpions, ﬁsh, camels, ﬂamingos, apes, horses, mice, spider, dinosaurs, elephants, moose, shrimps, bats, butterﬂies, turtles, hippopotamuses, dogs, cats, sheep, seahorse, snail and zebra.
Revised: The speciﬁc animals used are kangaroos, whales, dolphins, cows, lions, frogs, bats, insects, mongooses, scorpions, ﬁsh, camels, ﬂamingos, apes, horses, mice, spider, dinosaurs, elephants, moose, shrimps, bats, butterﬂies, turtles, hippopotamuses, dogs, cats, sheep, seahorse, snail and zebra.

Original: The speciﬁc vehicles used are military trucks, motorcycles, naval ships, pirate ships, submarines, sailing ships, trolleys, trains, airplanes, helicopters, jets, zeppelin, radar tower, construction vehicles (loaders, dump trucks, bulldozer), farming vehicles (harvester, gantry crane, tractor), ﬁre truck, tank, combat vehicles, and trailers.
Revised: The speciﬁc vehicles used are military trucks, motorcycles, naval ships, pirate ships, submarines, sailing ships, trolleys, trains, airplanes, helicopters, jets, zeppelin, radar tower, construction vehicles (loaders, dump trucks, bulldozer), farming vehicles (harvester, gantry crane, tractor), ﬁre truck, tank, combat vehicles, and trailers.

Original: The speciﬁc weapons used are guns, missiles, rocket launchers, and grenades.
Revised: The speciﬁc weapons used are guns, missiles, rocket launchers, and grenades.

Original: The appliances used are refrigerators, stoves, washing machines, and ovens.
Revised: The appliances used are refrigerators, stoves, washing machines, and ovens.

Original: The household items used are cabinets, armoire, grandfather clocks, bathtubs, bureaus, night stand, table, bed, bookcase, ofﬁce desk, glasses (drinking), throne chair, kitchen utensils (knives, forks, spoons), sofa, clothing iron, plates, sewing machine, and dressing mirror.
Revised: The household items used are cabinets, armoire, grandfather clocks, bathtubs, bureaus, night stand, table, bed, bookcase, ofﬁce desk, glasses (drinking), throne chair, kitchen utensils (knives, forks, spoons), sofa, clothing iron, plates, sewing machine, and dressing mirror.

Original: The electronics used are computer monitor, computer mouse, hair dryer, The instruments category includes bassoon, clarinet, drums, guitar, violin, harp, and keyboard.
Revised: The electronics used are computer monitor, computer mouse, hair dryer, The instruments category includes bassoon, clarinet, drums, guitar, violin, harp, and keyboard.

Original: The miscellaneous category includes rocket, space capsule, space shuttle, lunar module, glasses (wearable), weight machine, balance beam, bench press, bowling ball and pins, and pens.
Revised: The miscellaneous category includes rocket, space capsule, space shuttle, lunar module, glasses (wearable), weight machine, balance beam, bench press, bowling ball and pins, and pens.

Original: Several categories and instances were excluded from Model Bank Library due to their occurrence in the simulation environment such as playground equipment and various types of foliage and trees.
Revised: Several categories and instances were excluded from Model Bank Library due to their occurrence in the simulation environment such as playground equipment and various types of foliage and trees.

Original: The sizes of instances used in the dataset might not reﬂect the actual scale that would otherwise naturally occur.
Revised: The sizes of instances used in the dataset might not reﬂect the actual scale that would otherwise naturally occur.

Original: Similarly the location of instances in the dataset are not necessarily reﬂective of where they are likely to occur in nature.
Revised: Similarly the location of instances in the dataset are not necessarily reﬂective of where they are likely to occur in nature.