A N E MPIRICAL E XPLORATION OF O PEN -S ET R ECOGNITION VIA L IGHTWEIGHT S TATISTICAL P IPELINES
Anonymous authors
Paper under double-blind review
A BSTRACT
Machine-learned safety-critical systems need to be self-aware and reliably know their unknowns in the open-world.
This is often explored through the lens of anomaly/outlier detection or out-of-distribution modeling.
One popular formulation is that of open-set classiﬁcation, where an image classiﬁer trained for 1-of- K classes should also recognize images belonging to a ( K + 1) th “other” class, not present in the training set.
Recent work has shown that, somewhat surprisingly, most if not all existing open-world methods do not work well on high-dimensional open-world images (Shafaei et al., 2019).
In this paper, we carry out an empirical exploration of open-set classiﬁcation, and ﬁnd that combining classic statistical methods with carefully computed features can dramatically outperform prior work.
We extract features from off-the-shelf (OTS) state-of-the-art networks for the underlying K -way closed-world task.
We leverage insights from the retrieval community for computing feature descriptors that are low-dimensional (via pooling and PCA) and normalized (via L2-normalization), enabling the modeling of training data densities via classic statistical tools such as kmeans and Gaussian
Mixture Models (GMMs).
Finally, we (re)introduce the task of open-set semantic segmentation, which requires classifying individual pixels into one of K known classes or an “other" class.
In this setting, our feature-based statistical models noticeably outperform prior open-world methods.
I NTRODUCTION
Embodied perception and autonomy require systems to be self-aware and reliably know their unknowns .
This requirement is often formulated as the open set recognition problem (Scheirer et al., 2012), meaning that the system, e.g., a K -way classiﬁcation model, should recognize anomalous examples that do not belong to one of K closed-world classes.
This is a signiﬁcant challenge for machine-learned systems that notoriously over-generalize to anomalies and unknowns on which they should instead raise a warning ﬂag (Amodei et al., 2016).
Open-world benchmarks: Curating open-world benchmarks is hard (Liu et al., 2019).
One common strategy re-purposes existing classiﬁcation datasets into closed vs open examples – e.g., declaring
MNIST digits 0-5 as closed and 6-9 as open (Neal et al., 2018; Oza & Patel, 2019;
Geng et al., 2020). In contrast, anomaly/out-of-distribution (OOD) benchmarks usually generate anomalous samples by adding examples from different datasets - e.g., declaring CIFAR as anomalous for MNIST (Ge et al., 2017; Oza & Patel, 2019; Liu et al., 2019).
Most open-world protocols assume open-world data is not available during training (Liang et al., 2018; Oza & Patel, 2019).
Interestingly, Dhamija et al.
Hendrycks et al.
(2019b)
ﬁnd that, if some open examples are available during training, one can learn simple open-vs-closed binary classiﬁers that are remarkably effective.
However, Shafaei et al.
(2019) comprehensively compare various well-known open-world methods through rigorous experiments, and empirically show that none of the compared methods generalize to high-dimensional open-world images.
Intuitively, classiﬁers can easily overﬁt to the available set of open-world images, which won’t likely exhaustively span the open world outside the K classes of interest.
In this paper, we carry out a rigorous empirical exploration of open-set recognition of highdimensionial images.
We explore simple statistical models such as Nearest Class Means (NCMs), kmeans and Gaussian Mixture Models (GMMs).
Our hypothesis is that such classic statistical methods can reliably model the closed-world distribution (through the closed-world training data),
[Figure]
[Figure]
r o a d bu il d .
v e g e t .
ca r s i
d
e w a l k s ky po l e p e r s on t e rr a i n f e n ce w a ll t r a f ﬁ c
s i
gn b i c y c
l e bu s t r u c k t r a f ﬁ c li gh t t r a i n r i d e r m o t
o r c y c l e s t a t
i c g r o und p a r k i n g d y n a m i c
b r i d g e r a il t r a c k t unn e l c a r ava n t r a il
er g u a r d r a il p
o
l e g r o up nu m b e r o f p i x
e l s and help avoid overﬁtting (an issue in open-vs-closed classiﬁers).
Traditionally, such simple models have been used to address the open-world (Chandola et al., 2009; Geng et al., 2020), but are largely neglected in the recent literature.
We revisit these simple methods, and ﬁnd them quite effective once crucial techniques are considered, as summarized by contributions below.
Contribution 1: We build classic statistical models on top of off-the-shelf (OTS) features computed by the underlying K -way classiﬁcation network.
We ﬁnd it crucial to use OTS features that have been pre-trained and post-processed appropriately (discussed further below).
Armed with such features, we ﬁnd classic statistical models such as kmeans and GMMs (Murphy, 2012) can outperform prior work.
We describe two core technical insights below.
Insight-1 Pre-training networks (e.g., on ImageNet (Deng et al., 2009)) is a common practice for traditional closed-world tasks.
However, to the best of our knowledge, open-world methods do not sufﬁciently exploit pre-training (Oza & Patel, 2019).
Hendrycks et al.
(2019a) report that pre-training improves anomaly detection using softmax conﬁdence thresholding (Hendrycks & Gimpel, 2017).
We ﬁnd pretraining to be a crucial factor in learning better representations that support more sophisticated open-world reasoning.
Intuitively, pre-trained networks expose themselves to diverse data that may look similar to open-world examples encountered at test-time.
We operationalize this intuition by building statistical models on top of existing discriminative networks, which tend to make use of pre-training by design.
We demonstrate this signiﬁcantly outperforms features trained from scratch, as most prior open-set work does.
Insight-2 Low-dimensional normalized features.
While some existing open-world methods also exploit OTS features (Lee et al., 2018), we ﬁnd it crucial to make use of insufﬁciently well-known best practices for feature extraction.
Speciﬁcally, to reduce dimensionality, we pool spatially (Gong et al., 2014) and use principle component analysis (PCA) (Turk & Pentland, 1991).
Then, to ensure features are invariant to scalings, we adopt L2 normalization (Gong et al., 2014; Gordo et al., 2017).
While these are somewhat standard practices for deep feature extraction in areas such as retrieval, their combination is not well explored in the open-set literature (Bendale & Boult, 2016; Grathwohl et al., 2019).
Given a particular OTS K -way classiﬁcation network, we determine the “right” feature processing through validation.
In particular, we ﬁnd that L2-normalization greatly boosts open-world recognition performance; spatial pooling and PCA altogether reduce feature dimension by three orders of magnitude without degrading performance, resulting in a lightweight pipeline.
Contribution 2: We re(introduce) the problem of open-set semantic segmentation.
Interestingly, classic benchmarks explicitly evaluate background pixels outside the set of K classes of interest (Everingham et al., 2015).
However, contemporary benchmarks such as Cityscapes (Cordts et al., 2016) ignore such pixels during evaluation.
As a result, most contemporary segmentation networks also ignore such pixels during training.
Perhaps surprisingly, such ignored pixels include vulnerable objects like strollers and wheelchairs.
Misclassifying such objects may have serious implications for real-world autonomous systems (see Figure 1).
Instead of ignoring these pixels, we use them to explore open-world recognition by repurposing them as open-world examples.
Interestingly, this setup naturally allows for open-pixels in the train-set, a protocol advocated by (Dhamija et al., 2018;
Hendrycks et al., 2019b).
We benchmark various open-world methods on this setup, and show that our suggested simple statistical models still outperform typical open-world methods.
Similar to past work, we also ﬁnd that simple open-vs-closed binary classiﬁers serve as strong baselines, provided one has enough training examples of open pixels that span the open-world.
R ELATED W ORK
Open-set recognition .
There are multiple lines of work addressing the open-world problems in the context of K -way classiﬁcation, such as anomaly/out-of-distribution detection (Chandola et al., 2009;
Zong et al., 2018; Hendrycks et al., 2019b), novelty/outlier detection (Pidhorskyi et al., 2018).
Deﬁned on K -way classiﬁcation, these problems can be crisply formulated as open-set recognition (Scheirer et al., 2012; Bendale & Boult, 2016; Lee et al., 2018; Geng et al., 2020).
Given a testing example, these methods compute the likelihood that it belongs to the open-world via post-hoc functions like density estimation (Zong et al., 2018), uncertainty modeling (Gal & Ghahramani, 2016; Liang et al., 2018; Kendall & Gal, 2017) and reconstruction error of the testing example (Pidhorskyi et al., 2018; Dehaene et al., 2020).
Different from the above sophisticated methods, we train simple statistical models (e.g., GMM) which can work much better by following our proposed pipeline.
Feature extraction.
Off-the-shelf (OTS) features can be extracted from the discriminative network and act as powerful embeddings (Donahue et al., 2014).
Using OTS features for open-set recognition has been explored in prior work (Oza & Patel, 2019; Grathwohl et al., 2019; Lee et al., 2018).
OTS features can be logits, softmax and other intermediate feature activations computed by the discriminative network.
Early open-set methods modify the softmax (Hendrycks & Gimpel, 2017; Bendale & Boult, 2016).
Grathwohl et al.
(2019) learn an energy-based model over the logit features for anomaly detection.
Oza & Patel (2019) reconstruct input images from penultimate-layer features and use the reconstruction error as the open-set likelihood.
Most related to our work is Lee et al.
(2018), who build Gaussian models over OTS features for anomaly detection, but relies on input image perturbation for better open-set classiﬁcation performance.
In contrast, we study even simpler statistical models such as kmeans and GMM, and show that proper feature processing (via
L2-normalization and PCA) greatly boosts the efﬁcacy and efﬁciency of open-set recognition.
O PEN -S ET R ECOGNITION VIA L IGHTWEIGHT S TATISTICAL P IPELINES
In this section, we discuss various design choices in our pipeline, including (1) training schemes for the underlying closed-world task, (2) methods for extracting and repurposing closed-world feature descriptors for open-world recognition, and (3) the statistical density estimation models built on such extracted features.
We conclude with (4) an analysis of the additional compute required for self-aware processing ( via the addition of an open -world "head" on top of the closed world network) , pointing out that minimal additional processing is needed.
Network training strategies.
Virtually all state-of-the-art deep classiﬁers make use of large-scale pre-training, e g., on ImageNet (Deng et al., 2009) , which seems to consistently improve towards the state-of-the-art performance on the closed-world data (
Sun et al., 2017; Mahajan et al., 2018) However, many, if not all, open-world methods trains the discriminant network purely on the closedworld data without pre-training (Oza & Patel, 2019; Hendrycks & Gimpel, 2017).
We argue that a pre-trained network also serves as an abstraction of the (pseudo) open world.
Intuitively, such a pre-trained model has already seen diverse data that may look similar to the open-world examples that will be encountered at test-time, particularly if ImageNet does not look similar to the (closed) training set for the task of interest.
Recently, Hendrycks et al.
(2019a) show that pre-training improves open-world robustness with a simplistic method that thresholds softmax conﬁdence (Hendrycks & Gimpel, 2017).
Our diagnostic study shows that our explored statistical models, as well as prior methods, do perform much better when built on a pre-trained network than a network trained from scratch!
Feature extraction.
OTS features generated at different layers of the trained discriminative model can be repurposed for open-set recognition (Lee et al., 2018)
Most
methods leverage softmax (Hendrycks & Gimpel, 2017) and logits (Bendale & Boult, 2016; Grathwohl et al., 2019) which can be thought of as features extracted at top layers.
Similar to (Lee et al., 2018), we ﬁnd it crucial to analyze features from intermediate layers for open-set recognition, for which logits and softmax may be too invariant to be effective for open-set recognition (see Figure 3).
One immediate challenge to extract features from an intermediate layer is their high dimensionality, e.g., of size 512x7x7 from ResNet18 (He et al., 2016).
To reduce feature dimension, we simply (max or average) pool the feature activations spatially into a 512-dim feature vectors (Yang & Ramanan, 2015).
We further use PCA, which can reduce dimension by 10 × (from 512-dim to 50-dim) without sacriﬁcing performance.
We ﬁnd this dimensionality particularly important for learning second-order covariance statistics as in GMMs, described below.
Finally, following (Gong et al., 2014; Gordo et al., 2017), we ﬁnd it crucial to L2-normalize extracted features (see Figure 2).
Statistical models.
Given the above extracted features, we can learn various generative statistical models to capture the conﬁdence/probability that a test example belongs to the closed-world distribution.
We explore simple parametric models such as Nearest Class Means (NCMs) (Mensink et al., 2013) and class-conditional Gaussian models (Lee et al., 2018; Grathwohl et al., 2019), as well as non-parametric models such has nearest neighbors (NN) (Boiman et al., 2008; Júnior et al., 2017).
We ﬁnally explore an intermediate regime of mixture models, including (class-conditional) GMMs and kmeans (Chandola et al., 2009;? Cao et al., 2016; Geng et al., 2020).
Our models label a test example as open-world when the inverse probability (e.g., of the most-likely class-conditional GMMs) or distance (e.g., to the closest class centroid) is above a threshold.
One beneﬁt of such simple statistical models is that they are interpretable and relatively easier to diagnose failures.
For example, one failure mode is an open-world sample being misclassiﬁed as a closed-world class.
This happens when open-world data lie close to a class-centroid or Gaussian component mean (see Figure 3-left).
Note that a single statistical model may have several hyperparameters – GMMs can have multiple Gaussian components and different structures of second-order covariance, e.g., either a single scalar, a vector or a full-rank general covariance per component, as denoted by “spherical” , “diag” and “full” , respectively.
We make use of a validation set to determine the hyperparameters (as well as feature processing steps listed above).
Lightweight Pipeline.
We re-iterate that the above feature extraction and statistical models result in a lightweight pipeline for open-set recognition.
We now analyze the number of additional parameters in our pipeline.
Naively learning a GMM over features from the last convolutional layer result in massive second-order statistics, on the order of (512 × 7 × 7) 2 for a 512x7x7 Res feature map.
We ﬁnd that spatial pooling and PCA can reduce dimensionality to 50 , which requires only 50 2 covariance parameters (a reduction of 10 5 ).
We ﬁnd linear dimensionality reduction more effective than sparse covariance matrices (e.g., assuming diagonal structure).
The appendix includes additional experiments.
Given a class-conditional ﬁve-component GMM (the largest found to be effective through cross validation), this requires 128KB storage per class, or 594KB for allclasses in Cityscapes.
This is less than 0.1% of the compute of the underlying closed-world network (e.g., HRNet at 250 MB), making it a quite practical addition that enables self-aware processing on real-time autonomy stacks.
E XPERIMENT
We extensively validate our proposed lightweight statistical pipeline under standard open-set recognition benchmarks, typically focused on image classiﬁcation.
We also consider open-set semantic segmentation, revisiting classic formulations of semantic segmentation that make use of a background label (Everingham et al., 2015).
We start by introducing implementation details, evaluation metrics and baselines.
We then present comprehensive evaluations on each setup.
Implementation.
As discussed earlier, open-world recognition is often explored through the lens of open-set classiﬁcation.
To ensure our approaches retain high-accuracy on the original closedworld tasks, we build statistical models on top of off-the-shelf (OTS) state-of-the-art networks.
For open-set image classiﬁcation, we ﬁne-tune an ImageNet-pretrained ResNet network (Res18/50 in our experiments)
(He et al., 2016) exclusively on the closed-train-set using cross-entropy loss.
For open-set semantic segmentation we use HRNet (Wang et al., 2019), a highly-ranked model on the Cityscapes leaderboard (Cordts et al., 2016).
We extract features at the penultimate layer of each discriminative network (other layers also apply but we do not explore them in this work).
We conduct experiments with PyTorch (Paszke et al., 2017) on a single Titan X GPU.
Evaluation Metric .
Following past work (Hendrycks & Gimpel, 2017; Lee et al., 2018), we evaluate binary detection of open-world examples using the area under the receiver operating characteristic curve (AUROC) (Davis & Goadrich, 2006).
AUROC is a calibration-free and threshold-less metric, simplifying comparisons between methods.
For open-set semantic segmentation, we also use AUROC to evaluate the performance of recognizing “background” pixels as open-world examples.
This is different from traditional practice in segmentation benchmarks (Everingham et al. 2015) which treat such “background” pixels as just another class.
Baselines.
Our statistical pipeline supports various statistical models.
We study the simple models proposed in Section 3, including NN, kmeans, NCM, and GMMs.
All models, including baselines to which we compare, are based on the same underlying classiﬁcation network.
Hyperparameters for all models (e.g., number of mixtures) are tuned on a validation set 1 . • Classiﬁers.
Hendrycks et al.
(2019b) learn a binary open-vs-closed classiﬁer (CLS 2 ) for anomaly detection.
Following classic work in semantic segmentation (Everingham et al., 2015), we also evaluate a ( K +1)-way classiﬁer (CLS ( K +1) ).
We use the softmax score corresponding to the ( K +1)
th “other” class as the open-set likelihood.
Both methods require open-set examples during training.
• Likelihoods.
Many probabilistic models measure open-set likelihood on OTS features, including Max Softmax Probability (MSP) (Hendrycks & Gimpel, 2017) and Entropy (Steinhardt & Liang, 2016) (derived from softmax probabilities).
OpenMax (Bendale & Boult, 2016) ﬁts logits to Weibull distributions (Scheirer et al., 2011) that recalibrate softmax outputs for open-set recognition.
C2AE (Oza & Patel, 2019) learns an additional K -way classiﬁer on OTS features based on reconstruction errors, which are then used as open-set likelihood function.
GDM (Lee et al., 2018) learns a Gaussian Discriminant Model on OTS features and designs open-set likelihood based on Mahalanobis distance.
CROSR (Yoshihashi et al., 2019) trains a reconstruction-based model that jointly performs closed-set K -way classiﬁcation and open-set recognition.
G-Open (Ge et al., 2017) and OSRCI (Neal et al., 2018) turn to Generative Adversarial Networks (GANs) to generate fake images that augment the closed-set training set, and train a discriminative model for open-set recognition. CGDL (Sun et al., 2020) learns class-conditional Gaussian model and relies on reconstruction error for open-set recognition. The last three methods (CROSR, G-Open and CGDL) train ground-up models in contrast to our statistical models that operate on OTS features of an already-trained K -way classiﬁcation network.
As we focus on an empirical exploration rather than achieving the state-of-the-art, we refer readers to more recent approaches for the state-of-the-art by training ground-up models with sophisticated techniques (Zhang et al., 2020; Chen et al., 2020).
[Table]
• Bayesian Networks. Bayesian neural networks compute uncertainty estimates via Monte Carlo estimates (MCdrop) (Gal & Ghahramani, 2016; Loquercio et al., 2020) and calibrated Max Softmax Probability (MSP c )
(Liang et al., 2018), which can also be used as open-set likelihoods.
We implement MCdrop via 500 samples.
4. S ETUP -I: SINGLE - DATASET OPEN - SET RECOGNITION
Setup.
We begin by following the standard benchmark protocol used in most prior work; split a single dataset into open and closed sets w.r.t class labels (e.g., deﬁne MNIST digits 0-5 as the closed-set, and digits 6-9 as the open-set).
This is a common practice in open-set recognition (Neal et al., 2018;
Oza & Patel, 2019).
Notably, methods do not have access to open-set examples during training.
Datasets.
MNIST / CIFAR / SVHN are popular datasets used in the open-set recognition literature (Neal et al., 2018; Hendrycks & Gimpel, 2017).
All three datasets contain ten classes with balanced numbers of images per class.
Standard protocol randomly splits six (four) classes of train/validation-sets into closed (open) train/validation-sets, respectively.
We repeat ﬁve times and report average AUROC for each method.
Through cross-validation, we ﬁnd reliable OTS features can be computed by average-pooling features from the last convolutional layer down to 512-dim, projecting down to 50-dim via PCA, and L2-normalizing.
Results.
Table 1 shows that, perhaps surprisingly, simple statistical models (like kmeans and GMMs) deﬁned on such normalized features already performs on par to many prior methods/
Because GDM (Lee et al., 2018) does not L2-normalize features, we evaluate a variant that does (GDM L 2 ).
The improved performance demonstrates the importance of feature normalization, which although is well known in the image retrieval community, is not widely used in open-set recognition.
We hereby focus on statistical models trained on normalized features, providing raw vs. normalized comparisons in the appendix.
4. S ETUP -II: CROSS - DATASET O PEN - SET RECOGNITION
Setup.
In these experiments, we use the cross-dataset protocol advocated by (Shafaei et al., 2019), where some outlier examples are sampled from a different dataset for training/validation. (e.g., train on TinyImageNet-closed as closed-set, validate on MNIST-open as outlier set, and test on CIFAR-open as open-set).
Conclusions drawn under this setup may generalize better due to less dataset bias in the experimental protocol (Torralba & Efros, 2011).
Datasets .
We use TinyImageNet as the closed-world dataset (for K -way classiﬁcation), which has 200 classes of 64x64 images, split into 500/50/50 images as the train/val/test sets.
Following (Shafaei et al., 2019), we construct open val/test sets using cross-dataset images (Torralba & Efros, 2011), including MNIST, SVHN, CIFAR and Cityscapes.
For example, we use an outlier dataset (e.g.,
MNIST train-set) to tune/train an open-world method, and test on another dataset as the open-set (e.g., CIFAR test-set).
We use bilinear interpolation to resize all images into 64x64 to match TinyImageNet image resolution.
Through cross validation, we ﬁnd reliable OTS features can be computed by average-pooling features from the last convolutional layer down to 2048-dim, projecting down to 200-dim via PCA, and L2-normalizing.
Results for Table 2 are summarized below:
[Table] • Simple statistical models (e.g., NCM and kmeans) can outperform prior open-set methods (e.g., C2AE and GDM).
We ﬁnd that L2-normalization greatly contributes to the success of these simple statistical methods (cf.
details in the appendix).
Both the metric learning and image retrieval (Mensink et al., 2012; Musgrave et al., 2020) literature have shown the importance of L2-normalization.
Informally, open-set recognition queries the testing example and measures how close it is to any of the closed-world training examples (Musgrave et al., 2020).
• Interestingly, kmeans performs slightly better than GMMs.
Considering that the former can be seen as a special case of GMMs that have an identity covariance, we conjecture that learning other types of covariance (e.g., a full-rank covariance matrix) does not help when the underlying K -way network has already provided compact feature representations.
• From last row pair, we can see pre-training notably improves all the methods.
GDM L 2 outperforms the original GDM which operates on raw features (without L2-normalization).
This further conﬁrms the importance of L2-normalization in feature extraction for open-set recognition.
• Perhaps surprisingly, OpenMax does not work well in this setup (though we have spent considerable effort tuning it).
This is consistent with the results in (Dhamija et al., 2018; Shafaei et al., 2019), and we conjecture the reason is that OpenMax cannot effectively recognize cross-dataset anomalous inputs using logit features because they are too invariant to be useful for open-set recognition (Figure 3).
Similar lackluster results hold for other methods that operate on logit features (Entropy and MSP).
4. S ETUP -III: O PEN - SET S EMANTIC S EGMENTATION
Setup.
In these experiments we (re)introduce the task of open-set segmentation by repurposing “background” pixels in contemporary segmentation benchmarks (Cityscapes) as open-world pixels.
As elaborated before, such pixels are either traditionally treated as just another class for segmentation evaluation (Everingham et al., 2015) or ignored completely.
Instead, we evaluate them using openworld metrics such as AUROC.
We will show our statistical methods also outperform other typical open-world methods.
As this setup has natural access to open-world pixels during training, we explore the training of simple open-vs-closed classiﬁers.
We train CLS on the OTS features of the state-of-the-art HRNet, which has already exploited all closed-world pixels in the train-set
The binary open-vs-closed classiﬁer CLS 2 outperforms CLS ( K +1) , presumably due to that the former being trained on balanced batches.
With fewer than 50 images, GMMs outperform such discriminative models.
But with enough open training examples, simple binary classiﬁcation performs remarkably well.
However, because GMMs do not require any open training examples, it cannot overﬁt to them and so may generalize better to the open-world.
Datasets .
Cityscapes (Cordts et al., 2016) provides per-pixel annotations for urban scene images (1024x2048-resolution) for autonomous driving research.
We construct our train- and val-sets from its 2,975 training images, in which we use the last 10 images as val-set and the rest as train-set.
We use its ofﬁcial 500 validation images as our test-set.
The “background” pixels (shown in white of ground-truth visual in Figure 4) are the open-world examples in this setup.
Through validation, we ﬁnd reliable OTS features can be computed by projecting features from the last convolutional layer from 720 down to 100-dim via PCA, and L2-normalizing.
Results.
For our statistical models (as well as GDM), we randomly sample 5000 closed-world pixel features from each class, as it is prohibitively space-consuming to use all the pixel features from the
Cityscapes train- set
We show quantitative comparison in Table 3 and list salient conclusions below
• Clearly, our simple statistical models (e.g., NN and GMM) perform signiﬁcantly better than the classic open-world methods (e.g., MSP and OpenMax).
However, when training on large amounts of open-pixels, CLS methods achieve signiﬁcantly better performance.
This clearly shows the beneﬁt of training on open-world pixels (Hendrycks et al., 2019b).
We do note that GMMs do not need any open pixels during learning, and so may generalize better to novel open-world scenarios not encountered in the training set (Figure 5).
• GDM performs poorly, probably due to arbitrary scales of the raw features that are too uninformative to be used for open-set pixel recognition.
We note that other statistical methods all struggle with raw pooled features (cf. appendix).
However, once we L2-normalize the pixel features to be scale-invariant, these statistical methods perform signiﬁcantly better (as reported in this table).
• Figure 4 shows qualitative results.
MSP predicts segment boundaries as open-pixels.
This makes sense as the MSP mostly returns aleatoric uncertainties corresponding to ambiguous pixel sensor measurements around object boundaries (Kendall & Gal, 2017).
In contrast, GMM reports openpixels on truly novel objects, such as the street-shop and rollator , both of which are ignored by the semantic segmentation network during training HRNet (Wang et al., 2019).
These regions appear to be caused by epistemic uncertainty arising from the lack of training data (Kendall & Gal, 2017).
• Figure 6 plots AUROC performance vs model size for various statistical models.
Notably, NN consumes the most memory, even more than the underlying networks.
GMMs perform the best and are quite lightweight, only consuming 0.6MB when built on the HRNet model (250MB). [Table]
C ONCLUSION
We explore an empirical exploration of open-set recognition via lightweight statistical pipelines.
We ﬁnd simple statistical models quite effective if built on properly processed off-the-shelf features computed by the discriminative networks (originally trained for the closed-world tasks).
Our pipelines endow K -way networks with the ability to be “self-aware”, with negligible additional compute costs (0.1%).
Finally, we (re)introduce the task of open-set semantic segmentation by repurposing background pixels as open-world examples, requiring classiﬁcation of individual pixels into one of
K known/closed-world classes and
A PPENDIX O UTLINE
As elaborated in the main paper, we introduce a lightweight statistical pipeline for open-set recognition by repurposing off-the-shelf (OTS) features computed by a state-of-the-art recognition network.
As our pipeline does not require (re)training the underlying network, it is guaranteed to replicate the state-of-the-art performance of the network on the (closed-world) task for which it was trained, but still allows the ﬁnal recognition system to properly identify never-before-seen data from the open-world.
In the appendix, we expand on our pipeline, including more experiments, analyses and visualizations.
We outline the appendix below.
Section A: Data statistics for open-set semantic segmentation.
We provide data details used for open-set semantic segmentation (Setup-III), motivated by safety concerns in autonomous stacks as shown in Figure 1 left.
Section B: Detailed results by statistical models.
We provide detailed results on the open-set recognition including open-set image recognition (Setup-II), and open-set semantic segmentation (Setup-III).
We detail the performance of the various statistical models studied in the main paper, including Nearest Neighbor (NN), Nearest Class Mean (NCMs), kmeans and Gaussian Mixture
Models (GMMs).
Section C: Reduced dimension via PCA.
We show that PCA can reduce dimensionality signiﬁcantly (making our pipeline quite lightweight), while maintaining or even improving performance.
Section D: Performance vs. memory/compute.
We rigorously evaluate the memory/compute costs of our various statistical pipelines, emphasizing solutions that are both accurate and lightweight.
Section E: Visualization of Gaussian component means.
One beneﬁt of our simple statistical models is their interpretability; we visualize Gaussian means through centroid images, and demonstrate that they correspond to canonical objects (e.g., those with standard poses and clean background).
Section F: Open-Source demonstration.
We include code (via Jupyter Notebook) for open-set semantic segmentation, assuming one has access to precomputed features from HRNet (Wang et al., 2019).
A S ETUP FOR O PEN -S ET S EMANTIC S EGMENTATION
As we (re)introduce the task of open-set semantic segmentation for exploring open-set recognition, in which we re-purpose “backgroud” pixels of Cityscapes as open-world examples (that are from the ( K +1)
th “other” class).
We hereby list the statistics of open and closed-world examples (pixels).
Cityscapes training set has 2,975 images.
We use the ﬁrst 2,965 images for training, and hold out the last 10 as validation set for model selection.
We use the 500 Cityscapes validation images as our test set.
Here are the statistics for the full train/val/test sets.
• train-set for closed-pixels: 2,965 images providing 334M closed-set pixels.
• train-set for open-pixels: 2,965 images providing 44M open-set pixels.
• val-set for closed-pixels: 10 images providing 1M closed-set pixels.
• val-set for open-pixels: 10 images providing 0.2M open-set pixels.
• test-set for closed-pixels: 500 images providing 56M pixels.
• test-set for open-pixels: 500 images providing 8.3M pixels.
B D ETAILED R ESULTS BY S TATISTICAL P IPELINE
In Figure 10 on the last page of this document, we provide detailed results of various statistical models for open-world tasks, including open-set image recognition and open-set semantic segmentation.
In the main paper, we state that we tune and select statistical models (if it has hyper-parameters to tune) on the small validation set, and report on the test set with the selected (best-performing) model.
Such hyper-parameters can be the number of means/components in kmeans and GMM models, and covariance types in GMM — “spherical”, “diagonal” and “full” denote that the covariance matrix of each Gaussian component is controlled by a single scalar, a vector and full-rank matrix, respectively.
It demonstrates that validation can reliably tune the statistical models whose performance can be translated to the test sets.
Moreover, we also record the detailed results of whether using
Table 4 and 5 list details of various methods on cross-dataset evaluation (Setup-II), supplementing Table 2. Please refer to the caption for details.
C P ERFORMANCE VS . PCA R EDUCED D IMENSION
As analyzed in the main paper, PCA is an important technique to make our pipeline lightweight by considerably reducing feature dimensions.
We study how a statistical model performs under different reduced feature dimensions by PCA.
We choose the simplistic NCM method which does not induce randomness (unlike kmeans and GMM which require random initialization for learning).
We study this through open-set image recognition under Setup-II.
To simplify the study, we choose (resized)
Cityscapes images as open-set data, i.e., we use TinyImagenet/Cityscapes images as the closed/open set.
As we use the network Res50 in the diagnostic study, the original dimension of the pooled features is 2048.
In Fig.
7, we plot the performance (AUROC) of NCM as a function of reduced dimension by PCA.
Perhaps surprisingly, PCA even improves the open-world performance while signiﬁcantly reducing feature dimension (from 2048 to 100)!
D P ERFORMANCE VS . M EMORY /C OMPUTE
As seen previously, PCA reduces the feature dimension greatly and hence makes the statistical models quite lightweight.
We now study how lightweight different statistical models can be by considering the open-world performance.
We analyze the models learned for two tasks (open-world image classiﬁcation and open-world semantic segmentation), where the OTS features have dimension(extracted from Res50) and 720 (extracted from HRNet), respectively.
We use PCA to reduce the features dimensions to 200 and 100, respectively.
We focus on NN, kmeans and GMMs, all of which operate on the PCA reduced features (with L2normalization).
NN is the straightforward baseline that memorizes all training examples to recognize open-set examples.
For GMM, we study it by specifying three types of covariance – “spherical”, “diag” and “full” meaning the covariance matrix of each Gaussian component is controlled by a single scalar, a vector or a full-rank (symmetric) matrix, respectively.
For open-set semantic segmentation, it is prohibitively space-consuming to memorize all the pixel features of the whole train set.
So we randomly sample 5000 pixels from each of the 19 classes deﬁned by Cityscapes ( ∼ 200k in total).
In Figure 8, we draw the open-world performance (AUROC) for the two tasks w.r.t the total memory cost (i.e., the required space to save a model’s parameters).
We can see that NN takes the most memory [Figure] usage, which is even more than the underlying networks.
In contrast, GMM-spherical and k-means models are signiﬁcantly more compact, i e., ∼ 0.3MB for both tasks.
Moreover, on open-world image classiﬁcation (Figure 8-left), k-means and GMM-spherical achieve much better performance than the other models.
Interestingly GMM-full achieves the best and stable performance for open-set semantic segmentation (Figure 8-right) but not for open-set image classiﬁcation (Figure 8-left).
Despite this, we note that the validation performance (as plotted here) can be nicely translated to real test sets, as shown in Figure 10).
It is worth noting that the speciﬁed PCA-reduced dimension is not optimal that an even lower dimension can lead to better open-world performance (cf.
Figure 7).
We do not exhaustively explore this in this work, but instead emphasize that our pipeline is quite lightweight that can be tuned on speciﬁc tasks, e.g., 0.6MB GMM-full compared with 250MB HRNet for semantic segmentation.
E V ISUALIZATION OF G AUSSIAN M EANS
As statistical models are interpretable, we visualize what the statistical model can capture.
To do so, we visualize per-class Gaussian means through medoid images, which are training images that have features closest to their corresponding per-class mean feature.
We show the medoid images in
Figure 9, as well as some random images sorted by the cosine similarity (i.e., Euclidean distance on
L2-normalized features) to the Gaussian means within each class.
We can see the medoid images most likely capture the canonical objects of each class, e.g., those of with “standard” pose and clean background.
F O PEN -S OURCE D EMONSTRATION
We attach our code (via three Jupyter Notebook ﬁles) to demonstrate our exploration of open-set recognition.
One can run the code with access to networks (Res50 and HRNet (Wang et al., 2019)) trained for closed-world tasks.
We are not able to upload models or pre-computed features due to space limit, but we are committed to releasing them to the public after paper notiﬁcation.
We refer readers to the Jupyter Notebook ﬁles for self-explanatory descriptions
• “demo_Open-Set-Image-Recognition-Setup-II_GMM_Res50pt_pca_L2norm.ipynb”: We show how we train, select and evaluate GMMs on cross-dataset open-set image recognition (Setup-II).
• “demo_tsne_visual_res50pt.ipynb”: We show t-SNE visualizations of OTS features of cross-dataset open-set examples (Setup-II).
This intuitively demonstrates the beneﬁt of exploiting OTS features for open-set recognition.
• “demo_open-set-semantic-segmentation.ipynb”: We demonstrate how we train and evaluate GMM under Setup-III, open-set semantic segmentation. [Table] This supplements [Table]