The Semantic Shift Benchmark
Abstract
Most benchmarks for detecting semantic distribution shift do not consider how the semantics of the training set are defined.
In other words, it is often unclear whether the ‘unseen’ images contain semantically different objects from the same distribution (e.g ‘birds’ for a model trained on ‘cats’ and ‘dogs’) or to a different distribution entirely (e.g Gaussian noise for a model trained on ‘cats’ and ‘dogs’).
In this work, we propose ‘open-set’ class splits for models trained on ImageNet-1K which come from ImageNet-21K.
Critically, we structure the open-set classes based on semantic similarity to the closed-set using the WordNet hierarchy — we create ‘Easy’ and ‘Hard’ open-set splits to allow more principled analysis of the semantic shift phenomenon.
Together with similar challenges based on FGVC datasets, these evaluations comprise the ‘Semantic Shift Benchmark’.
1. Introduction
This report outlines the Semantic Shift Benchmark (SSB), a set of evaluations which isolate semantic shift, as compared to other forms of low-level distributional shifts.
Specifically, given a training set, the SSB contains carefully curated class splits coming from unseen categories, where the unseen categories are structured for semantic similarity to the training set.
In the case of ImageNet, we select categories from
ImageNet-21K (Ridnik et al., 2021) and bin them based on their distance in the WordNet hierarchy to the regular
ImageNet-1K classes.
The benchmark is motivated by the task of open-set recognition (OSR) which is the problem of detecting if a test-time instance comes from an unseen category (Scheirer et al., 2013; Neal et al., 2018; Bendale & Boult, 2016).
However, we note that it is applicable to other problems which deal with semantic shift (Han et al., 2019; Vinyals et al., 2016). {+Correspondence to: Sagar Vaze < sagar@robots.ox.ac.uk > . ICML 
+}
Shift Happens Workshop: Crowdsourcing metrics and test datasets beyond ImageNet.
, Baltimore, Maryland, USA, 2022.
Copyright 2022 by the author(s). While this report focuses on a large-scale ImageNet benchmark, we also describe three evaluations on fine-grained datasets which have clear definitions of a semantic class.
We describe these as they follow analogous principles to the
ImageNet evaluation and may be of interest to the community, though we acknowledge they are not ‘ImageNet-scale’ evaluations.
We benchmark strong maximum logit score (MLS) and ARPL (Chen et al., 2021)
baselines on the new benchmark suite to motivate future research.
2. Benchmark Design
ImageNet.
We introduce a large-scale evaluation for category shift, with open-set splits based on semantic distances to the training set.
Specifically, we designate the original
ImageNet-1K classes for the closed-set, and choose openset classes from the disjoint set of ImageNet-21K-P (Ridnik et al., 2021).
We exploit the hierarchical, tree-like semantic structure of the ImageNet database.
For instance, the class ‘elephant’ can be labelled at multiple levels of semantic abstraction (‘elephant’, ‘placental’, ‘mammal’, ‘vertebrate’, ‘animal’).
Thus, for each pair of classes between ImageNet1K and ImageNet-21K-P, we define the semantic distance between two classes as the total path distance between their nodes in the semantic tree.
Specifically, we adopt the Leacock Chodorow Similarity (LCS)
(Fellbaum, 1998), which measures lexical semantic similarity by finding the shortest path in the WordNet graph between two concepts, and scales that value by the maximum path length.
The LCS between two concepts i and j is defined as:
[Equation]
where p ( i, j ) denotes the shortest path length between i and j , and d denotes the taxonomy depth.
We compute this distance for all i in the ImageNet-1K classes, and all j in the disjoint ImageNet-21K classes.
Finally, we select ‘Easy’ and ‘Hard’ open-set splits by sorting the total distances to the closed-set and selecting two sets of 1000 categories.
We provide examples of our open-set splits for the ImageNet dataset in Figure 1.
Fine-grained classification datasets.
Consider the properties of fine-grained visual categorization (FGVC) datasets.
[Table] These datasets are defined by an ‘entry level’ category, such as flowers (Nilsback & Zisserman, 2008) or birds (Wah et al., 2011).
Within the dataset, all classes are variants of that single category, defining a single axis of semantic variation , e
[Table]
g. , ‘bird species’ in the case of birds.
Because the axis of variation is well defined, it is reasonable to expect a classifier to learn it given a number of example classes — namely, to learn what bird species are and how they can be distinguished.
This makes the analysis of semantic shift better posed .
We propose three FGVC datasets for the SSB: CUB (Wah et al., 2011), Stanford Cars (Krause et al., 2013), FGVCAircraft (Maji et al., 2013).
These datasets come with labelled attributes ( e.g. , has bill shape::hooked in
CUB), which can be used to characterize the differences between classes and the degree of semantic shift, and hence the difficulty of the open-set problem.
Key differences from prior work Most existing semantic shift evaluations operate on small scale datasets (Neal et al., 2018; Han et al., 2019).
Compared to other large-scale evaluations, our main contribution is in our aim to explicitly capture semantic novelty .
For instance, (Sun et al., 2021) considers Places365 and SUN to be semantically different to ImageNet-1K. These evaluations do not consider how the ‘OoD’ images relate to the taxonomy or semantic classification system defined in the training data.
As such, it is not clear whether the model is responding to a true semantic shift at test-time (which is the goal of OSR) or to some other low-level distribution shift.
In contrast, our use of WordNet to identify easy and hard open-set examples respects the classification system used to build the ImageNet-1K dataset.
We note that other works have used ImageNet-21K as examples of ‘unseen categories’ with respect to ImageNet-1K (Bendale & Boult, 2016; Kumar et al., 2021).
However, we structure for semantic distance based on the WordNet hierarchy, which we suggest provides a more principled basis for analyzing ‘semantic shift’.
ImageNet-O
(Hendrycks et al., 2021) is another dataset which selects individual ‘out-ofdistribution’ images from ImageNet-21K by selecting adversarial OoD examples with a trained ResNet-50.
However, respecting the idea of novel categories , we choose entire categories of images rather than only the most adversarially challenging instances.
As a result, our proposed ‘unseen class’ splits have 50k images each — as compared to [Table] in ImageNet-O — and facilitate a truly ImageNet-scale test set (see Table 2 for all split sizes).
We also suggest that identification of semantic novelty as defined by a human taxonomy is a distinct, and possibly independent, challenge to detecting adversarially OoD examples of current models.
Finally, we note this benchmark’s distinction from works such as (Shankar et al., 2020; Hendrycks et al., 2020; Liang & Zou, 2022). Specifically, these works focus on evaluating model robustness , and construct distribution shifts in which the semantics (i.e categories) remain the same, while other features of the images (e.g background or pose) are varied. In this way, these works are orthogonal and complimentary to the Semantic Shift Benchmark.
3. Evaluation Protocol
We provide benchmark results on the Semantic Shift Benchmark for the task of open-set recognition.
In Table 1 we evaluate the ‘Maximum Logit Score’ (MLS, (Vaze et al., 2022)) and ARPL+ (Chen et al., 2021), which are two state-of-the-art methods of open-set recognition.
For the ‘known/unknown’ class decision, we report AUROC as is standard practise, as well as accuracy to allow potential gains in open-set performance to be contextualized in the closed-set accuracy of a model.
The AUROC aggregates binary classification performance across all possible choices of threshold.
We also report Open-Set Classification Rate (OSCR) (Dhamija et al., 2018) which measures the trade-off between accuracy and open-set detection rate as a threshold on the confidence of the predicted class is varied.
We report results on ‘Easy’ and ‘Hard’ splits for all datasets, combining ‘Medium’ and ‘Hard’ examples into a single bin when applicable.
For the ImageNet benchmark, we use a ResNet50 pretrained on the ImageNet-1K dataset and evaluate on the unseen classes.
We finetune the ARPL model from a pretrained ImageNet-1K checkpoint.
In fine-grained classification, it is standard to pre-train models on ImageNet-1K.
This is unsuitable for the proposed fine-grained OSR setting, as ImageNet-1K contains overlapping classes with the proposed datasets.
Instead, we pre-train the network on
Places (Zhou et al., 2017) using MoCoV2 self-supervised weights (Chen et al., 2020; Zhao et al., 2021).
We find that careful consideration of the semantics of the open-set classes leads to harder splits significantly reducing OSR performance.
This is in contrast to ‘openness’ (Scheirer et al., 2013), the current measure used to assess the difficulty of an OSR problem, dependent only on the ratio of the number of closed to open-set classes.
On ImageNet, we find the harder split leads to 5-6% worse AUROC for both methods.
We also experimented with randomly subsampling first 1K and then 10K open-set classes, finding that introducing more classes during evaluation only reduced open-set performance by around 0.6% ( ≈ 10 × less than our proposed splits).