Original: Exploring Covariate and Concept Shift for Out-of-Distribution Detection
Revised: Exploring Covariate and Concept Shift for Out-of-Distribution Detection

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Traditionally the task of out-of-distribution (OOD) detection is associated withepistemic uncertainty estimation.
Revised: Traditionally the task of out-of-distribution (OOD) detection is associated with epistemic uncertainty estimation.

Original: While Bayesian models and generative modelshave been explored to provide principled uncertainty estimation, deterministicand discriminative models cannot provide such estimation by nature.
Revised: While Bayesian models and generative models have been explored to provide principled uncertainty estimation, deterministic and discriminative models cannot provide such estimation by nature.

Original: We proposeto characterize the difﬁculty of OOD detection by the extent of distribution shiftand theoretically derive two score functions for OOD detection.
Revised: We propose to characterize the difﬁculty of OOD detection by the extent of distribution shift and theoretically derive two score functions for OOD detection.

Original: Additionaly wepropose a geometrically-inspired method (Geometric ODIN) to improve OODdetection under distribution shift with only in-distribution data
Revised: Additionaly we propose a geometrically-inspired method (Geometric ODIN) to improve OOD detection under distribution shift with only in-distribution data.

Revised: View project page at https://sites.google.com/view/geometric-decomposition and a long version of the paper at https://arxiv.org/abs/2110.15231 .

Original:  Introduction
Revised:  Introduction

Original: Bayesian inference [1, 2, 3] has been regarded as the most principled method of uncertainty modeling [4] because it explicitly models two types of uncertainty: epistemic uncertainty and aleatoricuncertainty in the form posteriors over parameters and data likelihood respectively.
Revised: Bayesian inference [1, 2, 3] has been regarded as the most principled method of uncertainty modeling [4] because it explicitly models two types of uncertainty: epistemic uncertainty and aleatoric uncertainty in the form posteriors over parameters and data likelihood respectively.

Original: Practically, epistemic uncertainty is often assessed by a model’s out-of-distribution (OOD) detection performance [5]or calibration [6, 7], while aleatoric uncertainty can be assessed by in-distribution error detection [5].
Revised: Practically, epistemic uncertainty is often assessed by a model’s out-of-distribution (OOD) detection performance [5] or calibration [6, 7] while aleatoric uncertainty can be assessed by in-distribution error detection [5].

Original: Recent attempts [6, 7] to model uncertainty using deterministic models failed to disentangle thesetwo uncertainties due to their non-Bayesian nature.
Revised: Recent attempts [6, 7] to model uncertainty using deterministic models failed to disentangle these two uncertainties due to their non-Bayesian nature.

Original: However, it is still possible to capture them empirically in a deterministic model using a combination of density estimation and softmax-entropy [5].
Revised: However, it is still possible to capture them empirically in a deterministic model using a combination of density estimation and softmax-entropy [5].

Original: This leaves us the question: how to approach OOD detection/calibration for deterministic (asopposed to Bayesian) and discriminative (as opposed to generative) models?
Revised: This leaves us the question: how to approach OOD detection/calibration for deterministic (as opposed to Bayesian) and discriminative (as opposed to generative) models?

Original: This is arguably themost widely used class of models due to its speed (compared to Bayesian models) and simplicity(compared to generative models).
Revised: This is arguably the most widely used class of models due to its speed (compared to Bayesian models) and simplicity (compared to generative models).

Original: It seems that the conventional association of OOD data withepistemic uncertainty [3] fails under the scope of this type of models and a different perspectiveis needed to analyze them.
Revised: It seems that the conventional association of OOD data with epistemic uncertainty [3] fails under the scope of this type of models and a different perspective is needed to analyze them.

Original: A suitable alternative perspective is distribution shift [8].
Revised: A suitable alternative perspective is distribution shift [8].

Original: Intuitively,out-of-distribution data refers to data that are sampled from distributions different from the trainingdistribution.
Revised: Intuitively, out-of-distribution data refers to data that are sampled from distributions different from the training distribution.

Original: There are two dominant shift types: covariate shift and concept shift .
Revised: There are two dominant shift types: covariate shift and concept shift .

Original: The former usuallyrefers to change in style, e.g., clean vs. noised , and the latter refers to change in semantics, dogvs. leopard .
Revised: The former usually refers to change in style, e.g., clean vs. noised , and the latter refers to change in semantics, dog vs. leopard .

Original: We propose to characterize OOD datasets by the degree of shifts in each dimension.
Revised: We propose to characterize OOD datasets by the degree of shifts in each dimension.

Original: While existing works do not distinguish between different types of OOD datasets, these two typesof shifts should be examined and addressed separately.
Revised: While existing works do not distinguish between different types of OOD datasets, these two types of shifts should be examined and addressed separately.

Original: We follow [8] for the formal deﬁnition ofdataset shift, covariate shift and concept shift.
Revised: We follow [8] for the formal deﬁnition of dataset shift, covariate shift and concept shift.

Original: Let X ∈ R D denote the covariate which is the inputor feature and Y ∈ R denote the output label.
Revised: Let X ∈ R D denote the covariate which is the input or feature and Y ∈ R denote the output label.

Original: Dataset shift happens when the training joint distribution is not equal to the testing joint distribution P tr ( X, Y ) (cid:54)
Revised: Dataset shift happens when the training joint distribution is not equal to the testing joint distribution P tr ( X, Y ) (cid:54)

Original: =
Revised: =

Original: P tst ( X, Y ) .
Revised: P tst ( X, Y ) .

Original: Speciﬁcally, covariateshift appears when P tr ( Y | X )
Revised: Speciﬁcally, covariate shift appears when P tr ( Y | X )

Original: =
Revised: =

Original: P tst ( Y | X ) and P tr ( X ) (cid:54) =
Revised: P tst ( Y | X ) and P tr ( X ) (cid:54) =

Original: P tst ( X ) .
Revised: P tst ( X ) .

Original: Concept shift appears when
Revised: Concept shift appears when

Original: P tr ( Y | X ) (cid:54) =
Revised: P tr ( Y | X ) (cid:54) =

Original: P tst ( Y | X ) and P tr ( X ) =
Revised: P tst ( Y | X ) and P tr ( X ) =

Original: P tst ( X ) .
Revised: P tst ( X ) .

Original: The ﬁrst goal of this paper is to derive scorefunctions to reﬂect the shift in either P ( X ) or P ( Y | X ) .
Revised: The ﬁrst goal of this paper is to derive score functions to reﬂect the shift in either P ( X ) or P ( Y | X ) .

Original: We denote the score function that reﬂectschanges from P tr ( X ) to P tst ( X ) as covariate shift score function : g ( x ) : R D
Revised: We denote the score function that reﬂects changes from P tr ( X ) to P tst ( X ) as covariate shift score function : g ( x ) : R D

Original: → R and thescore function that reﬂects changes from P tr ( Y | X ) to P tst ( Y | X ) as concept shift score function : h ( y, x ) : R × R D
Revised: → R and the score function that reﬂects changes from P tr ( Y | X ) to P tst ( Y | X ) as concept shift score function : h ( y, x ) : R × R D

Original: → R .
Revised: → R .

Original: The second goal is to improve the sensitivity of these scores to theircorresponding distribution shift.
Revised: The second goal is to improve the sensitivity of these scores to their corresponding distribution shift.

Original: Speciﬁcally, we propose a geometrically inspired o ut-of-distributiond etection method with only in -distribution data ( Geometric ODIN ).
Revised: Speciﬁcally, we propose a geometrically inspired o ut-of-distribution d etection method with only in -distribution data ( Geometric ODIN ).

Original: We use CIFAR10C [9] and [-anew CIFAR
-]
Revised: We use CIFAR10C [9] and [-anew CIFAR
-]a new CIFAR100 Splits dataset to investigate gradual covariate shift and concept respectively.  Related Work Out-of-Distribution (OOD) detection methods can be largely divided into two camps depending on whether they require OOD data during training. [10, 11, 12] leverages anomalous data in training.

Revised: Our method belongs to the class of methods that do not assume the availability of OOD data during training.

Revised: [13] uses the maximum softmax probability (MSP) to detect incorrect predictions and

Original: Splits dataset to investigate gradual covariate shift and concept respectively.
Revised: OOD data. [14] proposes to use Mahalanobis distance by ﬁtting a Gaussian mixture model (GMM) in the feature space. [5] uses log density of the GMM model instead. [15] uses an energy score as the uncertainty metric.

Revised: ODIN [16] uses a combination of input processing and post-training tuning to improve OOD detection performance.

Revised: Generalized ODIN [17] (also [18]) includes an additional network in the last layer to improve OOD detection during training.

Revised: There are other interesting OOD detection approaches without OOD data such as using contrastive learning with various transformations [19, 20], training a deep ensemble of multiple models [21] and leveraging large pretrained models [22].

Revised: They require extended training time, hyperparameter tuning and careful selections of transformations, whereas our method does not introduce any hyperparameters and has negligible inﬂuence on standard cross-entropy training time.

Original:  Method2.1 Covariate and Concept Score Functions
Revised:  Method 3.1 Covariate and Concept Score Functions

Original: In this section, we theoretically derive two score functions, g ( x ) and h ( y, x ) , based on the KLdivergence between a uniform distribution U and a predicted distribution P ∈ R M , where M is thenumber of classes.
Revised: In this section, we theoretically derive two score functions, g ( x ) and h ( y, x ) , based on the KLdivergence between a uniform distribution U and a predicted distribution P ∈ R M , where M is the number of classes.

Original: By starting from KL-divergence, we hinge the subsequent derivation of scorefunctions on a physical meaningful uncertainty measure, i.e., how far the predicted distribution isfrom a uniform distribution.
Revised: By starting from KL-divergence, we hinge the subsequent derivation of score functions on a physical meaningful uncertainty measure, i.e., how far the predicted distribution is from a uniform distribution.

Original: This relationship ensures a natural interpretation of score functionsbecause predictions on distribution shifted data should have larger uncertainty, i.e. smaller distancefrom uniform.
Revised: This relationship ensures a natural interpretation of score functions because predictions on distribution shifted data should have larger uncertainty, i.e. smaller distance from uniform.

Original: We are speciﬁcally interested in softmax-linear models for classiﬁcation.
Revised: We are speciﬁcally interested in softmax-linear models for classiﬁcation.

Original: Theytypically consist of a feature extractor and a linear layer followed by a softmax activation.
Revised: They typically consist of a feature extractor and a linear layer followed by a softmax activation.

Original: Letf ∈ R D denote a feature vector from the feature extractor 1 .
Revised: Let f ∈ R D denote a feature vector from the feature extractor 1 .

Original: The output of the linear layer, i.e.,logits, l ∈ R M = < f , W > is deﬁned as the inner product between the feature vector and a weightmatrix in the linear layer.
Revised: The output of the linear layer, i.e., logits, l ∈ R M = < f , W > is deﬁned as the inner product between the feature vector and a weight matrix in the linear layer.

Original: Let l i = (cid:107) f
Revised: Let l i = (cid:107) f

Original: (cid:107) 2 (cid:107) w i (cid:107) 2 cos φ i denote the i th logit, P i = 
Revised: (cid:107) 2 (cid:107) w i (cid:107) 2 cos φ i denote the i th logit, P i = exp

Original: [-exp l i (cid:80) Mj =
-]
Revised: [-exp l i (cid:80) Mj =
-][Equation]

Original: exp l j denotethe predicted probability of the i th class.
Revised: denote the predicted probability of the i th class.

Original: The KL-divergence KL ( U || P ) can be written as following:
Revised: The KL-divergence KL ( U || P ) can be written as following:

Original: [Equation]
Revised: [Equation]

Original: Now we can use the inequality property of Log-Sum-Exp (LSE) 2 functions in Eq.
Revised: Now we can use the inequality property of Log-Sum-Exp (LSE) 2 functions in Eq.

Original: 2 to bound Eq.
Revised: 2 to bound Eq.

Original: [Equation]
Revised: [Equation]

Original: Therefore the KL-divergence (Eq. 1) can be bounded as follows:
Revised: Therefore the KL-divergence (Eq. 1) can be bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: where U = max j l j − 1 M (cid:80) Mi
Revised: where U = max j l j − 1 M (cid:80) Mi

Original: =1 l i .
Revised: =1 l i .

Original: U can be further decomposed into two multiplicative componentsby plugging in the deﬁnition of logits l i :
Revised: U can be further decomposed into two multiplicative components by plugging in the deﬁnition of logits l i :

Original: [Equation]
Revised: [Equation]

Original: We deﬁne the covariate shift score function as g ( x ) (cid:44) (cid:107) f (cid:107) 2 because the norm of a feature vectoris the sum of squared activation values and only depends on the input.
Revised: We deﬁne the covariate shift score function as g ( x ) (cid:44) (cid:107) f (cid:107) 2 because the norm of a feature vector is the sum of squared activation values and only depends on the input.

Original: Intuitively, activation ofa neural network on covariate-shifted data should be weaker than in-distribution data.
Revised: Intuitively, activation of a neural network on covariate-shifted data should be weaker than in-distribution data.

Original: Therefore,g ( x ) assigns a higher value to in-distribution data than to OOD data.
Revised: Therefore, g ( x ) assigns a higher value to in-distribution data than to OOD data.

Original: We deﬁne the concept shiftscore function ash ( y, x )
Revised: We deﬁne the concept shift score function as

Original: [Equation]
Revised: [Equation]

Original: w i (cid:107) 2 cos φ i because it is the differencebetween the cosine distance of the predicted class and the average cosine distance of all classes anddepends on both the input and ﬁnal class membership, assigned by the max operator.
Revised: 2 cos φ i because it is the difference between the cosine distance of the predicted class and the average cosine distance of all classes and depends on both the input and ﬁnal class membership, assigned by the max operator.

Original: Intuitively,class assignment should be less obvious under concept shift and the difference should be small.
Revised: Intuitively, class assignment should be less obvious under concept shift and the difference should be small.

Original: Consequently, h ( y, x ) assigns a higher value to in-distribution data than to OOD data.
Revised: Consequently, h ( y, x ) assigns a higher value to in-distribution data than to OOD data.

Original: In retrospect,our deﬁnition of covariate shift and concept shift scores supports existing ﬁndings that the features norms correspond to intra-class variance and angles reﬂect inter-class variation [11].
Revised: In retrospect, our deﬁnition of covariate shift and concept shift scores supports existing ﬁndings that the features norms correspond to intra-class variance and angles reﬂect inter-class variation [23].

Original: Intuitively,covariate shift represents non-semantic change within a speciﬁc class, i.e., intra-class variance;concept shift represents semantic changes, i.e, inter-class variation.
Revised: Intuitively, covariate shift represents non-semantic change within a speciﬁc class, i.e., intra-class variance; concept shift represents semantic changes, i.e, inter-class variation.

Original: Here we formalize the intuitionand observations in [11] as score functions derived analytically from a KL-divergence viewpoint.
Revised: Here we formalize the intuition and observations in [23] as score functions derived analytically from a KL-divergence viewpoint.

Original: More importantly, the combined score function U (Eq. 4) carries a physical meaning: it boundsthe KL-divergence between a uniform distribution and the predictive distribution.
Revised: More importantly, the combined score function U (Eq. 4) carries a physical meaning: it bounds the KL-divergence between a uniform distribution and the predictive distribution.

Original: Intuitively, asmall U indicates large uncertainty because U upper bounds KL ( U || P ) , and a large U indicatessmall uncertainty because it also appears in the lower bound.
Revised: Intuitively, a small U indicates large uncertainty because U upper bounds KL ( U || P ) , and a large U indicates small uncertainty because it also appears in the lower bound.

Original: A similar scoring function, S ( x )=(cid:107)
Revised: A similar scoring function, S ( x )

Original:  f (cid:107) 2 max j (cid:107) w j (cid:107) 2 cos φ j , is used in [12], but is only empirically motivated based on observationswith limited analytical insights such as its relationship to uncertainty and the functionality of each ofcomponents.
Revised: = (cid:107) f (cid:107) 2 max j (cid:107) w j (cid:107) 2 cos φ j , is used in [20], but is only empirically motivated based on observations with limited analytical insights such as its relationship to uncertainty and the functionality of each of components.

Original: In contrast, our derivation clearly shows the relationship between these score functionsand the KL-divergence, which is as an uncertainty measure, and disentangles their roles.
Revised: In contrast, our derivation clearly shows the relationship between these score functions and the KL-divergence, which is as an uncertainty measure, and disentangles their roles.

Original: 2.2 Geometric Out-of-Distribution Detection with In-Distribution Data
Revised: 3.2 Geometric Out-of-Distribution Detection with In-Distribution Data

Original: As derived in Eq. 4, the covariate score is a function of feature norms and the concept score is afunction of feature angles.
Revised: As derived in Eq. 4, the covariate score is a function of feature norms and the concept score is a function of feature angles.

Original: Consequently, improving the sensitivity of feature norms and featureangles to data shifts seems to be the natural next step to improve OOD detection.
Revised: Consequently, improving the sensitivity of feature norms and feature angles to data shifts seems to be the natural next step to improve OOD detection.

Original: Therefore, weadopt Geometric Sensitivity Decomposition (GSD) [13] to improve sensitivity to covariate andconcept shifts.
Revised: Therefore, we adopt Geometric Sensitivity Decomposition (GSD) [24] to improve sensitivity to covariate and concept shifts.

Original: Speciﬁcally, GSD improves sensitivity by extracting sensitive components from norms(cid:107) f ∗ (cid:107) 2 3 and angles | φ ∗ i | through a decomposition of them into: an instance-independent scalar andan instance-dependent variance factor as shown in Eq.
Revised: Speciﬁcally, GSD improves sensitivity by extracting sensitive components from norms (cid:107) f ∗ (cid:107) 2 3 and angles | φ ∗ i | through a decomposition of them into: an instance-independent scalar and an instance-dependent variance factor as shown in Eq.

Original: Instance-independent scalars C f and C φminimize the loss on the training set and instance-dependent components f
Revised: Instance-independent scalars C f and C φ minimize the loss on the training set and instance-dependent components f

Original: and φ i account sensitivelyfor variances in samples.
Revised: and φ i account sensitively for variances in samples.

Original: [Equation]
Revised: [Equation]

Original: With the decomposed components, the original logit l ∗ i 
Revised: With the decomposed components, the original logit l ∗ i =

Original: = (cid:107) f ∗
Revised: [Equation] f ∗

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: w i
Revised: w i

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  φ ∗ i , can be written as:
Revised: 2 cos φ ∗ i , can be written as:

Original: [Equation]
Revised: [Equation]

Original: where l i denote the new i th logit.
Revised: where l i denote the new i th logit.

Original: In Eq.
Revised: In Eq.

Original: 6, the new 4 feature f is a direct output of a feature extractor,and is modiﬁed by α and β .
Revised: 6, the new 4 feature f is a direct output of a feature extractor, and is modiﬁed by α and β .

Original: Note that the calculation of score functions in Sec.
Revised: Note that the calculation of score functions in Sec.

Original: 2.1 only uses thefeature and is independent of α and β .
Revised: 3.1 only uses the feature and is independent of α and β .

Original: Because cos C φ and C f are instance-independent, we can parametrize them separately from the mainnetwork.
Revised: Because cos C φ and C f are instance-independent, we can parametrize them separately from the main network.

Original: Unlike GSD which parametrizes them as instance-independent scalars, inspired by [14, 15],we make α ( f ) and β ( f ) instance-dependent scalars and use a single linear layer to learn them.
Revised: Unlike GSD which parametrizes them as instance-independent scalars, inspired by [17, 18], we make α ( f ) and β ( f ) instance-dependent scalars and use a single linear layer to learn them.

Original: Toenforce numerical constraints, i.e., 0 < α < 1 and β > 0 , α ( f ) uses a sigmoid activation and β ( f )uses a softplus activation .
Revised: To enforce numerical constraints, i.e., 0 < α < 1 and β > 0 , α ( f ) uses a sigmoid activation and β ( f ) uses a softplus activation .

Original: Finally, the relaxed output is:
Revised: Finally, the relaxed output is:

Original: [Equation]
Revised: [Equation]

Original: Now the new predicted norm (cid:107) f (cid:107) 2 and angle φ
Revised: Now the new predicted norm (cid:107) f (cid:107) 2 and angle φ

Original: i are more sensitive to input changes because theyencode variances in samples as shown in Eq.
Revised: i are more sensitive to input changes because they encode variances in samples as shown in Eq.

Original: Therefore, including β (related to norms) improvessensitivity to covariate shift and including α (related to angles) improves sensitivity to conceptshift.
Revised: Therefore, including β (related to norms) improves sensitivity to covariate shift and including α (related to angles) improves sensitivity to concept shift.

Original: Note that, under this construction, Generalized ODIN [14] is a special case of our proposedmethod.
Revised: Note that, under this construction, Generalized ODIN [17] is a special case of our proposed method.

Original: Generalized ODIN only includes the α ( f ) which only improves angle sensitivity but notnorm sensitivity.
Revised: Generalized ODIN only includes the α ( f ) which only improves angle sensitivity but not norm sensitivity.

Original: Unlike [14]’s probabilistic perspective 5 , our model builds on a geometric perspectiveand captures both covariate and concept shifts by improving norm and angle sensitivity.
Revised: Unlike [17]’s probabilistic perspective 5 , our model builds on a geometric perspective and captures both covariate and concept shifts by improving norm and angle sensitivity.

Original: The newmodel can be trained identically as the vanilla network without additional hyperparameter tuning. 
Revised: The new model can be trained identically as the vanilla network without additional hyperparameter tuning. 3.3 CIFAR100 Splits

Revised: [Table]

Revised: shark forest pine dinosaur boy porcupine wolf road

Revised: [Table]

Revised: Group girl rocket man tiger bee tank

Original: [Figure]
Revised: [Table]

Revised: Group willow worm chimpanzee skunk cup mushroom oak

Revised: [Table]

Revised: Group castle can bridge lobster house bed

Revised: [Table]

Revised: Group palm streetcar pepper keyboard bottle seal

Revised: [Table]

Revised: Group ﬂatﬁsh apple orange plate table

Revised: [Table]

Revised: wardrobe

Revised: [Table]

Revised: Special CIFAR100 Splits for Gradual Concept Shift While it is natural to associate covariate shift with increasing degrees of image corruption, ﬁnding a dataset to benchmark gradual concept shift is not straightforward because concept shift is traditionally thought as binary: overlapping or non-overlapping.

Revised: However, not all non-overlapping labels are the same.

Revised: For example, pickup truck (CIFAR100) is much closer to truck (CIFAR10) than sunﬂowers (CIFAR100) is semantically.

Revised: To create this gradual concept/semantic shift, we propose to divide the CIFAR100 dataset intosub-datasets with increasing conceptual difference from CIFAR10 classes.

Revised: Specially, we use Glove

Original: [Figure] 
Revised: [Figure] [Figure]

Revised: covariate shift ( α - β model) yields the best performance as shown in Fig.

Revised: 2b.

Revised: word embeddings [25] trained on the entire wikipedia2014 and Gigaword5 [26] to measure semantic closeness (inner product) between CIFAR100 and CIFAR10 classes.

Revised: The result is 10 subdatasets split from CIFAR100.

Revised: We list the splits in Tab. 1.

Original:  Experiments
Revised:  Experiments

Original: [Figure]

Original: The g ( x ) score captures covariate shift.
Revised: The g ( x ) score captures covariate shift.

Original: We compare the α - β , α -only, β -only variants and thevanilla model on CIFAR10C [9] corrupted by motion blur in Fig.
Revised: We compare the α - β , α -only, β -only variants and the vanilla model on CIFAR10C [9] corrupted by motion blur in Fig.

Original: 2 with increasing degrees of noise.
Revised: 2 with increasing degrees of noise.

Original: From 2a, we observe that 1) as covariate shift severity increases, OOD detection becomes easierbecause AUROC increases with increasing severity. 2) in practice, covariate shift does not happen inisolation from concept shift and vice versa because AUROC using either g ( x ) or h ( y, x ) increases.3) the vanilla model is more sensitive to the concept shift component because h ( y, x )
Revised: From 2a, we observe that 1) as covariate shift severity increases, OOD detection becomes easier because AUROC increases with increasing severity. 2) in practice, covariate shift does not happen in isolation from concept shift and vice versa because AUROC using either g ( x ) or h ( y, x ) increases.

Original:  > g ( x ) inthe vanilla model plot even though covariate shift is the dominant distribution shift in this example.4) when sensitivity to both covariate and concept shift is improved , the α - β model becomes moresensitive to the covariate shift component.
Revised: 3) the vanilla model is more sensitive to the concept shift component because h ( y, x ) > g ( x ) in the vanilla model plot even though covariate shift is the dominant distribution shift in this example. 4) when sensitivity to both covariate and concept shift is improved , the α - β model becomes more sensitive to the covariate shift component.

Original: This suggests that the dominant shift in this example isindeed covariate shift.
Revised: This suggests that the dominant shift in this example is indeed covariate shift.

Original: From Fig.
Revised: From Fig.

Original: 2b, we observe that the α - β model outperforms the β -only modelusing the combined score function U .
Revised: 2b, we observe that the α - β model outperforms the β -only model using the combined score function U .

Original: This suggests that improving sensitivity to both shifts andusing U yield the best OOD detection.
Revised: This suggests that improving sensitivity to both shifts and using U yield the best OOD detection.

Original: The h ( y, x ) score captures concept shift.
Revised: The h ( y, x ) score captures concept shift.

Original: To investigategradual concept shift, we propose to divide the CIFAR100 dataset into 10 sub-datasets with increasingconceptual difference from CIFAR10 classes (See Appendix 5.1).

Original: Following the previous section, webenchmark the α - β , α -only, β -only variants and the vanilla model on the newly created [-CIFAR
-] Splits.
Revised: Following the previous section, we benchmark the α - β , α -only, β -only variants and the vanilla model on the newly created [-CIFAR
-]CIFAR100 Splits.

Original: From Fig.
Revised: From Fig.

Original: 1a, we observe that 1) as concept shift severity increases, OOD detection becomeseasier because AUROC increases with increasing severity.
Revised: 1a, we observe that 1) as concept shift severity increases, OOD detection becomes easier because AUROC increases with increasing severity.

Original: 2) both concept shift and covariate shit arepresent because AUROC using either h ( y, x ) or g ( x ) increases.
Revised: 2) both concept shift and covariate shit are present because AUROC using either h ( y, x ) or g ( x ) increases.

Original: 3) the vanilla model is dominantlymore sensitive to the concept shift component because concept shift is the dominant distributionshift in CIFAR100 Splits by construction and vanilla ResNet is more sensitive to concept shift.
Revised: 3) the vanilla model is dominantly more sensitive to the concept shift component because concept shift is the dominant distribution shift in CIFAR100 Splits by construction and vanilla ResNet is more sensitive to concept shift.

Original: Thesame behavior is also observed on CIFAR10C. 4) when sensitivity to both shifts is improved, the α - βmodel is still more sensitive to concept shift ( h ( y, x )
Revised: The same behavior is also observed on CIFAR10C. 4) when sensitivity to both shifts is improved, the α - β model is still more sensitive to concept shift ( h ( y, x )

Original: > g ( x ) ).
Revised: > g ( x ) ).

Original: This reconﬁrms that the dominant shifttype is indeed concept shift.
Revised: This reconﬁrms that the dominant shift type is indeed concept shift.

Original: From Fig.
Revised: From Fig.

Original: 1b, we observe that all three variants perform similarly andall outperform the vanilla model.
Revised: 1b, we observe that all three variants perform similarly and all outperform the vanilla model.

Original: Combined with previous observations that the dataset has strongconcept shift and the vanilla model is already very sensitive to concept shift, improving sensitivity tothe covariate shift component yields equally good performance as improving sensitivity to both shifts.
Revised: Combined with previous observations that the dataset has strong concept shift and the vanilla model is already very sensitive to concept shift, improving sensitivity to the covariate shift component yields equally good performance as improving sensitivity to both shifts.

Original: We provide results on standard benchmarksagainst state-of-the-art methods in Appendix [-5.
-]
Revised: We provide results on standard benchmarks in Appendix [-5.
-]6.1.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we propose to characterize the spectrum of out-of-distribution data from the perspectiveof dataset shift, speciﬁcally covariate and concept shift.
Revised: In this work, we propose to characterize the spectrum of out-of-distribution data from the perspective of dataset shift, speciﬁcally covariate and concept shift.

Original: This categorization provides a differentperspective to study OOD detection for deterministic and discriminative models.
Revised: This categorization provides a different perspective to study OOD detection for deterministic and discriminative models.

Original: At representationlevel, we derive two score functions that represent and capture each shift separately.
Revised: At representation level, we derive two score functions that represent and capture each shift separately.

Original: At modelinglevel, inspired by these score functions, we propose a geometrically-inspired method, Geometric
Revised: At modeling level, inspired by these score functions, we propose a geometrically-inspired method, Geometric

Original: ODIN, to improve a model’s sensitivity to both shift.
Revised: ODIN, to improve a model’s sensitivity to both shift.

Original: Appendix

Original: 5.1 CIFAR100 Splits

Original: [Table]

Original: Special CIFAR100 Splits for Gradual Concept Shift While it is natural to associate covariateshift with increasing degrees of image corruption, ﬁnding a dataset to benchmark gradual conceptshift is not straightforward because concept shift is traditionally thought as binary: overlappingor non-overlapping.

Original: However, not all non-overlapping labels are the same.

Original: For example, pickuptruck (CIFAR100) is much closer to truck (CIFAR10) than sunﬂowers (CIFAR100) is semantically.

Original: To create this gradual concept/semantic shift, we propose to divide the CIFAR100 dataset into 10sub-datasets with increasing conceptual difference from CIFAR10 classes.

Original: [-Specially, we use Gloveword embeddings [16] trained on the entire wikipedia2014 and Gigaword
-]
Revised: [-Specially, we use Gloveword embeddings [16] trained on the entire wikipedia2014 and Gigaword
-]

Original: [17] to measure semanticcloseness (inner product) between CIFAR100 and CIFAR10 classes.

Original: The result is 10 subdatasetssplit from CIFAR100.

Original: We list the splits in Tab. 1.

Original: Our experiments demonstrate that the conceptualdifference measured by semantic similarity in language translates to a spectrum of near OOD datasetsmeasured by OOD detection difﬁculty in the image space.

Original:  5.2 Out-of-Distribution Detection Results
Revised:  Appendix 6.1 Out-of-Distribution Detection Results

Original: The α - β model is the best.
Revised: The α - β model is the best.

Original: In Tab. 2, we present OOD detection results against state-of-the-artmethods on existing near and far OOD categorization.
Revised: In Tab. 2, we present OOD detection results against state-of-the-art methods on existing near and far OOD categorization.

Original: For near OOD detection under strong conceptshift, CIFAR10 (ID) vs. CIFAR100 (OOD), both our α -only and α - β variants achieve the the bestperformance.
Revised: For near OOD detection under strong concept shift, CIFAR10 (ID) vs. CIFAR100 (OOD), both our α -only and α - β variants achieve the the best performance.

Original: This demonstrates that α ( x ) improves the sensitivity of angles and hence the sensitivityto concept-shifted data.
Revised: This demonstrates that α ( x ) improves the sensitivity of angles and hence the sensitivity to concept-shifted data.

Original: For near OOD detection under strong covariate shift, CIFAR10 (ID) vs.
Revised: For near OOD detection under strong covariate shift, CIFAR10 (ID) vs.

Original: CIFAR10C (OOD), the α - β variant achieves the the best performance.
Revised: CIFAR10C (OOD), the α - β variant achieves the the best performance.

Original: This suggests that β ( x )improves the sensitivity of norms and hence the sensitivity to covariate-shifted data.
Revised: This suggests that β ( x ) improves the sensitivity of norms and hence the sensitivity to covariate-shifted data.

Original: In [-CIFAR
 [Table]-] (ID) vs. CIFAR10/CIFAR100C (OOD) experiments, the performance of the α -only and α - β variantsare within variance and is close to some other compared methods, we can not make clear observationsfrom those experiments 6 .
Revised: In [-CIFAR
 [Table]-]CIFAR (ID) vs. CIFAR10/CIFAR100C (OOD) experiments, the performance of the α -only and α - β variants are within variance and is close to some other compared methods, we can not make clear observations from those experiments 6 .

Original: For far OOD detection, CIFAR10/CIFAR100 (ID) vs. SVHN (OOD), theα - β model achieves state-of-the-art performance.
Revised: For far OOD detection, CIFAR10/CIFAR100 (ID) vs. SVHN (OOD), the α - β model achieves state-of-the-art performance.

Original: This reconﬁrms that β ( x ) improves sensitivity tocovariate-shifted data, because SVHN has both covariate and concept shifts compared to the CIFARdatasets, and the α - β model outperforms the α -only variant, which only improves on concept shift,by a noticeable margin.
Revised: This reconﬁrms that β ( x ) improves sensitivity to covariate-shifted data, because SVHN has both covariate and concept shifts compared to the CIFAR datasets, and the α - β model outperforms the α -only variant, which only improves on concept shift, by a noticeable margin.

Original: In terms of score functions, the best performing one for the α - β model is
Revised: In terms of score functions, the best performing one for the α - β model is

Original: U , which is a product of g ( x ) and h ( y, x ) (Sec. 2.1), while that of the α -only model is h ( y, x ) .
Revised: U , which is a product of g ( x ) and h ( y, x ) (Sec. 3.1), while that of the α -only model is h ( y, x ) .

Original: Thisshows that depending on which component is more sensitive, different scoring functions are preferred.
Revised: This shows that depending on which component is more sensitive, different scoring functions are preferred.

Original: When the sensitivity of both norms and angles are improved, as in the α - β variant, the combinedscore function U performs well under different distribution shifts.
Revised: When the sensitivity of both norms and angles are improved, as in the α - β variant, the combined score function U performs well under different distribution shifts.