Original: M ULTISCALE S CORE M ATCHING FOR O UT - OFD ISTRIBUTION D ETECTION
Revised: M ULTISCALE S CORE M ATCHING FOR O UT - OFD ISTRIBUTION D ETECTION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present a new methodology for detecting out-of-distribution (OOD) images by utilizing norms of the score estimates at multiple noise scales.
Revised: We present a new methodology for detecting out-of-distribution (OOD) images by utilizing norms of the score estimates at multiple noise scales.

Original: A score is deﬁned to be the gradient of the log density with respect to the input data.
Revised: A score is deﬁned to be the gradient of the log density with respect to the input data.

Original: Our methodology is completely unsupervised and follows a straight forward training scheme.
Revised: Our methodology is completely unsupervised and follows a straight forward training scheme.

Original: First, we train a deep network to estimate scores for L levels of noise.
Revised: First, we train a deep network to estimate scores for L levels of noise.

Original: Once trained, we calculate the noisy score estimates for N in-distribution samples and take the L2norms across the input dimensions (resulting in an N x L matrix).
Revised: Once trained, we calculate the noisy score estimates for N in-distribution samples and take the L2norms across the input dimensions (resulting in an N x L matrix).

Original: Then we train an auxiliary model (such as a Gaussian Mixture Model) to learn the in-distribution spatial regions in this L -dimensional space.
Revised: Then we train an auxiliary model (such as a Gaussian Mixture Model) to learn the in-distribution spatial regions in this L -dimensional space.

Original: This auxiliary model can now be used to identify points that reside outside the learned space.
Revised: This auxiliary model can now be used to identify points that reside outside the learned space.

Original: Despite its simplicity, our experiments show that this methodology signiﬁcantly outperforms the stateof-the-art in detecting out-of-distribution images.
Revised: Despite its simplicity, our experiments show that this methodology signiﬁcantly outperforms the stateof-the-art in detecting out-of-distribution images.

Original: For example, our method can effectively separate CIFAR-10 (inlier) and SVHN (OOD) images, a setting which has been previously shown to be difﬁcult for deep likelihood models.
Revised: For example, our method can effectively separate CIFAR-10 (inlier) and SVHN (OOD) images, a setting which has been previously shown to be difﬁcult for deep likelihood models.

Revised: We make our code and results publicly available on Github 1 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Modern neural networks do not tend to generalize well to out-of-distribution samples.
Revised: Modern neural networks do not tend to generalize well to out-of-distribution samples.

Original: This phenomenon has been observed in both classiﬁer networks (Hendrycks & Gimpel (2017);
Revised: This phenomenon has been observed in both classiﬁer networks (Hendrycks & Gimpel (2017);

Original: Nguyen et al.
Revised: Nguyen et al.

Original: ; Szegedy et al.
Revised: ; Szegedy et al.

Original: and deep likelihood models (Nalisnick et al.
Revised: and deep likelihood models (Nalisnick et al.

Original: ; Hendrycks et al.
Revised: ; Hendrycks et al.

Original: ; Ren et al.
Revised: ; Ren et al.

Original: This certainly has implications for AI safety (Amodei et al.
Revised: This certainly has implications for AI safety (Amodei et al.

Original: , as models need to be aware of uncertainty when presented with unseen examples.
Revised: , as models need to be aware of uncertainty when presented with unseen examples.

Original: Moreover, an out-ofdistribution detector can be applied as an anomaly detector.
Revised: Moreover, an out-ofdistribution detector can be applied as an anomaly detector.

Original: Ultimately, our research is motivated by the need for a sensitive outlier detector that can be used in a medical setting.
Revised: Ultimately, our research is motivated by the need for a sensitive outlier detector that can be used in a medical setting.

Original: Particularly, we want to identify atypical morphometry in early brain development.
Revised: Particularly, we want to identify atypical morphometry in early brain development.

Original: This requires a method that is generalizable to highly variable, high resolution, unlabeled real-world data while being sensitive enough to detect an unspeciﬁed, heterogeneous set of atypicalities.
Revised: This requires a method that is generalizable to highly variable, high resolution, unlabeled real-world data while being sensitive enough to detect an unspeciﬁed, heterogeneous set of atypicalities.

Original: To that end, we propose multiscale score matching to effectively detect out-of-distribution samples.
Revised: To that end, we propose multiscale score matching to effectively detect out-of-distribution samples.

Original: Hyv¨arinen (2005) introduced score matching as a method to learn the parameters of a nonnormalized probability density model, where a score is deﬁned as the gradient of the log density with respect to the data.
Revised: Hyv¨arinen (2005) introduced score matching as a method to learn the parameters of a nonnormalized probability density model, where a score is deﬁned as the gradient of the log density with respect to the data.

Original: Conceptually, a score is a vector ﬁeld that points in the direction where the log density grows the most.
Revised: Conceptually, a score is a vector ﬁeld that points in the direction where the log density grows the most.

Original: The authors mention the possibility of matching scores via a nonparametric model but circumvent this by using gradients of the score estimate itself.
Revised: The authors mention the possibility of matching scores via a nonparametric model but circumvent this by using gradients of the score estimate itself.

Original: However,
Revised: However,

Original: Vincent (2011) later showed that the objective function of a denoising autoencoder (DAE) is equivalent to matching the score of a non-parametric Parzen density estimator of the data.
Revised: Vincent (2011) later showed that the objective function of a denoising autoencoder (DAE) is equivalent to matching the score of a non-parametric Parzen density estimator of the data.

Original: Thus, DAEs provide a methodology for learning score estimates via the objective:
Revised: Thus, DAEs provide a methodology for learning score estimates via the objective:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Here s θ ( x ) is the score network being trained to estimate the true score ∇ x log p data ( x ) , and q σ (˜ x ) = (cid:82)
Revised: Here s θ ( x ) is the score network being trained to estimate the true score ∇ x log p data ( x ) , and q σ (˜ x ) = (cid:82)

Original: q σ (˜ x | x ) p data ( x ) dx .
Revised: q σ (˜ x | x ) p data ( x ) dx .

Original: It should be noted that the score of the estimator only matches the true score when the noise perturbation is minimal i.e q σ (˜ x )
Revised: It should be noted that the score of the estimator only matches the true score when the noise perturbation is minimal i.e q σ (˜ x )

Original: ≈ p data ( x ) .
Revised: ≈ p data ( x ) .

Original: Recently, Song & Ermon (2019) employed multiple noise levels to develop a deep generative model based on score matching, called Noise Conditioned Score Network (NCSN).
Revised: Recently, Song & Ermon (2019) employed multiple noise levels to develop a deep generative model based on score matching, called Noise Conditioned Score Network (NCSN).

Original: Let { σ i } Li =1 be a positive geometric sequence that satisﬁes σ 1 σ 2 = ...
Revised: Let { σ i } Li =1 be a positive geometric sequence that satisﬁes σ 1 σ 2 = ...

Original: =
Revised: =

Original: σ L − 1 σ L > 1 .
Revised: σ L − 1 σ L > 1 .

Original: NCSN is a conditional network, s θ ( x, σ ) , trained to jointly estimate scores for various levels of noise σ i such that ∀ σ ∈ { σ i } Li =1 : s θ ( x, σ )
Revised: NCSN is a conditional network, s θ ( x, σ ) , trained to jointly estimate scores for various levels of noise σ i such that ∀ σ ∈ { σ i } Li =1 : s θ ( x, σ )

Original: ≈ ∇ x log q σ ( x ) .
Revised: ≈ ∇ x log q σ ( x ) .

Original: In practice, the network is explicitly provided a one-hot vector denoting the noise level used to perturb the data.
Revised: In practice, the network is explicitly provided a one-hot vector denoting the noise level used to perturb the data.

Original: The network is then trained via a denoising score matching loss.
Revised: The network is then trained via a denoising score matching loss.

Original: They choose their noise distribution to be N (˜ x | x, σ 2 I ) ; therefore ∇ ˜ x log q σ (˜ x | x ) = − (˜ x − x/σ 2 ) .
Revised: They choose their noise distribution to be N (˜ x | x, σ 2 I ) ; therefore ∇ ˜ x log q σ (˜ x | x ) = − (˜ x − x/σ 2 ) .

Original: Thus the objective function is:
Revised: Thus the objective function is:

Original: [Equation]
Revised: [Equation]

Original: Song & Ermon (2019) set λ ( σ i ) =
Revised: Song & Ermon (2019) set λ ( σ i ) =

Original: σ 2 after empirically observing that ||
Revised: σ 2 after empirically observing that ||

Original: σs θ ( x, σ ) ||
Revised: σs θ ( x, σ ) ||

Original: 2 ∝ 1 .
Revised: 2 ∝ 1 .

Original: We similarly scaled our score norms for all our experiments.
Revised: We similarly scaled our score norms for all our experiments.

Original: Our work directly utilizes the training objective proposed by Song & Ermon (2019) i.e. we use an NCSN as our score estimator.
Revised: Our work directly utilizes the training objective proposed by Song & Ermon (2019) i.e. we use an NCSN as our score estimator.

Original: However, we use the score outputs for out-of-distribution (OOD) detection rather than for generative modeling.
Revised: However, we use the score outputs for out-of-distribution (OOD) detection rather than for generative modeling.

Original: We demonstrate how the space of multiscale score estimates can separate in-distribution samples from outliers, outperforming state-of-the-art methods.
Revised: We demonstrate how the space of multiscale score estimates can separate in-distribution samples from outliers, outperforming state-of-the-art methods.

Original: We also apply our method on real-world medical imaging data of brain MRI scans.
Revised: We also apply our method on real-world medical imaging data of brain MRI scans.

Original:  M ULTISCALE S CORE A NALYSIS
Revised:  M ULTISCALE S CORE A NALYSIS

Original: Consider taking the L2-norm of the score function: || s 
Revised: Consider taking the L2-norm of the score function: || s ( x ) || =

Original: [Equation]
Revised: ||∇ x log p ( x ) || = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ∇ x p ( x ) p ( x ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) .

Original: Since the data density term appears in the denominator, a high likelihood will correspond to a low norm.
Revised: Since the data density term appears in the denominator, a high likelihood will correspond to a low norm.

Original: Since out-of-distribution samples should have a low likelihood with respect to the indistribution log density (i.e. p ( x ) is small), we can expect them to have high score norms.
Revised: Since out-of-distribution samples should have a low likelihood with respect to the indistribution log density (i.e. p ( x ) is small), we can expect them to have high score norms.

Original: However, if these outlier points reside in “ﬂat” regions with very small gradients (e.g. in a small local mode), then their score norms can be low despite the point belonging to a low density region.
Revised: However, if these outlier points reside in “ﬂat” regions with very small gradients (e.g. in a small local mode), then their score norms can be low despite the point belonging to a low density region.

Original: This is our ﬁrst indicator informing us that a true score norm may not be sufﬁcient for detecting outliers.
Revised: This is our ﬁrst indicator informing us that a true score norm may not be sufﬁcient for detecting outliers.

Original: We empirically validate our intuition by considering score estimates for a relatively simple toy dataset:
Revised: We empirically validate our intuition by considering score estimates for a relatively simple toy dataset:

Original: FashionMNIST.
Revised: FashionMNIST.

Original: Following the denoising score matching objective (Equation 2), we can obtain multiple estimates of the true score by using different noise distributions q σ (˜ x | x ) .
Revised: Following the denoising score matching objective (Equation 2), we can obtain multiple estimates of the true score by using different noise distributions q σ (˜ x | x ) .

Original: Like Song & Ermon (2019), we choose the noise distributions to be zero-centered Gaussian scaled according to σ i .
Revised: Like Song & Ermon (2019), we choose the noise distributions to be zero-centered Gaussian scaled according to σ i .

Original: Recall that the scores for samples perturbed by the lowest σ noise should be closest to the true score.
Revised: Recall that the scores for samples perturbed by the lowest σ noise should be closest to the true score.

Original: Our analyses show that this alone was inadequate at separating inliers from OOD samples.
Revised: Our analyses show that this alone was inadequate at separating inliers from OOD samples.

Revised: (a) 1D score estimates from a model trained on

Original: [Figure]
Revised: [Figure]

Original: We trained a score network s FM ( x, σ L ) on FashionMNIST and used it to estimate scores of FashionMNIST ( x ∼ D FM ), MNIST ( x ∼ D M ) and CIFAR-10 ( x ∼ D C ) test sets.
Revised: We trained a score network s FM ( x, σ L ) on FashionMNIST and used it to estimate scores of FashionMNIST ( x ∼ D FM ), MNIST ( x ∼ D M ) and CIFAR-10 ( x ∼ D C ) test sets.

Original: Figure 1a shows the distribution of the score norms corresponding to the lowest noise level used.
Revised: Figure 1a shows the distribution of the score norms corresponding to the lowest noise level used.

Original: Note that CIFAR10 samples are appropriately given a high score by the model.
Revised: Note that CIFAR10 samples are appropriately given a high score by the model.

Original: However, the model is unable to distinguish FashionMNIST from MNIST, giving MNIST roughly the same scores as in-distribution samples.
Revised: However, the model is unable to distinguish FashionMNIST from MNIST, giving MNIST roughly the same scores as in-distribution samples.

Original: Though far from ideal, this result is still a considerable improvement on existing likelihood methods, which have been shown to assign higher likelihoods to OOD samples (Nalisnick et al.
Revised: Though far from ideal, this result is still a considerable improvement on existing likelihood methods, which have been shown to assign higher likelihoods to OOD samples (Nalisnick et al.

Original: Our next line of inquiry was to utilize multiple noise levels.
Revised: Our next line of inquiry was to utilize multiple noise levels.

Original: That is instead of simply considering s FM ( x, σ L ) , we analyze the L -dimensional space [ || s FM ( x, σ 1 )
Revised: That is instead of simply considering s FM ( x, σ L ) , we analyze the L -dimensional space [ || s FM ( x, σ 1 )

Original: || , ..., || s FM ( x, σ L ) || ] for x ∼ { D FM , D M , D C } .
Revised: || , ..., || s FM ( x, σ L ) || ] for x ∼ { D FM , D M , D C } .

Original: Our observations showed that datasets did tend to be separable in the Ldimensional space of score norms.
Revised: Our observations showed that datasets did tend to be separable in the Ldimensional space of score norms.

Original: Figure 1b visualizes the UMAP embeddings of scores calculated via a network trained to estimate L = 10 scales of σ s, with the lowest σ being the same as one in
Revised: Figure 1b visualizes the UMAP embeddings of scores calculated via a network trained to estimate L = 10 scales of σ s, with the lowest σ being the same as one in

Original: Figure 1a.
Revised: Figure 1a.

Original: 2. S CALES AND N EIGHBORHOODS
Revised: 2. S CALES AND N EIGHBORHOODS

Revised: Figure 3:

Revised: In (a) observe that Low-Density outliers have comparatively high gradient norms for both σ L and σ M .

Revised: However at this scale, Local-Mode points still have very small norms, causing them to be tightly packed around the in-distribution points.

Revised: In (b) we see that Local-Mode outliers achieve a gradient signal only when a sufﬁciently high scale is used

Revised: H

Original: To our knowledge multiscale score analysis has not been explored in the context of OOD detection.
Revised: To our knowledge multiscale score analysis has not been explored in the context of OOD detection.

Original: In this section we present an analysis in order to give an intuition for why multiple scales can be beneﬁcial.
Revised: In this section, we present an analysis in order to give an intuition for why multiple scales can be beneﬁcial.

Original: Consider the toy distribution shown in Figure 2.
Revised: Consider the toy distribution shown in Figure 2.

Original: We have three regions of interest: an inlier region with high density centered around x = 10 , an outlier region with low density around x = 30 , and a second outlier region with a local mode centered around x = 50 .
Revised: We have three regions of interest: an inlier region with high density centered around x = 10 , an outlier region with low density around x = 30 , and a second outlier region with a local mode centered around x = 50 .

Original: Recall that adding
Revised: Recall that adding

Original: Gaussian noise to a distribution is equivalent to convolving it with the Gaussian distribution.
Revised: Gaussian noise to a distribution is equivalent to convolving it with the Gaussian distribution.

Original: This not only allows us to visualize perturbations of our toy distribution, but also analytically compute the score estimates given any σ i .
Revised: This not only allows us to visualize perturbations of our toy distribution, but also analytically compute the score estimates given any σ i .

Original: Initially with no perturbation, both a point in the low-density region and one very close to (or at) the local-mode will have small gradients.
Revised: Initially with no perturbation, both a point in the low-density region and one very close to (or at) the local-mode will have small gradients.

Original: As we perturb the samples we smooth the original density, causing it to widen.
Revised: As we perturb the samples we smooth the original density, causing it to widen.

Original: The relative change in density at each point is dependent on neighboring modes.
Revised: The relative change in density at each point is dependent on neighboring modes.

Original: A large scale perturbation will proportionally take a larger neighborhood into account at each point of the convolution.
Revised: A large scale perturbation will proportionally take a larger neighborhood into account at each point of the convolution.

Original: Therefore, at a sufﬁciently large scale, nearby outlier points gain context from in-distribution modes.
Revised: Therefore, at a sufﬁciently large scale, nearby outlier points gain context from in-distribution modes.

Original: This results in an increased gradient signal in the direction of inliers.
Revised: This results in an increased gradient signal in the direction of inliers.

Original: Figure 3 plots the score norms of samples generated from the original density along with markers indicating our key regions.
Revised: Figure 3 plots the score norms of samples generated from the original density along with markers indicating our key regions.

Original: Note how even a small scale perturbation ( σ L = 0 . 1 ) is enough to bias the density of the Low-Density outliers towards the nearby in-distribution mode.
Revised: Note how even a small scale perturbation ( σ L = 0 . 1 ) is enough to bias the density of the Low-Density outliers towards the nearby in-distribution mode.

Original: A medium scale ( σ M = 10 ) Gaussian perturbation is still not wide enough to reach the inlier region from the LocalMode outlier densities, causing them to simply smooth away into ﬂat nothingness.
Revised: A medium scale ( σ M = 10 ) Gaussian perturbation is still not wide enough to reach the inlier region from the LocalMode outlier densities, causing them to simply smooth away into ﬂat nothingness.

Original: It is only after we perform a large scale ( σ H = 20 ) perturbation that the in-distribution mode gets taken into account, resulting in a higher gradient norm.
Revised: It is only after we perform a large scale ( σ H = 20 ) perturbation that the in-distribution mode gets taken into account, resulting in a higher gradient norm.

Original: This analysis allows us to intuit that larger noise levels account for a larger neighborhood context.
Revised: This analysis allows us to intuit that larger noise levels account for a larger neighborhood context.

Original: We surmise that given a sufﬁciently large scale, we can capture gradient signals from distant outliers.
Revised: We surmise that given a sufﬁciently large scale, we can capture gradient signals from distant outliers.

Original: It is imperative to note that one large scale is not guaranteed to work for all outliers.
Revised: It is imperative to note that one large scale is not guaranteed to work for all outliers.

Original: Consider outliers close to inlier modes such as the samples between Low-Density outliers and Inliers in Figure 2.
Revised: Consider outliers close to inlier modes such as the samples between Low-Density outliers and Inliers in Figure 2.

Original: σ H results in an overlap in the score distribution of inliers and Low-Density outliers.
Revised: σ H results in an overlap in the score distribution of inliers and Low-Density outliers.

Original: This makes it difﬁcult to differentiate the aforementioned “in-between” outliers from the in-distribution samples.
Revised: This makes it difﬁcult to differentiate the aforementioned “in-between” outliers from the in-distribution samples.

Original: However, this large scale was necessary to get a big enough neighborhood context in order to capture the more distant Local-Mode outliers.
Revised: However, this large scale was necessary to get a big enough neighborhood context in order to capture the more distant Local-Mode outliers.

Original: Therefore, we postulate that a range of scales is necessary for separating outliers.
Revised: Therefore, we postulate that a range of scales is necessary for separating outliers.

Original: Admittedly, selecting this range according to the dataset is not a trivial problem.
Revised: Admittedly, selecting this range according to the dataset is not a trivial problem.

Original: In a very recent work, Song & Ermon (2020) outlined some techniques for selecting { σ i } Li =1 for
Revised: In a very recent work, Song & Ermon (2020) outlined some techniques for selecting { σ i } Li =1 for

Original: NCSNs from the perspective of generative modelling.
Revised: NCSNs from the perspective of generative modelling.

Original: Perhaps there is a similar analog to OOD detection.
Revised: Perhaps there is a similar analog to OOD detection.

Original: We leave such analyses for future work and use the default range for NCSN in all our experiments.
Revised: We leave such analyses for future work and use the default range for NCSN in all our experiments.

Original: However, we observed that our defaults are surprisingly generalizable, evident by the fact that all our experiments in Section 5 were performed with the same scale range.
Revised: However, we observed that our defaults are surprisingly generalizable, evident by the fact that all our experiments in Section 5 were performed with the same scale range.

Original: In Section 5.5, we further analyze how varying the scale range effects downstream accuracy and observe that our defaults already provide near optimal performance.
Revised: In Section 5.5, we further analyze how varying the scale range effects downstream accuracy and observe that our defaults already provide near optimal performance.

Original: 2. P ROPOSED T RAINING S CHEME
Revised: 2. P ROPOSED T RAINING S CHEME

Original: In this work, we propound the inclusion of all noisy score estimates for the task of separating in- and out-of-distribution points, allowing for a Multiscale Score Matching Analysis (MSMA).
Revised: In this work, we propound the inclusion of all noisy score estimates for the task of separating in- and out-of-distribution points, allowing for a Multiscale Score Matching Analysis (MSMA).

Original: Concretely, given L noise levels, we calculate the L2-norms of per-sample scores for each level, resulting in an L -dimensional vector for each input sample.
Revised: Concretely, given L noise levels, we calculate the L2-norms of per-sample scores for each level, resulting in an L -dimensional vector for each input sample.

Original: Motivated by our observations, we posit that indistribution data points occupy distinct and dense regions in the L -dimensional score space.
Revised: Motivated by our observations, we posit that indistribution data points occupy distinct and dense regions in the L -dimensional score space.

Original: The cluster assumption states that decision boundaries should not pass high density regions, but instead lie in low density regions.
Revised: The cluster assumption states that decision boundaries should not pass high density regions, but instead lie in low density regions.

Original: This implies that any auxiliary method trained to learn in-distribution regions should be able to identify OOD data points that reside outside the learned space.
Revised: This implies that any auxiliary method trained to learn in-distribution regions should be able to identify OOD data points that reside outside the learned space.

Original: Thus, we propose a two step unsupervised training scheme.
Revised: Thus, we propose a two step unsupervised training scheme.

Original: First, we train a NCSN model s IN ( x, σ L ) to estimate scores for inlier samples, given { σ i } Li =1 levels of noise.
Revised: First, we train a NCSN model s IN ( x, σ L ) to estimate scores for inlier samples, given { σ i } Li =1 levels of noise.

Original: Once trained, we calculate all L noisy score estimates for the N training samples and take the L2-norms across the input dimensions: [ || s IN ( X, σ 1 ) || 22 , ..., || s IN ( X, σ L ) ||22 ] .
Revised: Once trained, we calculate all L noisy score estimates for the N training samples and take the L2-norms across the input dimensions: [ || s IN ( X, σ 1 ) || 2 2 , ..., || s IN ( X, σ L ) ||

Original: This results in an N x L matrix.
Revised: This results in an N x L matrix.

Original: We now train an auxiliary model (such as a Gaussian Mixture Model) on this matrix to learn the spatial regions of in-distribution samples in the L -dimensional space.
Revised: We now train an auxiliary model (such as a Gaussian Mixture Model) on this matrix to learn the spatial regions of in-distribution samples in the L -dimensional space.

Original:  L EARNING C ONCENTRATION IN THE S CORE S PACE
Revised:  L EARNING C ONCENTRATION IN THE S CORE S PACE

Original: We posit that learning the “density” of the inlier data in the L -dimensional score (norm) space is sufﬁcient for detecting out-of-distribution samples.
Revised: We posit that learning the “density” of the inlier data in the L -dimensional score (norm) space is sufﬁcient for detecting out-of-distribution samples.

Original: The term “density” can be interpreted in a myriad of ways.
Revised: The term “density” can be interpreted in a myriad of ways.

Original: We primarily focus on models that fall under three related but distinct notions of denseness: spatial clustering, probability density, and nearest (inlier) neighbor graphs.
Revised: We primarily focus on models that fall under three related but distinct notions of denseness: spatial clustering, probability density, and nearest (inlier) neighbor graphs.

Original: All three allow us to threshold the associated metric to best separate OOD samples.
Revised: All three allow us to threshold the associated metric to best separate OOD samples.

Original: Spatial clustering is conceptually the closest to our canonical understanding of denseness: points are tightly packed under some metric (usually Euclidean distance).
Revised: Spatial clustering is conceptually the closest to our canonical understanding of denseness: points are tightly packed under some metric (usually Euclidean distance).

Original: Ideally OOD data should not occupy the same cluster as the inliers.
Revised: Ideally OOD data should not occupy the same cluster as the inliers.

Original: We train Gaussian Mixture Models (GMMs) to learn clusters in the inlier data.
Revised: We train Gaussian Mixture Models (GMMs) to learn clusters in the inlier data.

Original: GMMs work under the assumption that the data is composed of k-components whose shapes can be described by a (multivariate) Gaussian distribution.
Revised: GMMs work under the assumption that the data is composed of k-components whose shapes can be described by a (multivariate) Gaussian distribution.

Original: Thus for a given datum, we can calculate the joint probability of it belonging to any of the k-components.
Revised: Thus for a given datum, we can calculate the joint probability of it belonging to any of the k-components.

Original: Probability density estimation techniques aim to learn the underlying probability density function p data ( x ) which describes the population.
Revised: Probability density estimation techniques aim to learn the underlying probability density function p data ( x ) which describes the population.

Original: Normalizing ﬂows are a family of ﬂexible methods that can learn tractable density functions (Papamakarios et al.
Revised: Normalizing ﬂows are a family of ﬂexible methods that can learn tractable density functions (Papamakarios et al.

Original: They transform complex distributions into a simpler one (such as a Gaussian) through a series of invertible, differential mappings.
Revised: They transform complex distributions into a simpler one (such as a Gaussian) through a series of invertible, differential mappings.

Original: The simpler base distribution can then be used to infer the density of a given sample.
Revised: The simpler base distribution can then be used to infer the density of a given sample.

Original: We use Masked
Revised: We use Masked

Original: Autoregressive Flows introduced by Papamakarios et al.
Revised: Autoregressive Flows introduced by Papamakarios et al.

Original: (2017), which allows us to use neural networks as the transformation functions.
Revised: (2017), which allows us to use neural networks as the transformation functions.

Original: Once trained, we can use the likelihood of the inliers to determine a threshold after which samples will be considered outliers.
Revised: Once trained, we can use the likelihood of the inliers to determine a threshold after which samples will be considered outliers.

Original: Finally, we consider building k-nearest neighbor (k-NN) graphs to allow for yet another thresholding metric.
Revised: Finally, we consider building k-nearest neighbor (k-NN) graphs to allow for yet another thresholding metric.

Original: Conceptually, the idea is to sort all samples according to distances to their k-closest (inlier) neighbor.
Revised: Conceptually, the idea is to sort all samples according to distances to their k-closest (inlier) neighbor.

Original: Presumably, samples from the same distribution as the inliers will have very short distances to training data points.
Revised: Presumably, samples from the same distribution as the inliers will have very short distances to training data points.

Original: Despite its simplicity, this method works surprisingly well.
Revised: Despite its simplicity, this method works surprisingly well.

Original: Practically, k-NN distances can be computed quickly compute by using efﬁcient data structures (such as
Revised: Practically, k-NN distances can be computed quickly compute by using efﬁcient data structures (such as

Original: KD Trees).
Revised: KD Trees).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Hendrycks & Gimpel (2017) should be commended for creating an OOD baseline and establishing an experimental test-bed which has served as a template for all OOD work since.
Revised: Hendrycks & Gimpel (2017) should be commended for creating an OOD baseline and establishing an experimental test-bed which has served as a template for all OOD work since.

Original: Their purported method was thresholding of softmax probabilities of a well-trained classiﬁer.
Revised: Their purported method was thresholding of softmax probabilities of a well-trained classiﬁer.

Original: Their results have since been beaten by more recent work.
Revised: Their results have since been beaten by more recent work.

Original: Liang et al.
Revised: Liang et al.

Original: (2017) propose ODIN as a post-hoc method that utilizes a pretrained network to reliably separate OOD samples from the inlier distribution.
Revised: (2017) propose ODIN as a post-hoc method that utilizes a pretrained network to reliably separate OOD samples from the inlier distribution.

Original: They achieve this via i) perturbing the input image in the gradient direction of the highest (inlier) softmax probability and ii) scaling the temperature of the softmax outputs of the network for the best OOD separation.
Revised: They achieve this via i) perturbing the input image in the gradient direction of the highest (inlier) softmax probability and ii) scaling the temperature of the softmax outputs of the network for the best OOD separation.

Original: They follow the setting from Hendrycks & Gimpel (2017) and show very promising results for the time.
Revised: They follow the setting from Hendrycks & Gimpel (2017) and show very promising results for the time.

Original: However, ODIN heavily depends on careful tuning of its hyperparameters
Revised: However, ODIN heavily depends on careful tuning of its hyperparameters

Original: DeVries & Taylor (2018) train their networks to predict conﬁdence estimates in addition to softmax probabilities, which can then be used to threshold outliers.
Revised: DeVries & Taylor (2018) train their networks to predict conﬁdence estimates in addition to softmax probabilities, which can then be used to threshold outliers.

Original: They show signiﬁcant improvements over Hendrycks & Gimpel (2017) and some improvements over ODIN.
Revised: They show signiﬁcant improvements over Hendrycks & Gimpel (2017) and some improvements over ODIN.

Original: Another concurrent work by Lee et al.
Revised: Another concurrent work by Lee et al.

Original: (2018) jointly trained a GAN alongside the classiﬁer network to generate ‘realistic’
Revised: (2018) jointly trained a GAN alongside the classiﬁer network to generate ‘realistic’

Original: OOD examples, requiring an additional OOD set during training time.
Revised: OOD examples, requiring an additional OOD set during training time.

Original: The ﬁnal trained network is also unable to generalize to other unseen datasets.
Revised: The ﬁnal trained network is also unable to generalize to other unseen datasets.

Original: It is important to note that our method is trained completely unsupervised while the baselines are not, potentially giving them additional information about the idiosyncrasies of the inlier distribution.
Revised: It is important to note that our method is trained completely unsupervised while the baselines are not, potentially giving them additional information about the idiosyncrasies of the inlier distribution.

Original: Ren et al.
Revised: Ren et al.

Original: (2019) proposed to jointly train deep likelihood models alongside a “background” likelihood model that learns the population level background statistics, taking the ratio of the two resulting likelihoods to produce a “contrastive score”.
Revised: (2019) proposed to jointly train deep likelihood models alongside a “background” likelihood model that learns the population level background statistics, taking the ratio of the two resulting likelihoods to produce a “contrastive score”.

Original: They saw very good results for grayscale images (FashionMnist vs MNIST) and saw a considerable improvement in separating CIFAR and SVHN compared to Nalisnick et al.
Revised: They saw very good results for grayscale images (FashionMnist vs MNIST) and saw a considerable improvement in separating CIFAR and SVHN compared to Nalisnick et al.

Original: Some prior work has indeed used gradients of the log likelihoods for OOD but they do not frame it in the context of score matching.
Revised: Some prior work has indeed used gradients of the log likelihoods for OOD but they do not frame it in the context of score matching.

Original: Grathwohl et al.
Revised: Grathwohl et al.

Original: (2020) posits that a discriminative model can be reinterpreted as a joint energy (negative loglikelihood) based model (JEM).
Revised: (2020) posits that a discriminative model can be reinterpreted as a joint energy (negative loglikelihood) based model (JEM).

Original: One of their evaluation experiments used the energy norms (which they dub ‘Approximate Mass JEM’) for OOD detection.
Revised: One of their evaluation experiments used the energy norms (which they dub ‘Approximate Mass JEM’) for OOD detection.

Original: Even though they saw improvements over only using log-likelihoods, their reported AUCs did not beat ODIN or other competitors.
Revised: Even though they saw improvements over only using log-likelihoods, their reported AUCs did not beat ODIN or other competitors.

Original: Peculiarly, they also observed that for tractable likelihood models, scores were anti-correlated with the model’s likelihood and that neither were reliable for OOD detection.
Revised: Peculiarly, they also observed that for tractable likelihood models, scores were anti-correlated with the model’s likelihood and that neither were reliable for OOD detection.

Original: Zhai et al.
Revised: Zhai et al.

Original: (2016) also used energy (negative log probability) gradient norms, but their experiments were limited to intra-dataset anomalies.
Revised: (2016) also used energy (negative log probability) gradient norms, but their experiments were limited to intra-dataset anomalies.

Original: To our knowledge, no prior work has explicitly used score matching for OOD detection.
Revised: To our knowledge, no prior work has explicitly used score matching for OOD detection.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section we demonstrate MSMA’s potential as an effective OOD detector.
Revised: In this section we demonstrate MSMA’s potential as an effective OOD detector.

Original: We ﬁrst train a
Revised: We ﬁrst train a

Original: NCSN model as our score estimator, and then an auxiliary model on the score estimates of the training set.
Revised: NCSN model as our score estimator, and then an auxiliary model on the score estimates of the training set.

Original: Following Liang et al.
Revised: Following Liang et al.

Original: and DeVries & Taylor (2018), we use CIFAR-10 and
Revised: and DeVries & Taylor (2018), we use CIFAR-10 and

Original: SVHN as our “inlier” datasets alongside a collection of natural images as ”outlier” datasets.
Revised: SVHN as our “inlier” datasets alongside a collection of natural images as ”outlier” datasets.

Original: We retrieve the natural image from ODIN’s publicly available GitHub repo 1 .
Revised: We retrieve the natural image from ODIN’s publicly available GitHub repo 2 .

Original: This helps maintain a fair comparison (e.g. it ensures we test on the same random crops as ODIN).
Revised: This helps maintain a fair comparison (e.g. it ensures we test on the same random crops as ODIN).

Original: We will denote Liang et al.
Revised: We will denote Liang et al.

Original: (2017) as ODIN and DeVries & Taylor (2018) as Conﬁdence in our tables.
Revised: (2017) as ODIN and DeVries & Taylor (2018) as Conﬁdence in our tables.

Original: We also distinguish between CIFAR and SVHN and compare our results to the state-of-the-art.
Revised: We also distinguish between CIFAR and SVHN and compare our results to the state-of-the-art.

Original: Additionally, we report our results for FashionMNIST vs MNIST in Section A.6.
Revised: Additionally, we report our results for FashionMNIST vs MNIST in Section A.6.

Original: 5. D ATASETS AND E VALUATION M ETRICS
Revised: 5. D ATASETS AND E VALUATION M ETRICS

Original: We consider CIFAR-10 (Krizhevsky et al.
Revised: We consider CIFAR-10 (Krizhevsky et al.

Original: and SVHN (Netzer et al.
Revised: and SVHN (Netzer et al.

Original: as our inlier datasets.
Revised: as our inlier datasets.

Original: For out-of-distribution datasets, we choose the same as Liang et al.
Revised: For out-of-distribution datasets, we choose the same as Liang et al.

Original: (2017): TinyImageNet 2 , LSUN (Yu et al.
Revised: (2017): TinyImageNet 3 , LSUN (Yu et al.

Original: iSUN (Xu et al. (2015)).
Revised: iSUN (Xu et al. (2015)).

Original: Similar to DeVries & Taylor (2018), in our main experiments we report only resized versions of LSUN and TinyImageNet .
Revised: Similar to DeVries & Taylor (2018), in our main experiments we report only resized versions of LSUN and TinyImageNet .

Original: We also leave out the synthetic Uniform and Gaussian noise samples from our main experiments as we performed extremely well in all of those experiments.
Revised: We also leave out the synthetic Uniform and Gaussian noise samples from our main experiments as we performed extremely well in all of those experiments.

Original: We refer the reader to A.4 for the full report including all datasets.
Revised: We refer the reader to A.4 for the full report including all datasets.

Original: Lastly following DeVries & Taylor (2018), we consider All Images : a combination of all non-synthetic OOD datasets outlined above (including cropped versions).
Revised: Lastly following DeVries & Taylor (2018), we consider All Images : a combination of all non-synthetic OOD datasets outlined above (including cropped versions).

Original: Note that this collection effectively requires a single threshold for all datasets, thus arguably reﬂecting a real world out-of-distribution setting.
Revised: Note that this collection effectively requires a single threshold for all datasets, thus arguably reﬂecting a real world out-of-distribution setting.

Original: To measure thresholding performance we use the metrics established by previous baselines (Hendrycks & Gimpel (2017), Liang et al.
Revised: To measure thresholding performance we use the metrics established by previous baselines (Hendrycks & Gimpel (2017), Liang et al.

Original: FPR at 95% TPR is the
Revised: FPR at 95% TPR is the

Original: False Positive Rate (FPR) when the True Positive Rate (TPR) is 95%.
Revised: False Positive Rate (FPR) when the True Positive Rate (TPR) is 95%.

Original: Detection Error is the minimum possible misclassiﬁcation probability over all thresholds.
Revised: Detection Error is the minimum possible misclassiﬁcation probability over all thresholds.

Original: AUROC is Area Under the ROC
Revised: AUROC is Area Under the ROC

Original: Curve.
Revised: Curve.

Original: AUPR is Area Under the Precision Recall Curve.
Revised: AUPR is Area Under the Precision Recall Curve.

Original: More details are given in A.3.
Revised: More details are given in A.3.

Original: 5. C OMPARISON A GAINST P REVIOUS OOD M ETHODS
Revised: C OMPARISON A GAINST P REVIOUS OOD M ETHODS

Original: We compare our work against Conﬁdence Thresholding (DeVries & Taylor (2018)) and ODIN (Liang et al.
Revised: We compare our work against Conﬁdence Thresholding (DeVries & Taylor (2018)) and ODIN (Liang et al.

Original: For all experiments we report the results for the in-distribution testset vs the out-of-distribution datasets.
Revised: For all experiments we report the results for the in-distribution testset vs the out-of-distribution datasets.

Original: Note that All Images* is a version of All Images where both ODIN and Conﬁdence Thresholding perform input preprocessing.
Revised: Note that All Images* is a version of All Images where both ODIN and Conﬁdence Thresholding perform input preprocessing.

Original: Particularly, they perturb the samples in the direction of the softmax gradient of the classiﬁer: ˜ x
Revised: Particularly, they perturb the samples in the direction of the softmax gradient of the classiﬁer: ˜ x

Original: = x − (cid:15) sign ( −∇ x logS y ( x ; T )) .
Revised: = x − (cid:15) sign ( −∇ x logS y ( x ; T )) .

Original: They then perform a grid search over (cid:15) ranges, selecting the value that achieves best separation onsamples randomly held for each out-of-distribution set.
Revised: They then perform a grid search over (cid:15) ranges, selecting the value that achieves best separation onsamples randomly held for each out-of-distribution set.

Original: ODIN performs an additional search over
Revised: ODIN performs an additional search over

Original: T ranges, while Conﬁdence Thresholding uses a default of T = 1000 .
Revised: T ranges, while Conﬁdence Thresholding uses a default of T = 1000 .

Original: We do not perform any such input modiﬁcation.
Revised: We do not perform any such input modiﬁcation.

Original: Note that ODIN uses input thresholding for individual OOD datsets as well, while Conﬁdence Thresholding does not.
Revised: Note that ODIN uses input thresholding for individual OOD datsets as well, while Conﬁdence Thresholding does not.

Original: Finally, for the sake of brevity we only report FPR (95%
Revised: Finally, for the sake of brevity we only report FPR (95%

Original: TPR) and AUROC .
Revised: TPR) and AUROC .

Original: All other metric comparisons are available in the appendix (A.4)
Revised: All other metric comparisons are available in the appendix (A.4).

Original: [Table]
Revised: [Table]

Original: 5. S EPARATING CIFAR-10 FROM SVHN
Revised: 5. S EPARATING CIFAR-10 FROM SVHN

Original: Since this setting (CIFAR-10 as in-distribution and SVHN as out-of-distribution) is not considered in the testbed used by ODIN or Conﬁdence Thresholding
Revised: Since this setting (CIFAR-10 as in-distribution and SVHN as out-of-distribution) is not considered in the testbed used by ODIN or Conﬁdence Thresholding

Original: [Table]
Revised: [Table]

Original: we consider these results separately and evaluate them in the context of likelihood methods.
Revised: we consider these results separately and evaluate them in the context of likelihood methods.

Original: This experiment has recently gained attention following Nalisnick et al.
Revised: This experiment has recently gained attention following Nalisnick et al.

Original: (2018) showing how deep generative models are particularly inept at separating high dimensional complex datasets such as these two.
Revised: (2018) showing how deep generative models are particularly inept at separating high dimensional complex datasets such as these two.

Original: We describe our results for each auxiliary model in Table 2.
Revised: We describe our results for each auxiliary model in Table 2.

Original: Here we note that all three methods deﬁnitively outperform the previous state of the art (see Table 3), with KD-Trees preforming the best.
Revised: Here we note that all three methods deﬁnitively outperform the previous state of the art (see Table 3), with KD-Trees preforming the best.

Original: Likelihood Ratios Ren et al.
Revised: Likelihood Ratios Ren et al.

Original: JEM (Grathwohl et al.
Revised: JEM (Grathwohl et al.

Original: (2020)) are two unsupervised methods that have tackled this problem and have reported current state-of-the-art results.
Revised: (2020)) are two unsupervised methods that have tackled this problem and have reported current state-of-the-art results.

Original: Table 3 summarizes the results that were reported by these papers.
Revised: Table 3 summarizes the results that were reported by these papers.

Original: Both report AUROCs, with Ren et al.
Revised: Both report AUROCs, with Ren et al.

Original: (2019) additionally reporting AUPR(In) and
Revised: (2019) additionally reporting AUPR(In) and

Original: FPR at 80% TPR.
Revised: FPR at 80% TPR.

Original: Since each method proposes a different detection function, we also provide them for reference.
Revised: Since each method proposes a different detection function, we also provide them for reference.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 5. A GE BASED OOD FROM B RAIN MRI S CANS 
Revised: 5. A GE BASED OOD FROM B RAIN MRI S CANS OOD Age

Original: [Table]
Revised: [Table]

Original: We therefore only bold the betterperformer for ages 1, 2, and 4.
Revised: All values are shown in percentages. ↓ indicates lower values are better and ↑ indicates higher values are better. Keep in mind that the baseline was trained to classify 1 year olds.

Original: In this section we report our method’s performance on a real world dataset.
Revised: In this section we report our method’s performance on a real world dataset.

Original: Here the task is to detect brain Magnetic Resonance Images (MRIs) from pediatric subjects at an age (1 - 6 years) that is younger than the inlier data (9 - 11 years of age).
Revised: Here the task is to detect brain Magnetic Resonance Images (MRIs) from pediatric subjects at an age (1 - 6 years) that is younger than the inlier data (9 - 11 years of age).

Original: We expect visible differences in image contrast and local brain morphometry between the brains of a toddler and an adolescent.
Revised: We expect visible differences in image contrast and local brain morphometry between the brains of a toddler and an adolescent.

Original: As a child grows, their brain matures and the corresponding scans appear more like the prototypical adult brain.
Revised: As a child grows, their brain matures and the corresponding scans appear more like the prototypical adult brain.

Original: This provides an interesting gradation of samples being considered out-of-distribution with respect to age.
Revised: This provides an interesting gradation of samples being considered out-of-distribution with respect to age.

Original: We employ 3500 high resolution T1-weighted MR images obtained through the NIH large scale ABCD study (Casey et al.
Revised: We employ 3500 high resolution T1-weighted MR images obtained through the NIH large scale ABCD study (Casey et al.

Original: (2018)), which represent data from the general adolescent population (9-11 years of age).
Revised: (2018)), which represent data from the general adolescent population (9-11 years of age).

Original: This implies that our in-distribution dataset will have high variation.
Revised: This implies that our in-distribution dataset will have high variation.

Original: After standard preprocessing, we extracted for each dataset three mid-axial slices and resized them to be 90x110 pixels, resulting in ∼ 11k axial images (10k training, 1k testing).
Revised: After standard preprocessing, we extracted for each dataset three mid-axial slices and resized them to be 90x110 pixels, resulting in ∼ 11k axial images (10k training, 1k testing).

Original: For our outlier data, we employ MRI datasets of children aged 1, 2, 4 and 6 years (500 each) from the UNC EBDS database Stephens et al.
Revised: For our outlier data, we employ MRI datasets of children aged 1, 2, 4 and 6 years (500 each) from the UNC EBDS database Stephens et al.

Original: ; Gilmore et al.
Revised: ; Gilmore et al.

Original: Our methodology was effectively able to identify younger age groups as out-of-distribution.
Revised: Our methodology was effectively able to identify younger age groups as out-of-distribution.

Original: Table 4 reports the results for GMMs trained for this task.
Revised: Table 4 reports the results for GMMs trained for this task.

Original: As expected, the separation performance decreases as age increases.
Revised: As expected, the separation performance decreases as age increases.

Original: Note that we kept the same hyperparameters for our auxiliary methods as in the previous experiments despite this being a higher resolution scenario.
Revised: Note that we kept the same hyperparameters for our auxiliary methods as in the previous experiments despite this being a higher resolution scenario.

Original: We also note that our Flow model and KD Tree perform equally as well and refer the reader to A.5.
Revised: We also note that our Flow model and KD Tree perform equally as well and refer the reader to A.5.

Original: Following Liang et al.
Revised: Following Liang et al.

Original: and DeVries & Taylor (2018), we compare our results to the baseline methodology proposed by Hendrycks & Gimpel (2017).
Revised: and DeVries & Taylor (2018), we compare our results to the baseline methodology proposed by Hendrycks & Gimpel (2017).

Original: We trained a standard ResNet-50 (He et al.
Revised: We trained a standard ResNet-50 (He et al.

Original: (2016)) to classify between inliers and 1 year olds and tested its performance on unseen outliers.
Revised: (2016)) to classify between inliers and 1 year olds and tested its performance on unseen outliers.

Original: We observed that MSMA generally outperforms the baseline.
Revised: We observed that MSMA generally outperforms the baseline.

Original: Note that the baseline classiﬁer is biased towards the out-of-distribution detection task since it was trained to separate 1 year olds, wheres MSMA is trained completely unsupervised.
Revised: Note that the baseline classiﬁer is biased towards the out-of-distribution detection task since it was trained to separate 1 year olds, wheres MSMA is trained completely unsupervised.

Revised: Lastly, in Section A. we also report results of f-AnoGAN Schlegl et al.

Revised: (2019) applied on this out-of-distribution task.

Original: 5. H YPERPARAMETER A NALYSIS
Revised: 5. H YPERPARAMETER A NALYSIS

Original: [Figure]
Revised: [Figure]

Original: We performed experiments to analyze how sensitive MSMA is to two main hyperparameters utilized in NCSN: the number of noise levels ( L ) and the largest scale used ( σ H ).
Revised: We performed experiments to analyze how sensitive MSMA is to two main hyperparameters utilized in NCSN: the number of noise levels ( L ) and the largest scale used ( σ H ).

Original: We opted out of varying the smallest noise scale and kept it the same across experiments ( σ L = 0 . 01 ).
Revised: We opted out of varying the smallest noise scale and kept it the same across experiments ( σ L = 0 . 01 ).

Original: Our rationale is that noise perturbations to images at such a small scale are imperceptible to the human eye and are adequate to give an estimate of the true score.
Revised: Our rationale is that noise perturbations to images at such a small scale are imperceptible to the human eye and are adequate to give an estimate of the true score.

Original: For all experiments, we train on CIFAR-10 and evaluate on the All Images dataset described in Section 5.1.
Revised: For all experiments, we train on CIFAR-10 and evaluate on the All Images dataset described in Section 5.1 and plot AUROC in Figure 5.

Original: In order to reduce GPU memory usage and computation time, we had to reduce the batch size from 128 to 64, which is why we see a performance dip from the main experiment in Section 5.2.
Revised: In order to reduce GPU memory usage and computation time, we had to reduce the batch size from 128 to 64, which is why we see a performance dip from the main experiment in Section 5.2.

Original: We keep the same hyperparameters for our auxiliary models as in the main experiment.
Revised: We keep the same hyperparameters for our auxiliary models as in the main experiment.

Original: Figure 5 shows the area under the ROC curves for separating CIFARfrom All Images .

Original: For the number of scales L , we test the values 1, 3, 10, 15, and 20, with σ H ﬁxed at the default value ( σ H = 1 ) .
Revised: For the number of scales L , we test the values 1, 3, 10, 15, and 20, with σ H ﬁxed at the default value ( σ H = 1 ) .

Original: Recall that we follow the original NCSN schema by (Song & Ermon, 2019), and utilizes a geometric sequence of sigma scales from σ H to σ L with L steps (endpoint inclusive).
Revised: Recall that we follow the original NCSN schema by (Song & Ermon, 2019), and utilizes a geometric sequence of sigma scales from σ H to σ L with L steps (endpoint inclusive).

Original: Thus, changing L changes the intermediate noise scales.
Revised: Thus, changing L changes the intermediate noise scales.

Original: Our results in Figure 5a show that MSMA is optimal near the default ( L = 10 ).
Revised: Our results in Figure 5a show that MSMA is optimal near the default ( L = 10 ).

Original: Increasing L does not signiﬁcantly vary the performance, while small values such as L = 3 are not be adequate at providing enough intermediate scales.
Revised: Increasing L does not signiﬁcantly vary the performance, while small values such as L = 3 are not be adequate at providing enough intermediate scales.

Original: Note that
Revised: Note that

Original: L = 1 is the degenerate case where only the largest noise scale is used.
Revised: L = 1 is the degenerate case where only the largest noise scale is used.

Original: This highlights the need for a range of scales as argued in Section 2.1 and empirically shows that simply using one large scale is not enough.
Revised: This highlights the need for a range of scales as argued in Section 2.1 and empirically shows that simply using one large scale is not enough.

Original: Figure 5b plots the affect of varying the largest noise scale σ H .
Revised: Figure 5b plots the affect of varying the largest noise scale σ H .

Original: We test the values 0.5, 1.0, 2.0 and 10, with the default number of scales L = 10 .
Revised: We test the values 0.5, 1.0, 2.0 and 10, with the default number of scales L = 10 .

Original: Again, we observe that our default σ H = 1 performs the best and we do not notice any improvement form varying it.
Revised: Again, we observe that our default σ H = 1 performs the best and we do not notice any improvement form varying it.

Original: Considering how images are rescaled to [0,1] before they are passed to the network, we posit that σ H = 1 . 

Revised: Considering how images are rescaled to [0,1] before they are passed to the network, we posit that σ H = 1 .

Original: already introduces large noise and increasing it further seems to degrade results to varying degrees.
Revised: already introduces large noise and increasing it further seems to degrade results to varying degrees.

Original: Lastly, we would like to emphasize that all our main out-of-distribution experiments in Section were performed with the same default hyperparameters, without any tuning.
Revised: Lastly, we would like to emphasize that all our main out-of-distribution experiments in Section were performed with the same default hyperparameters, without any tuning.

Original: Despite this disadvantage, MSMA still outperforms its competitors ODIN
Revised: Despite this disadvantage, MSMA still outperforms its competitors ODIN

Original: (Liang et al.
Revised: (Liang et al.

Original: , Conﬁdence Thresholding (DeVries & Taylor (2018)), and Likelihood Ratios (Ren et al.
Revised: , Conﬁdence Thresholding (DeVries & Taylor (2018)), and Likelihood Ratios (Ren et al.

Original: (2019)), all of which need some ﬁne-tuning of hyperparameters.
Revised: (2019)), all of which need some ﬁne-tuning of hyperparameters.

Original: Recognize that tuning requires apriori knowledge of the type of out-of-distribution samples the user expects.
Revised: Recognize that tuning requires apriori knowledge of the type of out-of-distribution samples the user expects.

Original: From the analysis in this section and our main experiment, we can conﬁdently advocate the use of our defaults as they seem to generalize well across datasets and do not require such apriori knowledge.
Revised: From the analysis in this section and our main experiment, we can conﬁdently advocate the use of our defaults as they seem to generalize well across datasets and do not require such apriori knowledge.

Original: Admittedly, if the form of anomalies are known at training time then it would indeed be possible to tune MSMA’s hyperparameters to ﬁt a particular need.
Revised: Admittedly, if the form of anomalies are known at training time then it would indeed be possible to tune MSMA’s hyperparameters to ﬁt a particular need.

Original: However, we leave such an analysis for future work as it starts to encroach the domain of anomaly detection whereas the work presented in this paper is mainly concerned with introducing
Revised: However, we leave such an analysis for future work as it starts to encroach the domain of anomaly detection whereas the work presented in this paper is mainly concerned with introducing

Original: MSMA as a performant, easy to implement, and generalizable out-of-distribution detector.
Revised: MSMA as a performant, easy to implement, and generalizable out-of-distribution detector.

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: We introduced a methodology based on multiscale score matching and showed that it outperformed state-of-the-art methods, with minimal hyperparameter tuning.
Revised: We introduced a methodology based on multiscale score matching and showed that it outperformed state-of-the-art methods, with minimal hyperparameter tuning.

Original: Our methodology is easy to implement, completely unsupervised and generalizable to many OOD tasks.
Revised: Our methodology is easy to implement, completely unsupervised and generalizable to many OOD tasks.

Original: Even though we only reported two metrics in the main comparison, we emphasize that we outperform previous state-ofthe-art in every metric for all benchmark experiments.
Revised: Even though we only reported two metrics in the main comparison, we emphasize that we outperform previous state-ofthe-art in every metric for all benchmark experiments.

Original: Next, it is noteworthy that in our real-world experiment, the brain MR images are unlabeled.
Revised: Next, it is noteworthy that in our real-world experiment, the brain MR images are unlabeled.

Original: Since our model is trained completely unsupervised, we have to make very few inductive biases pertaining to the data.
Revised: Since our model is trained completely unsupervised, we have to make very few inductive biases pertaining to the data.

Original: Note that we plan to apply our method to high resolution 3D MRIs, which can reach upto 256x256x256.
Revised: Note that we plan to apply our method to high resolution 3D MRIs, which can reach upto 256x256x256.

Original: Consider how likelihood methods such as Glow Kingma & Dhariwal (2018) already struggle to identify out-of-distribution samples for lower resolution 2D images ( Nalisnick et al.
Revised: Consider how likelihood methods such as Glow Kingma & Dhariwal (2018) already struggle to identify out-of-distribution samples for lower resolution 2D images ( Nalisnick et al.

Original: It is unclear whether they would preform adequately at this task when they are scaled up to the dimensionality of 3D MRIs.
Revised: It is unclear whether they would preform adequately at this task when they are scaled up to the dimensionality of 3D MRIs.

Original: Furthermore, there is evidence that deep likelihood models are difﬁcult to train in such high resolution regimes (Papamakarios (2019)), especially given low sample sizes.
Revised: Furthermore, there is evidence that deep likelihood models are difﬁcult to train in such high resolution regimes (Papamakarios (2019)), especially given low sample sizes.

Original: Our model’s objective function is based on a denoising (autoencoding), and can be easily extended to any network architecture.
Revised: Our model’s objective function is based on a denoising (autoencoding), and can be easily extended to any network architecture.

Original: For example, our preliminary results show that MSMA was able to achieve similar performance to Section 5.4 when applied to their full 3D MRI counterparts by utilizing a publicly available 3D U-Net (Zhou et al.
Revised: For example, our preliminary results show that MSMA was able to achieve similar performance to Section 5.4 when applied to their full 3D MRI counterparts by utilizing a publicly available 3D U-Net (Zhou et al.

Original: We plan to explore these high resolution scenarios as our next future direction.
Revised: We plan to explore these high resolution scenarios as our next future direction.

Original: Our excellent results highlight the possibility of using MSMA as a fast, general purpose anomaly detector which could be used for tasks ranging from detection of medical pathologies to data cleansing and fault detection.
Revised: Our excellent results highlight the possibility of using MSMA as a fast, general purpose anomaly detector which could be used for tasks ranging from detection of medical pathologies to data cleansing and fault detection.

Original: From an application perspective, we plan to apply this methodology to the task of detecting images of atypically maturing children from a database of typical inliers.
Revised: From an application perspective, we plan to apply this methodology to the task of detecting images of atypically maturing children from a database of typical inliers.

Original: Lastly, our observations have uncovered a peculiar phenomenon exhibited by multiscale score estimates, warranting a closer look to understand the theoretical underpinnings of the relationship between low density points and their gradient estimates. 
Revised: Lastly, our observations have uncovered a peculiar phenomenon exhibited by multiscale score estimates, warranting a closer look to understand the theoretical underpinnings of the relationship between low density points and their gradient estimates. A CKNOWLEDGMENTS

Revised: We would like to thank the UNC Early Brain Development Study (PI John Gilmore) and the Adolescent Brain Cognitive Development (ABCD) Study (https://abcdstudy.org), held in the NIMH Data

Revised: Archive (NDA).

Original: A A PPENDIX A.D ATASET D ETAILS
Revised: The ABCD data repository grows and changes over time. The ABCD data used in this report came from DOI 10.15154/1503209 (v2.1). Other funding was provided through NIH grants HD053000, MH070890, MH111944, HD079124, EB and MH115046. A.

Revised: [Equation]

Revised: D ATASET D ETAILS

Original: All the datsets considered are described below.
Revised: All the datsets considered are described below.

Original: CIFAR-10:
Revised: CIFAR-

Original: The CIFAR-10 dataset (Krizhevsky et al.
Revised: The CIFAR-10 dataset (Krizhevsky et al.

Original: (2009)) consists of 60,000 32x32 colour images in 10 classes, such as horse, automobile, cat etc.
Revised: (2009)) consists of 60,000 32x32 colour images in 10 classes such as horse, automobile, cat etc.

Original: There are 50,000 training images and 10, test images.
Revised: There are 50,000 training images and 10, test images.

Original: SVHN:
Revised: SVHN:

Original: The Street View Housing Numbers (SVHN) dataset (Netzer et al.
Revised: The Street View Housing Numbers SVHN ) dataset ( Netzer

Revised: et al.

Original: (2011)) consists of 32x images depicting house numbers ranging from 0 through 9.
Revised: consists of 32x images depicting house numbers ranging from 0 through 9.

Original: We use the ofﬁcial splits: 73,257 digits for training, 26,032 digits for testing.
Revised: We use the ofﬁcial splits: 73,257 digits for training, 26,032 digits for testing.

Original: TinyImageNet:
Revised: TinyImageNet

Original: This dataset 3 is a subset of the ImageNet dataset (Deng et al.
Revised: This dataset 4 is a subset of the ImageNet dataset (Deng et al.

Original: The test set has 10,000 images with 50 samples for each of the 200 classes.
Revised: The test set has 10,000 images with 50 samples for each of the 200 classes.

Original: Liang et al.
Revised: Liang et al.

Original: (2017) created two 32x32 pixel versions of this dataset: TinyImageNet (crop) which contains random crops of the original test samples and TinyImageNet (resize) which contains downscaled test images.
Revised: (2017) created two 32x32 pixel versions of this dataset: TinyImageNet (crop) which contains random crops of the original test samples and TinyImageNet (resize) which contains downscaled test images.

Original: LSUN: The Large Scene UNderstanding (LSUN) produced by (Yu et al. (2015)) consists of 10, test images belonging to one of 10 different scene classes such as bedroom, kitchen etc.
Revised: LSUN: The Large Scene UNderstanding (LSUN) produced by (Yu et al. (2015)) consists of 10, test images belonging to one of 10 different scene classes such as bedroom, kitchen etc.

Original: Liang et al.
Revised: Liang et al.

Original: (2017) created two 32x32 pixel versions of this dataset as well: a randomly cropped LSUN (crop) and a downsampled LSUN (resize) .
Revised: (2017) created two 32x32 pixel versions of this dataset as well: a randomly cropped LSUN (crop) and a downsampled LSUN (resize) .

Original: iSUN: This dataset was procured by (Xu et al.
Revised: iSUN This dataset was procured by (Xu et al.

Original: and is a subsample of the SUN image database.
Revised: and is a subsample of the SUN image database.

Original: We use 32x32 pixel downscaled versions of the original 8,925 test images.
Revised: We use 32x32 pixel downscaled versions of the original 8,925 test images.

Original: Uniform: This dataset consists of 10,000 synthetically generated 32x32 RGB images produced by sampling each pixel from an i.i.d uniform distribution in the range [0,1].
Revised: Uniform This dataset consists of 10,000 synthetically generated 32x32 RGB images produced by sampling each pixel from an i.i.d uniform distribution in the range [0,1].

Original: Gaussian: These are 10,000 synthetic 32x32 RGB images where each pixel is sampled from an i.i.d
Revised: Gaussian These are 10,000 synthetic 32x32 RGB images where each pixel is sampled from an i.i.d

Original: Gaussian distribution centered at 0.5 with a standard deviation of 1.
Revised: Gaussian distribution centered at 0.5 with a standard deviation of 1.

Original: The pixel values are clipped to be within [0, 1] to keep the values within the expected range of (normalized) images.
Revised: The pixel values are clipped to be within [0, 1] to keep the values within the expected range of (normalized) images.

Original: All Images : Following DeVries & Taylor (2018), this dataset is a combination of all non-synthetic
Revised: All Images : Following DeVries & Taylor (2018), this dataset is a combination of all non-synthetic

Original: OOD datasets outlined above: TinyImageNet (crop) , TinyImageNet (resize) , LSUN (crop) , LSUN (resize) and iSUN .
Revised: OOD datasets outlined above: TinyImageNet (crop) , TinyImageNet (resize) , LSUN (crop) , LSUN (resize) and iSUN .

Original: Therefore this contains 48,925 images from a variety of data distributions.
Revised: Therefore this contains 48,925 images from a variety of data distributions.

Original: Note that this collection effectively requires a single threshold for all datasets, thus arguably reﬂecting a real world out-of-distribution setting.
Revised: Note that this collection effectively requires a single threshold for all datasets, thus arguably reﬂecting a real world out-of-distribution setting.

Original: A. E XPERIMENTAL D ETAILS
Revised: A. E XPERIMENTAL D ETAILS

Original: We use the NCSN model provided by Song & Ermon (2019).
Revised: We use the NCSN model provided by Song & Ermon (2019).

Original: In particular, we use the Tensorﬂow implementation provided through a NeurIPS reproducibilty challenge submission, Matosevic et al.
Revised: In particular, we use the Tensorﬂow implementation provided through a NeurIPS reproducibilty challenge submission, Matosevic et al.

Original: The model architecture used is a ReﬁneNet with 128 ﬁlters.
Revised: The model architecture used is a ReﬁneNet with 128 ﬁlters.

Original: The batch size is also ﬁxed to 128.
Revised: The batch size is also ﬁxed to 128.

Original: We train for 200k iterations using the Adam optimizer.
Revised: We train for 200k iterations using the Adam optimizer.

Original: Following Song & Ermon (2019), we use L = 10 standard deviations for our Gaussian noise perturbation such that { σ i } Li =1 is a geometric sequence with σ 1 = 1 and σ 10 = 0 .
Revised: Following Song & Ermon (2019), we use L = 10 standard deviations for our Gaussian noise perturbation such that { σ i } Li =1 is a geometric sequence with σ 1 = 1 and σ 10 = 0 .

Original: We use the same hyperparameters for training on both CIFAR and SVHN.
Revised: We use the same hyperparameters for training on both CIFAR and SVHN.

Original: For our experiment on brain MRI images (Section 5.4), we trained our model with 64 ﬁlters and a batch size of 32 due to memory constraints caused by the higher resolution images.
Revised: For our experiment on brain MRI images (Section 5.4), we trained our model with 64 ﬁlters and a batch size of 32 due to memory constraints caused by the higher resolution images.

Original: For the baseline network in Section5.4, we used the standard ResNet50 available in Keras and added a global average pooling layer followed by a dense layer of 1024 nodes followed by a softmax layer with binary output.
Revised: For the baseline network in Section5.4, we used the standard ResNet50 available in Keras and added a global average pooling layer followed by a dense layer of 1024 nodes followed by a softmax layer with binary output.

Original: We trained for 10 epochs using Adam with default learning rate and momentum.
Revised: We trained for 10 epochs using Adam with default learning rate and momentum.

Original: For unseen classes, we follow Hendrycks et al.
Revised: For unseen classes, we follow Hendrycks et al.

Original: and pick the maximum prediction probability as our outlier score.
Revised: and pick the maximum prediction probability as our outlier score.

Original: Since the network was trained to classify one year olds, we report the 1 year metrics using the logits corresponding to the outlier class only in lieu of taking the max across both classes.
Revised: Since the network was trained to classify one year olds, we report the 1 year metrics using the logits corresponding to the outlier class only in lieu of taking the max across both classes.

Original: We train our auxiliary models on the same training set that was used to train the NCSN model, thereby circumventing the need for a separate held out tuning set.
Revised: We train our auxiliary models on the same training set that was used to train the NCSN model, thereby circumventing the need for a separate held out tuning set.

Original: For our Gaussian Mixture Models, we mean normalize the data and perform a grid search over the number of components (ranging from 2 to 20), using 10-fold cross validation.
Revised: For our Gaussian Mixture Models, we mean normalize the data and perform a grid search over the number of components (ranging from 2 to 20), using 10-fold cross validation.

Original: Our normalizing ﬂow model is constructed with a
Revised: Our normalizing ﬂow model is constructed with a

Original: MAF using two hidden layers with 128 units each, and a Standard Normal as the base distribution.
Revised: MAF using two hidden layers with 128 units each, and a Standard Normal as the base distribution.

Original: It is trained for a 1000 epochs with a batch size of 128.
Revised: It is trained for a 1000 epochs with a batch size of 128.

Original: Finally for our nearest neighbor model, we train a KD Tree to store (k=5)-nearest neighbor distances of the in-distribution training set.
Revised: Finally for our nearest neighbor model, we train a KD Tree to store (k=5)-nearest neighbor distances of the in-distribution training set.

Original: We keep the same hyperparameter settings for all experiments.
Revised: We keep the same hyperparameter settings for all experiments.

Original: Note that in Section 5.2, as ODIN and Conﬁdence Thresholding were trained with a number of different architectures, we report the ones that performed the best for each respective method.
Revised: Note that in Section 5.2, as ODIN and Conﬁdence Thresholding were trained with a number of different architectures, we report the ones that performed the best for each respective method.

Original: Speciﬁcally, we use the results of VGG13 for Conﬁdence Thresholding and DenseNet-BC for ODIN.
Revised: Speciﬁcally, we use the results of VGG13 for Conﬁdence Thresholding and DenseNet-BC for ODIN.

Original: A. E VALUATION M ETRIC D ETAILS
Revised: A. E VALUATION M ETRIC D ETAILS

Original: To measure thresholding performance we use the metrics established by previous baselines (Hendrycks & Gimpel (2017), Liang et al.
Revised: To measure thresholding performance we use the metrics established by previous baselines (Hendrycks & Gimpel (2017), Liang et al.

Original: These include:
Revised: These include:

Original: FPR at 95% TPR:
Revised: FPR at 95% TPR:

Original: This is the False Positive Rate (FPR) when the True Positive Rate (TPR) is 95%.
Revised: This is the False Positive Rate (FPR) when the True Positive Rate (TPR) is 95%.

Original: This metric can be interpreted as the probability of misclassifying an outlier sample to be indistribution when the TPR is as high as 95%.
Revised: This metric can be interpreted as the probability of misclassifying an outlier sample to be indistribution when the TPR is as high as 95%.

Original: Let TP, FP, TN, and FN represent true positives, false positives, true negatives and false negatives respectively.
Revised: Let TP, FP, TN, and FN represent true positives, false positives, true negatives and false negatives respectively.

Original: FPR=FP/(FP+TN), TPR=TP/(FN+TP).
Revised: FPR=FP/(FP+TN), TPR=TP/(FN+TP).

Original: Detection Error:
Revised: Detection Error:

Original: This measures the minimum possible misclassiﬁcation probability over all thresholds.
Revised: This measures the minimum possible misclassiﬁcation probability over all thresholds.

Original: Practically this can be calculated as min δ 0 .
Revised: Practically this can be calculated as min δ 0 .

Original: 5(1 − TPR )
Revised: 5(1 − TPR )

Original: + 0 .
Revised: + 0 .

Original: 5 FPR, where it is assumed that we have an equal probability of seeing both positive and negative examples in the test set.
Revised: 5 FPR, where it is assumed that we have an equal probability of seeing both positive and negative examples in the test set.

Original: AUROC:
Revised: AUROC:

Original: This measures area under (AU) the Receiver Operating Curve (ROC) which plots the relationship between FPR and TPR.
Revised: This measures area under (AU) the Receiver Operating Curve (ROC) which plots the relationship between FPR and TPR.

Original: It is commonly interpreted as the probability of a positive sample (in-distribution) having a higher score than a negative sample (out-of-distribution).
Revised: It is commonly interpreted as the probability of a positive sample (in-distribution) having a higher score than a negative sample (out-of-distribution).

Original: It is a threshold independent, summary metric.
Revised: It is a threshold independent, summary metric.

Original: AUPR: Area Under the Precision Recall Curve (AUPR) is another threshold independent metric that considers the PR curve, which plots Precision(= TP/(TP+FP) ) versus Recall(= TP/(TP+FN) ).
Revised: AUPR: Area Under the Precision Recall Curve (AUPR) is another threshold independent metric that considers the PR curve, which plots Precision(= TP/(TP+FP) ) versus Recall(= TP/(TP+FN) ).

Original: AUPR-In and AUPR-Out consider the in-distribution samples and out-of-distribution samples as the positive class, respectively.
Revised: AUPR-In and AUPR-Out consider the in-distribution samples and out-of-distribution samples as the positive class, respectively.

Original: This helps take mismatch in sample sizes into account.
Revised: This helps take mismatch in sample sizes into account.

Original: A. C OMPLETE R ESULTS FOR E XPERIMENTS IN S ECTION 5. 
Revised: A. C OMPLETE R ESULTS FOR E XPERIMENTS IN S ECTION 5.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: A. P ERFORMANCE ON B RAIN MRI
Revised: A. P ERFORMANCE ON B RAIN MRI

Original: [Table]
Revised: [Table]

Original: A. S EPARATING F ASHION -MNIST FROM MNIST
Revised: A. S EPARATING F ASHION -MNIST FROM MNIST

Original: Table 14 summarizes the results for our FashionMNIST (inlier) and MNIST outlier experiments.
Revised: Table 14 summarizes the results for our FashionMNIST (inlier) and MNIST outlier experiments.

Original: We report the same metrics as the original We observe that our results (with default hyperparameters) are not as good as their colored image counter parts.
Revised: We report the same metrics as the original We observe that our results (with default hyperparameters) are not as good as their colored image counter parts.

Original: However, we are able to beat ODIN and raw likelihoods.
Revised: However, we are able to beat ODIN and raw likelihoods.

Original: While it may be possible to tune the hyperprameters like Likelihood Ratios, we leave that analysis for future work.
Revised: While it may be possible to tune the hyperprameters like Likelihood Ratios, we leave that analysis for future work.

Original: Furthermore, we tried to run Likelihood Ratios using the provided code but were unable to reproduce the results presented in the original paper.
Revised: Furthermore, we tried to run Likelihood Ratios using the provided code but were unable to reproduce the results presented in the original paper.

Original: Thus, we compare against their reported metrics for completeness but cannot make conclusions about their validity
Revised: Thus, we compare against their reported metrics for completeness but cannot make conclusions about their validity

Original: [Table]
Revised: [Table]

Revised: A. P ERFORMANCE OF F -A NO GAN ON B RAIN MRI

Revised: We also compared our methodology to f-AnoGAN due to its promising results as an anomaly detector and its popularity in the medical community.

Revised: We use the hyperparameters suggested in the original paper and trained till convergence.

Revised: Our results show that f-AnoGAN does not outperform

Revised: MSMA in any experiment

Revised: [Table]

Revised: and is unable to match the classiﬁer baseline reported in Section 5.4.

Revised: However, we acknowledge that it is a fast method, both in terms of training and inference

Revised: [Table]

Revised: and may be useful for some cases where pixel-wise anomalies are required (a current limitation of MSMA)

Revised: [Table]