Original: Density-driven Regularization for Out-of-distribution Detection
Revised: Density-driven Regularization for Out-of-distribution Detection

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Detecting out-of-distribution (OOD) samples is essential for reliably deployingdeep learning classifiers in open-world applications.
Revised: Detecting out-of-distribution (OOD) samples is essential for reliably deploying deep learning classifiers in open-world applications.

Original: However, existing detectorsrelying on discriminative probability suffer from the overconfident posterior estimate for OOD data.
Revised: However, existing detectors relying on discriminative probability suffer from the overconfident posterior estimate for OOD data.

Original: Other reported approaches either impose strong unprovenparametric assumptions to estimate OOD sample density or develop empiricaldetectors lacking clear theoretical motivations.
Revised: Other reported approaches either impose strong unproven parametric assumptions to estimate OOD sample density or develop empirical detectors lacking clear theoretical motivations.

Original: To address these issues, we proposea theoretical probabilistic framework for OOD detection in deep classification networks, in which two regularization constraints are constructed to reliably estimatesample density to identify OOD.
Revised: To address these issues, we propose a theoretical probabilistic framework for OOD detection in deep classification networks, in which two regularization constraints are constructed to reliably calibrate and estimate sample density to identify OOD.

Original: Specifically, the density consistency regularizationenforces the agreement between analytical and empirical densities of observablelow-dimensional categorical labels.
Revised: Specifically, the density consistency regularization enforces the agreement between analytical and empirical densities of observable low-dimensional categorical labels.

Original: The contrastive distribution regularization separates the densities between in distribution (ID) and distribution-deviated samples.
Revised: The contrastive distribution regularization separates the densities between in distribution (ID) and distribution-deviated samples.

Original: A simple and robust implementation algorithm is also provided, which can beused for any pre-trained neural network classifiers.
Revised: A simple and robust implementation algorithm is also provided, which can be used for any pre-trained neural network classifiers.

Original: To the best of our knowledge,we have conducted the most extensive evaluations and comparisons on computervision benchmarks.
Revised: To the best of our knowledge, we have conducted the most extensive evaluations and comparisons on computer vision benchmarks.

Original: The results show that our method significantly outperformsstate-of-the-art detectors, and even achieves comparable or better performance thanmethods utilizing additional large-scale outlier exposure datasets.
Revised: The results show that our method significantly outperforms state-of-the-art detectors, and even achieves comparable or better performance than methods utilizing additional large-scale outlier exposure datasets.

Original: Our code will beopen-sourced upon acceptance.

Original:  Introduction
Revised:  Introduction

Original: Deepneural networks have achieved impressive accuracy on many classification tasks when evaluatedin a static and closed environment, in which the test samples are sampled from the same distributionas the training data.
Revised: Deep networks have achieved impressive accuracy on many classification tasks when evaluated in a static and closed environment, where the test samples are sampled from the same distribution as the training data.

Original: However, when deployed in real-world applications, deep learning  models oftenencounter anomalous samples different from the training data.
Revised: However, when deployed in real-world applications, deep learning (DL) models often encounter anomalous samples different from the training data.

Original: Detecting such unexplored samplesis referred to as out-of-distribution (OOD) detection [1–13], which is crucial for reliable decisionmaking and essential in safety-critical applications such as autonomous driving, medical diagnosis,secure authentication and financial risk management [1, 3, 8, 9].
Revised: Detecting such unexplored samples is referred to as out-of-distribution (OOD) detection [1–13], which is crucial for reliable decision making, especially in safety-critical applications such as autonomous driving, medical diagnosis and secure authentication[1, 3, 8, 9].

Original: This topic is particularly importantfor deeplearning classifiers because it has been well documented that it can be easily fooled by OODsamples to produce high confidence predictions [4, 7–9, 11, 14–16].
Revised: This topic is particularly important for deep classifiers because it has been well documented that it can be easily fooled by OOD samples to produce high confidence predictions [14, 15, 7, 8, 11, 9, 16, 4].

Original: In recent years, most reported and favored strategies for deep-learning based OOD detection are todesign the detectors by inspecting the distribution of high-level features extracted by the relevanttarget task network.
Revised: In recent years, most reported and favored strategies for DL-based OOD detection are to design the detectors by inspecting the distribution of high-level features extracted by the relevant target task network.

Original: The studies are briefly reviewed below.
Revised: The studies are briefly reviewed below.

Original: As a pilot study, Hendrycks et al. proposed to detect OOD samples based on maximal softmaxprobability (Msp) [16], i.e., max y p ( y | x ) , which has become a common baseline for OOD detection[4–7, 9, 10, 12, 13, 17,  18].
Revised: As a pilot study, Hendrycks et al. proposed to detect OOD samples based on maximal softmax probability (Msp) [16], i.e., max y p ( y | x ) , which has become a common baseline for OOD detection [10, 4–7, 12, 9, 17, 13, 18].

Original: Subsequently, Liang et al. proposed the Odin method, which providedtemperature scaling and input perturbation strategies, to push the Msp score further apart for ID and OOD data [17].
Revised: Subsequently, Liang et al. proposed the Odin method, which provided temperature scaling and input perturbation strategies, to push the Msp score further apart for ID and OOD data [17].

Original: Hsu et al.
Revised: Hsu et al.

Original: generalized the Odin method to avoid access to OOD samples forparameter tuning [5].
Revised: generalized the Odin method to avoid access to OOD samples for parameter tuning [5].

Original: Another similar technique, which uses softmax probability for OOD detection,is pNML [3].
Revised: Another similar technique, which uses softmax probability for OOD detection, is pNML [3].

Original: In pNML, Bibas et al. detected OOD samples using their proposed regret scorelog (cid:80) y p
Revised: In pNML, Bibas et al. detected OOD samples using their proposed regret score log (cid:80) y p

Original: ( y | x ) , where p ( y | x ) denotes online fine-tuned softmax probability by adding the test samplex to the training set with an arbitrary label y .
Revised: ( y | x ) , where p ( y | x ) denotes online fine-tuned softmax probability by adding the test sample x to the training set with an arbitrary label y .

Original: However, these methods relying on softmax probabilityscore p ( y | x ) can suffer from overconfidence problem since many studies have shown that neuralnetworks can produce arbitrarily high softmax probability for OOD inputs [1, 4, 8, 10,  12,19, 20].
Revised: However, these methods relying on softmax probability score p ( y | x ) can suffer from overconfidence problem since many studies have shown that neural networks can produce arbitrarily high softmax probability for OOD inputs [4, 19, 1, 10, 8, 12, 20].

Original: Another line of research utilizes the distribution of extracted high-level features f ( x ) to estimate
Revised: Another line of research utilizes the distribution of extracted high-level features f ( x ) to estimate

Original: ID data density p ( x ) for OOD detection as modeling p ( x ) directly with high-dimensional input xis prohibitive in most cases.
Revised: ID data density p ( x ) for OOD detection as modeling p ( x ) directly with high-dimensional input x is prohibitive in most cases.

Original: For example, Lee et al.
Revised: For example, Lee et al.

Original: [9] proposed using the Mahalanobis distancebetween the test sample’s feature representations and the class-conditional gaussian distributions inthe intermediate layers to train a logistic regression OOD detector.
Revised: [9] proposed using the Mahalanobis distance between the test sample’s feature representations and the class-conditional gaussian distributions in the intermediate layers to train a logistic regression OOD detector.

Original: Essentially, this Mah methodconstructed the OOD score using max y p ( f ( x ) | y ) , in which p ( f ( x ) | y ) was assumed to obey Gaussiandistribution for the ID training data.
Revised: Essentially, this Mah method constructed the OOD score using max y p ( f ( x ) | y ) , in which p ( f ( x ) | y ) was assumed to obey Gaussian distribution for the ID training data.

Original: Afterward, Sehwag et al.
Revised: Afterward, Sehwag et al.

Original: [21] and Zhou et al.
Revised: [21] and Zhou et al.

Original: [19] furtherextended the Mah model for unsupervised learning (no training labels available) and semi-supervisedlearning settings (small set of training labels available), respectively.
Revised: [19] further extended the Mah model for unsupervised learning (no training labels available) and semi-supervised learning settings (small set of training labels available), respectively.

Original: However, the strong parametricassumption for these studies that class-conditional feature representation follows Gaussian distributionoften fails in practical applications.
Revised: However, the strong parametric assumption for these studies that class-conditional feature representation follows Gaussian distribution often fails in practical applications.

Original: Liu et al.
Revised: Liu et al.

Original: [4] interpreted the softmax function from an energyperspective which assumed that log( p ( x ))
Revised: [4] interpreted the softmax function from an energy perspective which assumed that log( p ( x ))

Original: ∝ log( (cid:80) y exp( f y ( x ) /T ))
Revised: ∝ log( (cid:80) y exp( f y ( x ) /T ))

Original: where T was temperaturecoefficient and f y ( x ) denoted the classification logit for input x and class y .
Revised: where T was temperature coefficient and f y ( x ) denoted the classification logit for input x and class y .

Original: The energy methodproposed to use Helmholtz free energy − T log( (cid:80) y exp( f y ( x ) /T )) as the OOD score.
Revised: The energy method proposed to use Helmholtz free energy − T log( (cid:80) y exp( f y ( x ) /T )) as the OOD score.

Original: Nevertheless,we will explain in the method section that the underlying assumption of the energy model does notnecessarily hold, which can lead to poor estimation of p ( x ) for OOD detection.
Revised: Nevertheless, we will explain in the method section that the underlying assumption of the energy model does not necessarily hold, which can lead to poor estimation of p ( x ) for OOD detection.

Original: Some other studies reported different empirical methods for OOD detection.
Revised: Some other studies reported different empirical methods for OOD detection.

Original: For example, Sastry etal.
Revised: For example, Sastry et al.

Original: [8] detected OOD samples by defining the deviation of test samples from training data based on alayer-wise Gram matrix that encoded pairwise feature correlations between channels.
Revised: [8] detected OOD samples by defining the deviation of test samples from training data based on a layer-wise Gram matrix that encoded pairwise feature correlations between channels.

Original: Inspired bythe contrastive learning strategy, Tack et al.
Revised: Inspired by the contrastive learning strategy, Tack et al.

Original: [22] contrasted different types of augmented trainingsamples and proposed an empirical OOD score by measuring the distance between the test sampleand nearest training samples in contrastive representation.
Revised: [22] contrasted different types of augmented training samples and proposed an empirical OOD score by measuring the distance between the test sample and nearest training samples in contrastive representation.

Original: Additionally, Sun et al.
Revised: Additionally, Sun et al.

Original: [1] observed that
Revised: [1] observed that

Original: OOD data tend to trigger unit activation with larger variations and positive skewness.
Revised: OOD data tend to trigger unit activation with larger variations and positive skewness.

Original: In light of this,the proposed ReAct model [1] directly truncated high activations in the penultimate layer, which wasexperimentally confirmed to be helpful in improving the performance of common OOD scores.
Revised: In light of this, the proposed ReAct model [1] directly truncated high activations in the penultimate layer, which was experimentally confirmed to be helpful in improving the performance of common OOD scores.

Original: In summary, the existing methods have one or more of the following limitations: (1) Due tolack of clear probabilistic definition of OOD detection, some studies identify OOD samples usingestimated discriminative probability p ( y | x ) instead of data distribution p ( x ) , which is susceptibleto the influence of overconfident posterior; (2) Some researchers estimate p ( x ) by imposing strongexplicit parametric or implicit assumptions of high-level extracted features f ( x ) , which howeverare not necessarily justified and can impair the OOD detection performance; (3) Several empiricalmethods lack a clear theoretical motivation.
Revised: In summary, the existing methods have one or more of the following limitations: (1) Due to lack of clear probabilistic definition of OOD detection, some studies identify OOD samples using estimated discriminative probability p ( y | x ) instead of data distribution p ( x ) , which can be susceptible to the influence of overconfident posterior; (2) Some researchers estimate p ( x ) by imposing strong explicit parametric or implicit assumptions of high-level extracted features f ( x ) , which however are not necessarily justified and can impair the OOD detection performance; (3) Several empirical methods lack a clear theoretical motivation.

Original: To overcome these limitations, we will explore how to reliably estimate ID data distributionp ( x ) for OOD detection using the high-level representation features.
Revised: To overcome these limitations, we will explore how to reliably estimate ID data distribution p ( x ) for OOD detection using the high-level representation features.

Original: First, we consider it isinappropriate to estimate the sample distribution p ( x ) directly from the distribution of latent featurerepresentation learned by the discriminative network since it only concerns the conditional probabilityp ( y | x ) without any constraints to guarantee the reliability of the estimated p ( x ) .
Revised: First, we consider it is inappropriate to estimate the sample distribution p ( x ) directly from the distribution of latent feature representation learned by the discriminative network since it only concerns the conditional probability p ( y | x ) without any constraints to guarantee the reliability of the estimated p ( x ) .

Original: To address thisissue, we propose a novel density-consistency regularization strategy: given that the distribution oflow dimensional variable Y can be estimated with high accuracy, this study enforces the consistencybetween empirical density (cid:98) p ( y ) and the estimated density p ( y )
Revised: To address this issue, we propose a novel density-consistency regularization strategy: given that the distribution of low dimensional variable Y can be estimated with high accuracy, this study enforces the consistency between empirical density (cid:98) p ( y ) and the estimated density p ( y )

Original: = (cid:82) p ( x, y )
Revised: = (cid:82) p ( x, y )

Original: dx = (cid:82) p ( x ) p ( y | x ) dx derivedfrom latent feature representation in deep layers, e.g., the last logit layer, on a batch-by-batchbasis.
Revised: dx = (cid:82) p ( x ) p ( y | x ) dx derived from latent feature representation in deep layers, e.g., the last logit layer, on a batch-by-batch basis.

Original: Theoretically, with sufficient training samples (batch numbers), combining this consistencyregularization with common supervised loss can reliably  estimate both p ( y | x ) and p ( x ) , thus enablingeffective OOD detection.
Revised: Theoretically, with sufficient training samples (batch numbers), combining this consistency regularization with common supervised loss can reliably calibrate and estimate both p ( y | x ) and p ( x ) , thus enabling effective OOD detection.

Original: Secondly, to ensure OOD samples highly deviated from the ID distributioncan also be identified, we further propose contrastive distribution regularization to produce lowdensity estimates for the distribution deviated samples.
Revised: Secondly, to ensure OOD samples highly deviated from the ID distribution can also be identified, we further propose contrastive distribution regularization to produce low-density estimates for the distribution deviated samples.

Original: These two regularizations are properlycombined to further improve the detection accuracy.
Revised: These two regularizations are properly combined to further improve the detection accuracy.

Original: The following methods section will investigate how to achieve the above density regularizations.
Revised: The following methods section will investigate how to achieve the above density regularizations.

Original: Itincludes building the consistency criterion with the law of large numbers, analyzing asymptotic prop- erties for the deviation from consistency with the central limit theorem, developing asymptotic-testbased regularization for the consistency hypothesis, and finally incorporating density regularizationwith contrastive distributions.
Revised: It includes building the consistency criterion with the law of large numbers, analyzing asymptotic properties for the deviation from consistency with the central limit theorem, developing asymptotic-test based regularization for the consistency hypothesis, and finally incorporating density regularization with contrastive distributions.

Original: Our contribution can be summarized as follows: • We propose a unified probabilistic framework for OOD detection in discriminative networks, which reveals that direct estimation of ID data distribution p ( x ) via latent feature representationscan be biased and introduces density-consistent and contrastive criteria to address this issue.
Revised: Our contribution can be summarized as follows: (1) We propose a unified probabilistic framework for OOD detection in discriminative networks, which reveals that direct estimation of ID data distribution p ( x ) via latent feature representations can be biased and introduces density-consistent and contrastive criteria to address this issue.

Original: Unlike previous studies, our method does not rely on any unproven assumptions.
Revised: Unlike previous studies, our method does not rely on any unproven assumptions.

Original: • We derive analytical regularization terms by analyzing the asymptotic properties of the density- consistent and contrastive criteria, and provide a simple yet surprisingly effective computational algorithm for OOD detection applicable to any deep networks with a softmax classifier.
Revised: (2) We derive analytical regularization terms by analyzing the asymptotic properties of the density-consistent and contrastive criteria, and provide a simple yet surprisingly effective computational algorithm for OOD detection applicable to any deep networks with a softmax classifier.

Original: • Extensive experiments on various standard OOD detection benchmarks demonstrate that the proposed density-driven OOD detector achieves state-of-the-art performance compared with many other existing methods.
Revised: (3) Extensive experiments on various standard OOD detection benchmarks demonstrate that the proposed density-driven OOD detector achieves state-of-the-art performance compared with many other existing methods.

Original:  Related work
Revised:  Related work

Original: [Figure]

Original: In general, OOD detectors can be broadly categorized into generative and discriminative approaches.
Revised: In general, OOD detectors can be broadly categorized into generative and discriminative approaches.

Original: OOD detection with generative models Deep generative models have been explored for OODdetection [2, 6, 7,  11, 23–28] as an intuitive strategy since they enable modeling likelihoods p θ ( x ) forhigh dimensional input.
Revised: OOD detection with generative models Deep generative models have been explored for OOD detection [6, 7, 23, 11, 24, 2, 25–28] as an intuitive strategy since they enable modeling likelihoods p θ ( x ) for high dimensional input.

Original: One contributing aspect of the generative approach is that it directly modelsthe input distribution without using labeled data[11].
Revised: One contributing aspect of this approach is that it directly models the input distribution without using labeled data[11].

Original: However, it requires additional generativenetworks for classification tasks, which can be prohibitively challenging to optimize [1, 4, 29].
Revised: However, it requires additional generative networks for classification tasks, which can be prohibitively challenging to optimize [4, 29, 1].

Original: Moreover,many studies have highlighted significant issues  that deep generative models can frequentlyassign spuriously higher likelihoods to OOD than ID samples [1, 2, 4–8, 10, 11, 23–25, 27, 30, 31],and the performance can often lag behind the discriminative approaches [1, 7, 29].
Revised: Moreover, significant issues have been highlighted that deep generative models can frequently assign spuriously higher likelihoods to OOD than ID samples [2, 7, 4, 25, 11, 24, 30, 10, 23, 6, 5, 1, 8, 31, 27], and the performance can often lag behind the discriminative approaches [7, 1, 29].

Original: Recent studiesrevealed that it might be because deep generative models can be confounded by background statistics[11], or because of the discordance between the model’s typical set and high-density region [23].
Revised: Recent studies revealed that it might be because deep generative models can be confounded by background statistics [11], or because of the discordance between the model’s typical set and high-density region [23].

Original: Discriminative model-based OOD detection
Revised: Discriminative model-based OOD detection

Original: The discriminative model-based approach, whichderives OOD score by using representations from a predefined classification task, has advantages overgenerative-based approaches by circumventing the difficult optimization process and obtaining betterdetection performance [1,  4, 6, 9], especially for complex datasets [22].
Revised: The discriminative model-based approach, which derives OOD score by using representations from a predefined classification task, has advantages over generative-based approaches by circumventing the difficult optimization process and obtaining better detection performance [1, 9, 4, 6], especially for complex datasets [22].

Original: This can be because supervised learning effectively learns discriminative representations useful in identifying in-distributionsamples by exploiting label information [6, 8, 19].
Revised: This can be because supervised learning effectively learns discriminative representations useful in identifying in-distribution samples by exploiting label information [8, 6, 19].

Original: A brief overview of the baseline Msp model [16]and state-of-the-art discriminative OOD detection methods, including Odin [17], Mah [9], Gram [8],
Revised: A brief overview of the baseline Msp model [16] and state-of-the-art discriminative OOD detection methods, including Odin [17], Mah [9], Gram [8],

Original: Energy [4], pNML [3] and ReAct [1], is presented in the introduction section.
Revised: Energy [4], pNML [3] and ReAct [1], is presented in the introduction section.

Original: Discriminative model-based OOD detection with outlier exposure Some researchers haveexplored using auxiliary datasets of outliers, which allow the model to be explicitly regularizedthrough fine-tuning, to increase the margin between the OOD scores of the ID and OOD data.
Revised: Discriminative model-based OOD detection with outlier exposure Some researchers have explored using auxiliary datasets of outliers, which allow the model to be explicitly regularized through fine-tuning, to increase the margin between the OOD scores of the ID and OOD data.

Original: Suchan outlier exposure strategy has been reported by Oe [10], Oecc [32] and Energy [4] models tobe helpful inimproving OOD detection performance.
Revised: Such an outlier exposure strategy has been reported by Oe [10], Oecc [32] and Energy [4] to be helpful in OOD detection.

Original: However, OOD samples are different andinfinite.
Revised: However, OOD samples are different and infinite.

Original: To avoid specific exposure OODs inducing a bias to the detector, researchers typically used alarge-scale diverse dataset as the outlier exposure dataset [4, 10, 32], e.g., 80 Million Tiny Images.
Revised: To avoid specific exposure OODs inducing biases, researchers typically used a large-scale diverse dataset as the outlier exposure dataset [10, 32, 4], e.g., 80 Million Tiny Images.

Original: Nevertheless, it is often not possible to have such a large amount of external data in many real-worldtasks, especially those with large ID datasets.
Revised: Yet it is often not possible to have such a large amount of external data in many real-world tasks, especially those with large ID datasets.

Original:  Methods 
Revised:  Methods 3.1 Probabilistic framework for OOD detection

Original: 3.1 Probabilistic framework for OOD detection

Original: The goal of a machine learning classifier is to learn the mapping from X to Y , where input X is usuallya high-dimensional random vector and Y is a discrete random variable.
Revised: The goal of a machine learning classifier is to learn the mapping from X to Y , where input X is usually a high-dimensional random vector and Y is a discrete random variable.

Original: Let g ( x, y ) be the probabilitydensity function (p.d.f.) of ( X, Y ) with respect to measure ν
Revised: Let g ( x, y ) be the probability density function (p.d.f.) of ( X, Y ) with respect to measure ν

Original: X × π Y , where ν X is Lebesgue measurein Euclidean space R m and π Y is discrete measure with π Y ( Y = y )
Revised: X × π Y , where ν X is Lebesgue measure in Euclidean space R m and π Y is discrete measure with π Y ( Y = y )

Original: = 1 , 1 ≤ y ≤ L .
Revised: = 1 , 1 ≤ y ≤ L .

Original: L is the number of classification categories.
Revised: L is the number of classification categories.

Original: The classification probabilities for discriminant model P ( Y | X )can be represented
Revised: The classification probabilities for discriminant model P ( Y | X ) can be represented

Original: [Equation]
Revised: [Equation]

Original: For deep learning models, researchers generally considered the following softmax values as P ( Y | X ) .
Revised: For deep learning models, researchers generally considered the following softmax values as P ( Y | X ) .

Original: [Equation]
Revised: [Equation]

Original: where h y ( X ) is the classification logits produced by the deep neural networks.
Revised: where h y ( X ) is the classification logits produced by the deep neural networks.

Original: By connecting
Revised: By connecting

Original: Eq.(1) and Eq.(2), exp( h y ( x )) can be considered as the Boltzmann-Gibbs form for g ( x, y ) from anenergy-based perspective [33, 34], i.e., 
Revised: Eq.(1) and Eq.(2), exp( h y ( x )) can be considered as the Boltzmann-Gibbs form for g ( x, y ) from an energy-based perspective [33, 34], i.e., [Equation] x,

Original: [Equation]
Revised: [Equation]

Original:  Alternatively, g ( x, y ) = 1 Z exp ( h y ( x )) and denominator Z ≜ (cid:80) 1 ≤ y ≤ L (cid:82) exp ( h y ( X ))
Revised: h [Equation] Alternatively, g ( x, y ) = 1 Z exp ( h y ( x )) and denominator Z ≜ (cid:80) 1 ≤ y ≤ L (cid:82) exp ( h y ( X ))

Original: dν X .
Revised: dν X .

Original: For OOD detection, we first argue that OOD score should be  constructed from P ( X ) ratherthan P ( Y | X ) , which is commonly used in previous studies [3, 5, 16, 17], since the estimated P ( Y = y
Revised: For OOD detection, we first argue that OOD score can be more appropriately constructed from P ( X ) rather than P ( Y | X ) , where the latter is commonly used in previous studies [16, 17, 5, 3], since the estimated P ( Y = y

Original: | X = x ) can create over-confidence probability [4, 9] even when P ( X = x ) is relativelysmall.
Revised: | X = x ) can create over-confidence probability [9, 4] even when P ( X = x ) is relatively small.

Original: In terms of P ( X ) , we can obtain the following estimated p.d.f.
Revised: In terms of P ( X ) , we can obtain the following estimated p.d.f.

Original: k ( x ) based on Eq.(3):
Revised: k ( x ) based on Eq.(3):

Original: [Equation]
Revised: [Equation]

Original: k ( x ) appears to be a reasonable OOD detection score.
Revised: k ( x ) appears to be a reasonable OOD detection score.

Original: Similarly, Liu et al. proposed to use Helmholtzfree energy − T log( (cid:80) 1 ≤ y ≤ L exp( h y ( x ) /T ))
Revised: Similarly, Liu et al. proposed to use Helmholtz free energy − T log( (cid:80) 1 ≤ y ≤ L exp( h y ( x ) /T ))

Original: as OOD score with temperature T [4], which is similarto k ( x ) from a computational perspective, and demonstrated its advantage over OOD scores basedon discriminative probability P ( Y | X ) [16, 17].
Revised: as OOD score with temperature T [4], which is similar to k ( x ) from a computational perspective, and demonstrated its advantage over OOD scores based on discriminative probability P ( Y | X ) [16, 17].

Original: However, directly calculating k ( x ) as the OOD score can lead to poor density estimationsince the prerequisite Eq. (3) is not clearly satisfied, which has been overlooked in previousstudies.
Revised: However, directly calculating k ( x ) as the OOD score can lead to poor density estimation since the prerequisite Eq.(3) is not clearly satisfied, which has been overlooked in previous studies.

Original: Particularly, classification models generally focus on improving the accuracy of P ( Y | X ) byclassification loss L cls without explicitly enforcing the validity of Eq.(3).
Revised: Particularly, classification models generally focus on improving the accuracy of P ( Y | X ) by classification loss L cls without explicitly enforcing the validity of Eq.(3).

Original: Therefore, this study aimsto develop an explicit regularization  to satisfy the constraint in Eq.(3) based on the empirical densityestimation of the ID training data.
Revised: Therefore, this study firstly aims to develop an explicit regularization for density calibration to satisfy the constraint in Eq.(3) based on the empirical density estimation of the ID training data.

Original: Specifically, since the empirical density function (EDF) of alow-dimensional random variable (r.v.) can be estimated with high accuracy, we start with estimatingthe EDF of discrete r.v. Y and enforce the consistency between its EDF and its analytical parametricp.d.f.
Revised: Specifically, since the empirical density function (EDF) of a low-dimensional random variable (r.v.) can be estimated with high accuracy, we start with estimating the EDF of discrete r.v. Y and enforce the consistency between its EDF and its analytical parametric p.d.f.

Original: of the following form based on Eq.(3).
Revised: of the following form based on Eq.(3).

Original: [Equation]
Revised: [Equation]

Original: In the following sections, we will construct the analytical consistency regularization and implementation algorithm for ensuring the constraints of Eq.(3), thereby improving OOD detection performance,using the law of large numbers (LLN), central limit theorem (CLT) and asymptotic test statistics.
Revised: In the following sections, we will construct the analytical consistency regularization and implementation algorithm for ensuring the constraints of Eq.(3), thereby improving OOD detection performance, using the law of large numbers (LLN), central limit theorem (CLT) and asymptotic test statistics.

Original: 3.2 Consistency criterion for low-dimensional density estimation
Revised: 3.2 Consistency criterion for low-dimensional density estimation

Original: As noted above, density function k
Revised: As noted above, density function k

Original: ( x ) , estimated by high layer features, can be a useful OOD score ifthe underlying assumption Eq.(3) holds.
Revised: ( x ) , estimated by high layer features, can be a useful OOD score if the underlying assumption Eq.(3) holds.

Original: To realize this assumption, we now enforce the consistencybetween analytical p.d.f v ( y ) derived by Eq.(3) and the empirical density function (EDF) for lowdimensional variable Y , which can be reliably estimated by (cid:98) v ( y )
Revised: To realize this assumption, we now enforce the consistency between analytical p.d.f v ( y ) derived by Eq.(3) and the empirical density function (EDF) for lowdimensional variable Y , which can be reliably estimated by (cid:98) v ( y )

Original: =
Revised: =

Original: n y /N .
Revised: n y /N .

Original: n y = (cid:80) 1 ≤ i ≤ N 1 { Y i = y }represents the number of samples in class y and N = (cid:80)
Revised: n y = (cid:80) 1 ≤ i ≤ N 1 { Y i = y } represents the number of samples in class y and N = (cid:80)

Original: Ly =1 n y is the sample size.
Revised: Ly =1 n y is the sample size.

Original: Namely, with
Revised: Namely, with

Original: Eq.(3), we derive the following property:
Revised: Eq.(3), we derive the following property:

Original: [Equation]
Revised: [Equation]

Original: By the law of large numbers, (cid:98) v ( y ) converges to v ( y ) almost surely (a.s.) and in probability as
Revised: By the law of large numbers, (cid:98) v ( y ) converges to v ( y ) almost surely (a.s.) and in probability as

Original: N
Revised: N

Original: → ∞ .
Revised: → ∞ .

Original: Thus, the pivotal variable (cid:82) exp( h y ( X ))) (cid:98) v ( y ) dν
Revised: Thus, the pivotal variable (cid:82) exp( h y ( X ))) [Equation] v [Equation]

Revised: y

Revised: [Equation]

Original:  X converges to Z a.s. and in probability.
Revised: dν X converges to Z a.s. and in probability.

Original: By the
Revised: By the

Original: Monte Carlo method, Z can be approximated by:
Revised: Monte Carlo method, Z can be approximated by:

Original: [Equation]
Revised: [Equation]

Original: Obviously, the right-hand side (RHS) of Eq.(7) also converges to Z a.s. and in probability as N → ∞ .
Revised: Obviously, the right-hand side (RHS) of Eq.(7) also converges to Z a.s. and in probability as N → ∞ .

Original: Thus, we arrive at the proposition that for each class 1 ≤ y ≤ L , the exponential of classificationlogit has a mean value proportional to the class ratio.
Revised: Thus, we arrive at the proposition that for each class 1 ≤ y ≤ L , the exponential of classification logit has a mean value proportional to the class ratio.

Original: Since deep learning models are trained in a batch mode, in which random batch sampling withreplacement allows bootstrap estimates, we now proceed to construct batch-wise statistics andanalytical regularization for Eq.(7).
Revised: Since deep learning models are trained in a batch mode, in which random batch sampling with replacement allows bootstrap estimates, we now proceed to construct batch-wise statistics and analytical regularization for Eq.(7).

Original: We first give the following lemma.
Revised: We first give the following lemma.

Original: The proof is in Appendix A.1of the supplementary material.
Revised: The proof is in Appendix A. of the supplementary material.

Original: Lemma 1.
Revised: Lemma 1.

Original: If Eq.
Revised: If Eq.

Original: (3) holds, then
Revised: (3) holds, then

Original: [Equation]
Revised: [Equation]

Original: Lemma 1 gives a theoretical batch-wise constraint for the classification logit.
Revised: Lemma 1 gives a theoretical batch-wise constraint for the classification logit.

Original: However, the convergence rate is not specified here, making it challenging to design an adaptive regularization subjectedto varying class sample n y and batch size N .
Revised: However, the convergence rate is not specified here, making it challenging to design an adaptive regularization subjected to varying class sample n y and batch size N .

Original: To address this issue, in the next section, we willspecify the convergence rate of O (1 / √ N ) for Eq.(8) using the central limit theorem and accordinglypropose the adaptive consistency regularization by our asymptotic teststatistics.
Revised: To address this issue, in the next section, we will specify the convergence rate of O (1 / √ N ) for Eq.(8) using the central limit theorem and accordingly propose the adaptive consistency regularization by our asymptotic test

Original:  3.3 Adaptive density-consistency regularization by asymptotic test
Revised: statistics. 3.3 Adaptive density-consistency regularization by asymptotic test

Original: To construct the density-consistency regularization adaptive to different class size n y and batch size
Revised: To construct the density-consistency regularization adaptive to different class size n y and batch size

Original: N , we now consider the convergence rate with respect to sample size for Eq.(8) by the followinglemma 2.
Revised: N , we now consider the convergence rate with respect to sample size for Eq.(8) by the following lemma 2.

Original: The proof is provided in Appendix A.2.
Revised: The proof is provided in Appendix A.2.

Original: Lemma 2.
Revised: Lemma 2.

Original: If Eq.
Revised: If Eq.

Original: (3) holds, then
Revised: (3) holds, then

Original: [Equation]
Revised: [Equation]

Original: 9) where ∆ y =
Revised: where ∆ y =

Original: [ ∇ g ( µ y )] ⊤ Σ y [ ∇ g ( µ y )] and g ([ x, y, z ] ⊤ ) =
Revised: [ ∇ g ( µ y )] ⊤ Σ y [ ∇ g ( µ y )] and g ([ x, y, z ] ⊤ ) =

Original: y/x − z .
Revised: y/x − z .

Original: µ y and Σ y represent the meanand covariance matrix for random vector u y = (cid:104)(cid:80) 1{+
+} ≤ l ≤ L exp (cid:0) h l ( X ) (cid:1) , exp ( h y ( X )) , 1 { Y = y } (cid:105) ⊤ .
Revised: µ y and Σ y represent the mean and covariance matrix for random vector u y = (cid:104)(cid:80) {+
+} ≤ l ≤ L exp (cid:0) h l ( X ) (cid:1) , exp ( h y ( X )) , 1 { Y = y } (cid:105) ⊤ .

Original: Lemma 1 and lemma 2, in fact, show that if implicit assumption Eq.(3) holds, i.e., 1 Z exp( h y ( x )) canaccurately approximate density g ( x, y ) , we can then construct batch-wise statistics Υ yN , in which1 √ N Υ yN converges to zero at the rate of O ( N − 1 / 2 ) by analyzing the distribution of low-dimensional
Revised: Lemma 1 and lemma 2, in fact, show that if implicit assumption Eq.(3) holds, i.e., 1 Z exp( h y ( x )) can accurately approximate density g ( x, y ) , we can then construct batch-wise statistics Υ yN , in which√ N Υ yN converges to zero at the rate of O ( N − 1 / 2 ) by analyzing the distribution of low-dimensional

Original: Y .
Revised: Y .

Original: More specifically, Υ yN converges to a particular Gaussian distribution whose parameter can bereliably estimated.
Revised: More specifically, Υ yN converges to a particular Gaussian distribution whose parameter can be reliably estimated.

Original: Thus to satisfy assumption Eq. (3) , which is deemed to provide a more accurate estimation of 
Revised: Thus to satisfy assumption Eq.(3), which is deemed to provide a more accurate estimation of P ( x ) for

Original: P ( x ) for OOD detection, we propose to construct the asymptotic-test statistics using Υ yN fromthe perspective of hypothesis testing, and creatively transform the hypothesis reject criteriainto a penalty term to support the assumption Eq. (3) can be accepted at any specific level ofsignificance.
Revised: OOD detection, we propose to construct the asymptotic-test statistics using Υ yN from the perspective of hypothesis testing, and creatively transform the hypothesis reject criteria into a penalty term to support the assumption Eq.(3) can be accepted at any specific level of significance.

Original: Specifically, we consider the following hypotheses:
Revised: Specifically [Equation] we consider the following hypotheses:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: = [Equation] First, assumption Eq.(3) implies null hypothesis H 0 , and accordingly leads to the following convergence in distribution by lemma 2:
Revised: First, assumption Eq.(3) implies null hypothesis H 0 , and accordingly leads to the following convergence in distribution by lemma 2:

Original: [Equation]
Revised: [Equation]

Original: where ∆ yN ≜ [ ∇ g ( µ yN )] ⊤ Σ yN [ ∇ g ( µ yN )] approximates and converges to ∆ y in probability.
Revised: where ∆ yN ≜ [ ∇ g ( µ yN )] ⊤ Σ yN [ ∇ g ( µ yN )] approximates and converges to ∆ y in probability.

Original: µ yN andΣ yN denote sample mean and sample covariance for µ y and Σ y , respectively.
Revised: µ yN and Σ yN denote sample mean and sample covariance for µ y and Σ y , respectively.

Original: From the perspective ofasymptotic test theory [35], we can construct a χ 2 -test with the following reject region which hasasymptotic significance level α .
Revised: From the perspective of asymptotic test theory [35], we can construct a χ 2 -test with the following reject region which has asymptotic significance level α .

Original: [Equation]
Revised: [Equation]

Original: where χ 21 ,α is the (1 − α ) th quantile of chi-squared distribution χ 2 (1) .
Revised: where χ 21 ,α is the (1 − α ) th quantile of chi-squared distribution χ 2 (1) .

Original: When Eq.(13) is observed,the asymptotic test rejects H 0 , as well as assumption Eq.(3).
Revised: When Eq.(13) is observed, the asymptotic test rejects H 0 , as well as assumption Eq.(3).

Original: Hence, to ensure the assumption Eq. (3) is rightly accepted, we propose to transform the hypothesis test rejection criteria into thefollowing regularization term:
Revised: Hence, to ensure the assumption Eq.(3) is rightly accepted, we propose to transform the hypothesis test rejection criteria into the following regularization term:

Original: [Equation]
Revised: [Equation]

Original: The regularization term λ will be added to the classification loss L cls , where hyperparameter α ∈ [0 , 1]controls the significance level and γ is the loss weight balancing the regularization term λ andclassification loss L cls .
Revised: The regularization term λ will be added to the classification loss L cls , where hyperparameter α ∈ [0 , 1] controls the significance level and γ is the loss weight balancing the regularization term λ and classification loss L cls .

Original: 3.4 Density regularization with contrastive distribution
Revised: 3.4 Density regularization with contrastive distribution

Original: The above density-consistency regularization (DCR) can calibrate classification logit based onobservable low-dimensional density P ( Y ) , which offers a more reliable P ( X, Y ) estimate for IDlike samples.
Revised: The above density-consistency regularization (DCR) can calibrate classification logit based on observable low-dimensional density P ( Y ) , which offers a more reliable P ( X, Y ) estimate for IDlike samples.

Original: Nevertheless, the real test OOD samples can come from distribution deviating farfrom the ID distribution.
Revised: Nevertheless, the real test OOD samples can come from distribution deviating far from the ID distribution.

Original: To ensure these samples with large distribution deviations can also beaccurately identified, we further propose the density-contrasting scheme, which enforces that randomdistribution-deviated (DD) samples produce low-density estimates.
Revised: To ensure these samples with large distribution deviations can also be accurately identified, we further propose the density-contrasting scheme, which enforces that random distribution-deviated (DD) samples produce low-density estimates.

Original: We first define the distribution-deviating operator S , which can be applied to ID data X to generate
Revised: We first define the distribution-deviating operator S , which can be applied to ID data X to generate

Original: DD samples (cid:101) X = S ( X ) .
Revised: DD samples (cid:101) X = S ( X ) .

Original: An ideal density estimator should yield the likelihood ratio P ( (cid:101) X ) P ( X ) ≪ 1 and[ P ( (cid:101) X ) P ( X ) ] r ≈ 0 for large constant r , i.e.,
Revised: An ideal density estimator should yield the likelihood ratio P ( (cid:101) X ) P ( X ) ≪ 1 and [ P ( (cid:101) X ) P ( X ) ] r ≈ 0 for large constant r , i.e.,

Original: [Equation]
Revised: [Equation]

Original: We incorporate this constraint into the above DCR by defining (cid:101) Υ yN :
Revised: We incorporate this constraint into the above DCR by defining (cid:101) Υ yN :

Original: [Equation]
Revised: [Equation]

Revised: where S denotes the distribution-deviating operator set.

Original: Here a constant q is added to ensure numerical stability.
Revised: Here a constant q is added to ensure numerical stability.

Original: Since the second term in RHS is consideredclose to zero, we can obtain that[ (cid:101) Υ yN ] ⊤ [∆ yN ] − 1 [ (cid:101) Υ yN ] also converges to χ 2 (1) approximately.
Revised: Since the second term in RHS is considered close to zero, we can obtain that [∆ [Equation] N ] [Equation]

Revised: also converges to χ 2 (1) approximately.

Original: Thus,our final generalized density regularization with contrastive distribution can be defined:
Revised: Thus, our final generalized density regularization with contrastive distribution can be defined:

Original: [Equation]
Revised: [Equation]

Original:  Algorithm 1in Appendix A.3 presents the pseudo-code for the training of our density driven regularization method.
Revised: Figure 1 shows the overview of the proposed density driven regularization (DDR) for OOD detection and Algorithm 1 presents the pseudo-code for the training of our DDR method 3 .

Original: It is worth noting that to increase numerical stability, we normalize the classificationlogit by subtracting the maximal logit before calculating (cid:101) λ in Algorithm 1.
Revised: It is worth noting that to increase numerical stability, we normalize the classification logit by subtracting the maximal logit before calculating (cid:101) λ in Algorithm 1.

Original: This normalization procedure is also widely used to avoid overflow when calculating softmax probability [36].
Revised: This normalization procedure is also widely used to avoid overflow when calculating softmax probability [36].

Original: We will show,in Appendix A.4, that the normalized logit essentially yields the same results as the original logit.
Revised: We will show, in Appendix A.3, that the normalized logit essentially yields the same results as the original logit.

Original: Finally, after fine-tuning, a monotonic transformation log( Z · k ( x ))
Revised: Finally, after fine-tuning, a monotonic transformation log( Z · k ( x ))

Original: =
Revised: =

Original: log( (cid:80) 1 ≤ y ≤ L exp( h y ( x ))) ofthe sample density k ( x ) is used as the OOD score in this study for any testing sample x , which istheoretically equivalent to directly adopting k ( x ) as the OOD score. 
Revised: log( (cid:80) 1 ≤ y ≤ L exp( h y ( x ))) of the sample density k ( x ) is used as the OOD score in this study for any testing sample x , which is theoretically equivalent to directly adopting k ( x ) as the OOD score. deep classifier distribution-

Revised: deviating operators 𝑷 𝒙 𝒊 ,𝒚 𝒍 𝑷 ( 𝒚 𝒍 )

Revised: CDR loss

Original: Experiments
Revised: DCRloss ………… ……………… 𝑿 ෩ 𝑿 … … 𝑷(𝒙 𝒊 ) … 𝑷 ( 𝒀 ) N o r m a li z e d h i s t o g r a m 0.

Revised: 0.6 (d) (e) 0.2 (b) 0.

Revised: (c) 0.

Revised: N o r m a li z e d h i s t

Revised: o g r a m 𝑷 ( 𝒀 ) (a) 𝑷(෥𝒙 𝒊 ,

Revised: 𝒚 𝒍 ) 𝑃 𝑋 𝑃 ෨𝑋 𝑃 𝑋 𝑃 ෨𝑋 𝑷(෥𝒙 𝒊 ) 0.

Original: We demonstrate the effectiveness of the proposed method using diverse deep convolutional neuralnetwork architectures on multiple OOD detection benchmark datasets, and compare it against variousstate-of-the-art methods.
Revised: We demonstrate the effectiveness of the proposed method using diverse deep convolutional neural network architectures on multiple OOD detection benchmark datasets, and compare it against various state-of-the-art methods.

Original: Datasets and experimental setup Following existing literature [1, 3,  4, 8–10, 16, 17, 19, 32],we train OOD models on the training set of CIFAR-10 [37], CIFAR-100 [37] and ImageNet [38],respectively, and evaluate OOD detectors in identifying OOD test samples from other datasets,including iSUN [39], iNaturalist [40], LSUN(R) [17], LSUN(C)
Revised: Datasets and experimental setup Following existing literature [1, 3, 19, 32, 4, 8, 10, 9, 17, 16], we train OOD models on the training set of CIFAR-10 [37], CIFAR-100 [37] and ImageNet [38], respectively, and evaluate OOD detectors in identifying OOD test samples from other datasets, including iSUN [39], iNaturalist [40], LSUN(R) [17], LSUN(C)

Original: [17], Gaussian [17], Places365 [41],
Revised: [17], Gaussian [17], Places365 [41],

Original: Textures [42] and SVHN [43].
Revised: Textures [42] and SVHN [43].

Original: ImageNet(R) [3] and Imagenet(C)
Revised: ImageNet(R) [3] and Imagenet(C)

Original: [3] are also used as OOD sets forthe CIFAR experiment.
Revised: [3] are also used as OOD sets for the CIFAR experiment.

Original: For the ImageNet experiment, CIFAR datasets are considered as OOD sets.
Revised: For the ImageNet experiment, CIFAR datasets are considered as OOD sets.

Original: These datasets are described and provided in previous studies [1, 4, 17].
Revised: These datasets are described and provided in previous studies [17, 4, 1].

Original: For fair comparison and to aid reproducibility, we use the pretrained model for standard ResNet-34[44], WideResNet-40 [45] and DenseNet-BC-100 [46] as the backbone network for the CIFAR experiment since they are widely used in previous OOD detection studies [1, 3,  4, 8–10, 17, 32].
Revised: For fair comparison and to aid reproducibility, we use the pretrained model for standard ResNet- [44], WideResNet-40 [45] and DenseNet-BC-100 [46] as the backbone network for the CIFAR experiment since they are widely used in previous OOD detection studies [1, 3, 32, 4, 8, 10, 9, 17].

Original: Following[1], we adopt the lightweight MobileNet-v2 [47] for the ImageNet experiment.
Revised: Following [1], we adopt the lightweight MobileNet-v2 [47] for the ImageNet experiment.

Original: It is noteworthy thatsince some of the following compared models, e.g., Gram [8] and Mah [9], require prohibitively largememory for large networks with high input dimensions, we only conduct experiments on lightweight MobileNet-v2 for the ImageNet dataset.
Revised: It is noteworthy that since some of the following compared models, e.g., Gram [8] and Mah [9], require prohibitively large memory for large networks with high input dimensions, we only conduct experiments on lightweight MobileNet-v2 for the ImageNet dataset.

Original: The pretrained weights for the above ResNet, WideResNet, DenseNet and MobileNet are provided by [1, 9, 17].
Revised: The pretrained weights for the above ResNet, WideResNet, DenseNet and MobileNet are provided by [17, 9, 1].

Original: When training our OOD detector, we use the same normalization, augmentation and training settingas in [4, 10], where thefine-tuning epoch is 10, and the initial learning rate  is 0.001 with cosinedecay [48].
Revised: When training our OOD detector, we use the same normalization, augmentation and training setting as in [10, 4], where the learning rate was adjusted by cosine decay schedule [48] and the fine-tuning epoch is 10.

Revised: The initial learning rates are set 10 − 3 and 10 − 4 for CIFAR and ImageNet experiments.

Original: Other hyperparameters are set as follows: batch size of 256, regularization weightγ = 10 − 3 , constant r =100, and fixed statistical significance level α =0.05.
Revised: Other hyperparameters are set as follows: batch size of 256, regularization weight γ = 10 − 2 , constan r =10, and fixed statistical significance level α =0.05.

Original: Constant q is a detachedtensor dynamically determined to linearly map the exponent from [ − 200 r, 0] to [ − 3 , 3] for ensuringnumerical stability.
Revised: Constant q is a detached tensor dynamically determined to linearly map the exponent from [ − 200 r, 0] to [ − 5 , 5] for ensuring numerical stability.

Original: It is worth noting that unlike many previous studies [4, 8–10, 17,  32], whichadjusted hyperparameters for different datasets or networks, our method can achieve state-ofthe-art performance across various experiments with fixed  hyperparameters.
Revised: It is worth noting that unlike many previous studies [4, 8, 10, 17, 9, 32], which adjusted hyperparameters for different datasets or networks, our method can achieve state-of-the-art performance across various experiments with fixed values of introduced hyperparameters.

Original: Moreover, we willalso demonstrate the robustness of our approach to hyperparameter changes in our ablation studies.
Revised: Moreover, we will also demonstrate the robustness of our approach to hyperparameter changes in our ablation studies.

Original: Our distribution-deviating operators S include random resized crop and random affine transformsrealized by TorchVision library [49], and uniform noise generator (see Appendix A.5 for details).
Revised: Our distribution-deviating operators S include random resized crop and random affine transforms realized by TorchVision library [49], and uniform noise generator (see Appendix A.4 for details).

Original: The proposed approach is compared to the state-of-the-art methods, including Msp [16], Odin [17],
Revised: The proposed approach is compared to the state-of-the-art methods, including Msp [16], Odin [17],

Original: Mah [9], Energy [4], Gram [8], ReAct [1] and pNML [3] for OOD detection without outlier exposure,as well as Energy [4], Oe [10] and Oecc [32] for OOD detection with outlier exposure.
Revised: Mah [9], Energy [4], Gram [8], ReAct [1] and pNML [3] for OOD detection without outlier exposure, Input: ID training set D in , batch size N , initialized/pre-trained network f θ 0 , significance level α , loss weight γ , constant r , deviating operator set S , learning rate η and training/fine-tuning iteration T . Output: classification network f θ T for t = 0 , 1 , ..., T − 1 do sample a batch of N training samples { ( X i , Y i ) } Ni =1 from D in generate distribution-deviated samples (cid:101) X S i =

Revised: S ( X i ) for S ∈ S initialize regularized loss (cid:101) λ

Revised: =calculate logit vector h ( X i ) :

Original: All comparedmethods are evaluated with their open-source codes and trained weights.{+h ( (cid:101) X S i ) : = f θ t ( (cid:101) X S i ) h ( X i ) ← h ( X i ) − C and h ( (cid:101) X S i ) ← h ( (cid:101) X S i ) − C where C is the maximal value in all logit vectors // ensuring numerical stability define Ω = { y | (cid:80) 1 ≤ i ≤ N 
+}
Revised: = f θ t ( X i ) and {+h ( (cid:101) X S i ) : = f θ t ( (cid:101) X S i ) h ( X i ) ← h ( X i ) − C and h ( (cid:101) X S i ) ← h ( (cid:101) X S i ) − C where C is the maximal value in all logit vectors // ensuring numerical stability define Ω = { y | (cid:80) 1 ≤ i ≤ N 
+}

Revised: Y

Revised: i

Revised: =

Revised: y } > 0 , 1 ≤ y ≤ L } for y ∈ Ω do define n y = (cid:80) 1 ≤ i ≤ N 1 { Y i =

Revised: y } and u yi =

Revised: [ (cid:80) 1 ≤ l ≤ L exp (cid:0) h l ( X i ) (cid:1) , exp ( h y ( X i )) , 1 { Y i = y } ] ⊤ define ¯ ψ :=

Revised: rN | S | (cid:80)

Revised: S∈ S , 1 ≤ i ≤ N log (cid:80) 1 ≤ l ≤ L exp ( h l ( (cid:101) X S i ) ) (cid:80) 1 ≤ l ≤ L exp( h l ( X i )) calculate detached tensor q by which ¯ ψ + q linearly map the original exponent value ¯ ψ in [ − 200 r, 0] to [ − 5 , 5] // ensuring numerical stability calculate (cid:101) Υ yN = √ N (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Revised: [Table]

Revised: − n

Revised: [Table]

Revised: + exp (cid:0) ¯ ψ + q (cid:1) //

Revised: Eq.(16) calculate ∆ yN =

Revised: [ ∇ g ( µ yN )] ⊤ Σ yN [ ∇ g ( µ yN )] // µ yN and Σ yN are the sample mean and covariance matrix for µ y and Σ y computed by u yi (lemma 2) (cid:101) λ ← (cid:101) λ

Revised: + γ 1 | Ω | max (cid:16) log (cid:16) [∆ yN ] − 1 [ (cid:101) Υ yN ] 2 /χ 21 ,α (cid:17) , 0 (cid:17) //

Revised: asymptotic test based regularization by Eq.(17) calculate supervised classification loss L cls using { (softmax ( f θ t ( X i )) , Y i ) } Ni = combine loss value L cls + (cid:101) λ update network parameter θ t +1 ← θ t − η ∇ θ t

Revised: ( L cls + (cid:101) λ ) by SGD based optimizer return f θ T as well as Energy [4] and Oecc [32] for OOD detection with outlier exposure.

Revised: 300K Random

Revised: Images[10] is used as the outlier exposure dataset to reproduce the Energy method.

Revised: All compared methods are evaluated with their open-source codes and trained weights.

Original: Following existing studies[1, 3,  4, 8–10, 16, 17, 32], commonly used metrics including AUROC, AURP and FPR (at 95%
Revised: Following existing studies [1, 3, 32, 4, 8, 10, 9, 17, 16], commonly used metrics including AUROC, AURP and FPR (at 95%

Original: TPR) are used to evaluate different OOD detectors.
Revised: TPR) are used to evaluate different OOD detectors.

Original: The evaluation code is adopted from [4, 10].
Revised: The evaluation code is adopted from [4].

Original: Allexperiments run on the PyTorch framework [50] with Nvidia A100 and GeForce RTX 3090 GPUs,with a maximum RAM of 512GB.
Revised: All experiments run on the PyTorch framework [50] with Nvidia A100 and GeForce RTX 3090 GPUs, with a maximum RAM of 512GB.

Original: Experimental results The experimental results on CIFAR-100 in terms of FPR95% are shown in Table 1.
Revised: Experimental results The experimental results on CIFAR-100 in terms of FPR95% are shown in Table 1.

Revised: See Appendix A.5 for the variance of the results over ten runs with different subsets of OOD data.

Original: Compared to other state-of-the-art detectors without outlier exposure, our method achievesthe best detection accuracy for diverse OOD datasets under different network architectures.
Revised: Compared to other state-of-the-art detectors without outlier exposure, our method achieves the best detection accuracy for diverse OOD datasets under different network architectures.

Original: Theaveraged FPR95% of the proposed method is significantly lower than that of other detectors withoutoutlier exposure, and even lower than the four state-of-the-art detectors that use an external large-scaleoutlier exposure dataset with 80 million images.
Revised: The averaged FPR95% of the proposed method is significantly lower than that of other detectors without outlier exposure, and even lower than the four state-of-the-art detectors that use an external large-scale outlier exposure dataset with 80 million images.

Original: Other evaluation results in terms ofAUROC and
Revised: Other evaluation results in terms of

Original:  AUPR also reveal a consistent performance trend, which can be found in Appendix A.6.
Revised: AUROC and AUPR also reveal a consistent performance trend, which can be found in Appendix A.5.

Original: It is worthnoting that the proposed method only utilizes classification logits for OOD detection.
Revised: It is worth noting that the proposed method only utilizes classification logits for OOD detection.

Original: In contrast,the best-performed existing method, i.e., Gram, uses features across all layers to detect OODs andrequires calculating huge gram matrices dynamically using all latent features of training samples,preventing end-to-end real-time prediction and becoming intractable for deep networkstrained onlarge datasets.
Revised: In contrast, the best-performed existing method, i.e., Gram, uses features across all layers to detect OODs and requires calculating huge gram matrices dynamically using all latent features of training samples, preventing end-to-end real-time prediction and becoming intractable for deep networks

Original:  Due to the page limit, the reader is referred to Appendix A.7 for detailed resultsof the CIFAR-10 experiment, in which our method also achieves the best detection accuracy inboth FPR95%, AUROC and AUPR compared to other existing methods without outlier exposure.
Revised: [Table] [Table] trained on large datasets (See Appendix A.10 for computational cost comparision with our approach). Due to the page limit, the reader is referred to Appendix A.6 for detailed results of the CIFARexperiment, in which our method also achieves the best detection accuracy in both FPR95%, AUROC and AUPR compared to other existing methods without outlier exposure.

Original: The performance of the proposed method is also comparable to the state-of-the-art OOD detectorsthat require large-scale additional outlier exposure datasets.
Revised: The performance of the proposed method is also comparable to the state-of-the-art OOD detectors that require large-scale additional outlier exposure datasets.

Original: Tables in Appendix A.8 also provide [Table] [Table] [Table]
Revised: Tables in Appendix A.7 also provide the experimental results of

Original: [Table] [Table] the experimental results of FPR95%, AUROC and AUPR values on ImageNet.
Revised: FPR95%, AUROC and AUPR values on ImageNet.

Original: Due to the lack oflarge-scale high-resolution outlier exposure datasets, which should theoretically be much larger thanthe ID dataset, none of the outlier-exposure OOD detectors had been evaluated on the ImageNetbenchmark in literature.
Revised: Due to the lack of large-scale high-resolution outlier exposure datasets, which should theoretically be much larger than the ID dataset, none of the outlier-exposure OOD detectors had been evaluated on the ImageNet benchmark in literature.

Original: Therefore, only OOD detectors without requiring outlier exposure datasetsare evaluated.
Revised: Therefore, only OOD detectors without requiring outlier exposure datasets are evaluated.

Original: It can be observed that the proposed method also achieves the highest averageddetection accuracy on the ImageNet benchmark.
Revised: It can be observed that the proposed method also achieves the highest averaged detection accuracy on the ImageNet benchmark.

Original: It is noteworthy that our method performs relativelypoorly on the iNaturalists and Places365 compared to other OOD datasets, as is also the case for othertraditional methods.
Revised: It is noteworthy that our method performs relatively poorly on the iNaturalists and Places365 compared to other OOD datasets, as is also the case for other traditional methods.

Original: It can be because the ImageNet includes semantically overlapping objects as theiNaturalists and Places365, as is shown in Appendix A.9, even though we have used the conceptuallydisjoint subsets of Textures and iNaturalists provided by the previous study [1] for model evaluation.
Revised: It can be because the ImageNet includes semantically overlapping objects as the iNaturalists and Places365, as is shown in Appendix A.9, even though we have used the conceptually disjoint subsets of Textures and iNaturalists provided by the previous study [1] for model evaluation.

Original: Ablation studies We conduct ablation experiments on CIFAR-100 using the WideResNet network.
Revised: Ablation studies We conduct ablation experiments on CIFAR-100 using the WideResNet network.

Original: Table 2 verifies the effectiveness of each component, i.e., density consistency regularization andcontrastive distribution regularization.
Revised: Table 2 verifies the effectiveness of each component, i.e., density consistency regularization and contrastive distribution regularization.

Original: We remark that the baseline model without regularizationcorresponds theoretically to the Energy model with no outlier exposure and temperature coefficientof T = 1 .
Revised: We remark that the baseline model without regularization [Table] [Table] corresponds theoretically to the Energy model with no outlier exposure and temperature coefficient of T = 1 .

Original: Table 3 verifies the robustness of the proposed method to hyperparameters, includingbatch size, regularization weight γ and constant r .
Revised: Table 3 verifies the robustness of the proposed method to hyperparameters, including batch size, regularization weight γ and constant r .

Original: The model performance is quite consistent whenincreasing or decreasing the default values by 100% and 50%, respectively.
Revised: The model performance is quite consistent when increasing or decreasing the default values by 100% and 50%, respectively.

Revised: More detailed results on different OOD datasets and more hyperparamter choices can be found in Appendix A.8.

Original: Please note that unlikemany previous approaches involving hyperparameter tuning [4, 8–10, 17,  32], our method uses fixed[Table] [Table] hyperparameters for OOD detection across different ID  datasets 
Revised: Please note that unlike many previous approaches involving hyperparameter tuning [4, 8, 10, 17, 9, 32], our method uses fixed hyperparameters for OOD detection across different ID datasets, OOD datasets and networks.

Original: [Table]

Original: OOD and

Original: Itfurther confirms the robustness of the proposed method.
Revised: It further confirms the robustness of the proposed method.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we propose a theoretical probabilistic framework for OOD detection, which derivesanalytical density regularization by investigating the inherent consistent and contrastive characteristicsof the ID sample density to  build an accurate sample density estimator.
Revised: In this paper, we propose a theoretical probabilistic framework for OOD detection, which derives analytical density regularization by investigating the inherent consistent and contrastive characteristics of the ID sample density to calibrate and build an accurate sample density estimator.

Original: Unlike many previous studies,our method neither relies on unproven assumptions about the latent feature distribution nor requiresadditional outlier exposure datasets to calibrate ID density estimates or tune hyperparameters.
Revised: Unlike many previous studies, our method neither relies on unproven assumptions about the latent feature distribution nor requires additional outlier exposure datasets to calibrate ID density estimates or tune hyperparameters.

Original: Asimple yet effective computational algorithm is also developed, which can be used for any pretrainedclassification network.
Revised: A simple yet effective computational algorithm is also developed, which can be used for any pretrained classification network.

Original: Extensive evaluations on benchmark datasets show that the proposed approachsignificantly outperforms existing state-of-the-art OOD detectors.
Revised: Extensive evaluations on benchmark datasets show that the proposed approach significantly outperforms existing state-of-the-art OOD detectors.

Original: Furthermore, our method evenachieves comparable or better performance than the approaches requiring external large-scale outlierexposure datasets.
Revised: Furthermore, our method even achieves comparable or better performance than the approaches requiring external large-scale outlier exposure datasets.

Original: A limitation of our approach is that the fixed hyperparameters may not be optimalfor a specific task, although it enables our fairly robust model to achieve state-of-the-art performancesacross benchmarks.
Revised: A limitation of our approach is that the fixed hyperparameters may not be optimal for a specific task, although it enables our fairly robust model to achieve state-of-the-art performances across benchmarks.

Original: We leave it for future work to investigate adaptively setting hyperparameterswith the ID dataset.
Revised: We leave it for future work to investigate adaptively setting hyperparameters with the ID dataset.

Original: We believe our accurate sample density estimator can also guide future studies inrelated machine learning fields, such as active learning, open-world classification, semi-supervisedlearning, etc. Checklist For all authors... (a) Do the main claims made in  the abstract andintroduction accurately reflect the paper’s
Revised: Additionally, since our approach requires fine-tuning, this can lead to the problem that there exists a discrepancy in classification probabilities between the original network and the fine-tuned network.

Original: contributions and scope?

Original: [Yes]

Original: (b) Did you describe the limitations of your work?

Original: [Yes] See Section Conclusion

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
Revised: This problem can be avoided using the original and the fine-tuned networks for the classification and OOD detection tasks, respectively.

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?

Original: [Yes]

Original: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how they were

Original: chosen)?

Original: [Yes] (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
Revised: However, it can double the inference time and computation load.

Original: [N/A]

Original: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., typeof GPUs, internal cluster, or cloud provider)? [Yes] If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: Finally, we believe our accurate sample density estimator can also guide future studies in related machine learning fields, such as active learning, open-world classification, semi-supervised learning, etc.

Original: [Yes]

Original: [Table]

Original: [Table]

Original: [Table]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] We provide code for the proposed method in the supplementary material. (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: Acknowledgments and Disclosure of Funding

Original: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?

Original: [N/A] If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable? [N/A]
Revised: This work is supported in part by National Key Research and Development Program of China (2021YFF1200800)

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board

Original: (IRB) approvals, if applicable?

Original: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?

Original: [N/A]