Original: Further Analysis of Outlier Detection with Deep Generative Models
Revised: Further Analysis of Outlier Detection with Deep Generative Models

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: The recent, counter-intuitive discovery that deep generative models (DGMs) canfrequently assign a higher likelihood to outliers has implications for both outlierdetection applications as well as our overall understanding of generative modeling.
Revised: The recent, counter-intuitive discovery that deep generative models (DGMs) can frequently assign a higher likelihood to outliers has implications for both outlier detection applications as well as our overall understanding of generative modeling.

Original: In this work, we present a possible explanation for this phenomenon, starting fromthe observation that a model’s typical set and high-density region may not conincide.
Revised: In this work, we present a possible explanation for this phenomenon, starting from the observation that a model’s typical set and high-density region may not conincide.

Original: From this vantage point we propose a novel outlier test, the empirical success ofwhich suggests that the failure of existing likelihood-based outlier tests does notnecessarily imply that the corresponding generative model is uncalibrated.
Revised: From this vantage point we propose a novel outlier test, the empirical success of which suggests that the failure of existing likelihood-based outlier tests does not necessarily imply that the corresponding generative model is uncalibrated.

Original: We alsoconduct additional experiments to help disentangle the impact of low-level textureversus high-level semantics in differentiating outliers.
Revised: We also conduct additional experiments to help disentangle the impact of low-level texture versus high-level semantics in differentiating outliers.

Original: In aggregate, these resultssuggest that modiﬁcations to the standard evaluation practices and benchmarkscommonly applied in the literature are needed
Revised: In aggregate, these results suggest that modiﬁcations to the standard evaluation practices and benchmarks commonly applied in the literature are needed.

Original:  Introduction
Revised:  Introduction

Original: Outlier detection is an important problem in machine learning and data science.
Revised: Outlier detection is an important problem in machine learning and data science.

Original: While it is natural toconsider applying density estimates from expressive deep generative models (DGMs) to detect outliers,recent work has shown that certain DGMs, such as variational autoencoders (VAEs [1]) or ﬂow-basedmodels [2], often assign similar or higher likelihood to natural images with signiﬁcantly differentsemantics than the inliers upon which the models were originally trained [3, 4].
Revised: While it is natural to consider applying density estimates from expressive deep generative models (DGMs) to detect outliers, recent work has shown that certain DGMs, such as variational autoencoders (VAEs [1]) or ﬂow-based models [2] often assign similar or higher likelihood to natural images with signiﬁcantly different semantics than the inliers upon which the models were originally trained [3, 4].

Original: For example, a modeltrained on CIFAR-10 may assign higher likelihood to SVHN images.
Revised: For example, a model trained on CIFAR-10 may assign higher likelihood to SVHN images.

Original: This observation seeminglypoints to the infeasibility of directly applying DGMs to outlier detection problems.
Revised: This observation seemingly points to the infeasibility of directly applying DGMs to outlier detection problems.

Original: Moreover, it alsocasts doubt on the corresponding DGMs: One may justiﬁably ask whether these models are actuallywell-calibrated to the true underlying inlier distribution, and whether they capture the high-levelsemantics of real-world image data as opposed to merely learning low-level image statistics [3].
Revised: Moreover, it also casts doubt on the corresponding DGMs: One may justiﬁably ask whether these models are actually well-calibrated to the true underlying inlier distribution and whether they capture the high-level semantics of real-world image data as opposed to merely learning low-level image statistics [3].

Original: Building on these concerns, various diagnostics have been deployed to evaluate the calibration ofnewly proposed DGMs [5–9], or applied when revisiting older modeling practices [10].
Revised: Building on these concerns, various diagnostics have been deployed to evaluate the calibration of newly proposed DGMs [5–9], or applied when revisiting older modeling practices [10].

Original: As we will review in Section 5, many contemporary attempts have been made to understand thisostensibly paradoxical observation.
Revised: As we will review in Section 5, many contemporary attempts have been made to understand this ostensibly paradoxical observation.

Original: Of particular interest is the argument from typicality .
Revised: Of particular interest is the argument from typicality .

Original: Samplesfrom a high-dimensional distribution will often fall on a typical set with high probability, but thetypical set itself does not necessarily have the highest probability density at any given point.
Revised: Samples from a high-dimensional distribution will often fall on a typical set with high probability, but the typical set itself does not necessarily have the highest probability density at any given point.

Original: Per thisline of reasoning, to determine if a test sample is an outlier, we should check if it falls on the typicalset of the inlier distribution rather than merely examining its likelihood under a given DGM.
Revised: Per this line of reasoning, to determine if a test sample is an outlier, we should check if it falls on the typical set of the inlier distribution rather than merely examining its likelihood under a given DGM.

Original: However,previous efforts to utilize similar ideas for outlier detection have not been consistently successful[3, 11].
Revised: However, previous efforts to utilize similar ideas for outlier detection have not been consistently successful [3, 11].

Original: Thus it is unclear whether the failure of the likelihood tests studied in [3] should be attributedto the discrepancy between typical sets and high-density regions or instead, the miscalibration ofthe corresponding DGMs.
Revised: Thus it is unclear whether the failure of the likelihood tests studied in [3] should be attributed to the discrepancy between typical sets and high-density regions or instead, the miscalibration of the corresponding DGMs.

Original: The situation is further complicated by the recent discovery that certainenergy-based models (EBMs) do actually assign lower likelihoods to these outliers [5, 6], even though we present experiments indicating that the probability density function (pdf) produced by these samemodels at out-of-distribution (OOD) locations can be inaccurate.
Revised: The situation is further complicated by the recent discovery that certain energy-based models (EBMs) do actually assign lower likelihoods to these outliers [5, 6], even though we present experiments indicating that the probability density function (pdf) produced by these same models at out-of-distribution (OOD) locations can be inaccurate.

Original: In this work we will attempt to at least partially disambiguate these unresolved ﬁndings.
Revised: In this work we will attempt to at least partially disambiguate these unresolved ﬁndings.

Original: To this end,
Revised: To this end,

Original: We ﬁrst present an outlier test generalizing the idea of the typical set test.
Revised: We ﬁrst present an outlier test generalizing the idea of the typical set test.

Original: Our test is based on theobservation that applying the typicality notion requires us to construct an independent and identicallydistributed (IID) sequence out of the inlier data, which may be too difﬁcult given ﬁnite samples andimperfect models.
Revised: Our test is based on the observation that applying the typicality notion requires us to construct an independent and identically distributed (IID) sequence out of the inlier data, which may be too difﬁcult given ﬁnite samples and imperfect models.

Original: For this reason, we turn to constructing sequences satisfying weaker criteria than
Revised: For this reason, we turn to constructing sequences satisfying weaker criteria than

Original: IID, and utilizing existing tests from the time series literature to check for these properties.
Revised: IID, and utilizing existing tests from the time series literature to check for these properties.

Original: Under theevaluation settings in previous efforts applying DGMs to outlier detection, our test is found to workwell, suggesting that the previously-observed failures of outlier tests based on the DGM likelihoodshould not be taken as unequivocal evidence of model miscalibration per se.
Revised: Under the evaluation settings in previous efforts applying DGMs to outlier detection, our test is found to work well, suggesting that the previously-observed failures of outlier tests based on the DGM likelihood should not be taken as unequivocal evidence of model miscalibration per se.

Original: We further support thisclaim by demonstrating that even the pdf from a simple multivariate Gaussian model can mimic thefailure modes of DGMs.
Revised: We further support this claim by demonstrating that even the pdf from a simple multivariate Gaussian model can mimic the failure modes of DGMs.

Original: Beyond these points, our experiments also reveal a non-trivial shortcoming of the existing outlierdetection benchmarks.
Revised: Beyond these points, our experiments also reveal a non-trivial shortcoming of the existing outlier detection benchmarks.

Original: Speciﬁcally, we demonstrate that under current setups, inlier and outlierdistributions can often be differentiated by a simple test using linear autocorrelation structures appliedin the original image space.
Revised: Speciﬁcally, we demonstrate that under current setups, inlier and outlier distributions can often be differentiated by a simple test using linear autocorrelation structures applied in the original image space.

Original: This implies that contrary to prior belief, these benchmarks do notnecessarily evaluate the ability of DGMs to capture semantic information in the data, and thusalternative experimental designs should be considered for this purpose.
Revised: This implies that contrary to prior belief, these benchmarks do not necessarily evaluate the ability of DGMs to capture semantic information in the data, and thus alternative experimental designs should be considered for this purpose.

Original: We present new benchmarksthat help to alleviate this problem.
Revised: We present new benchmarks that help to alleviate this problem.

Original: The rest of the paper is organized as follows:
Revised: The rest of the paper is organized as follows:

Original: In Section 2 we review the typicality argument andpresent our new outlier dectection test.
Revised: In Section 2 we review the typicality argument and present our new outlier dectection test.

Original: We then evaluate this test under a range of settings in Section3.
Revised: We then evaluate this test under a range of settings in Section 3.

Original: Next, Section 4 examines the difﬁculty of estimating pdfs at OOD locations.
Revised: Next, Section 4 examines the difﬁculty of estimating pdfs at OOD locations.

Original: And ﬁnally, wereview related work in Section 5 and present concluding discussions in Section 6.
Revised: And ﬁnally, we review related work in Section 5 and present concluding discussions in Section 6.

Original:  From Typicality to a White Noise Test 
Revised:  From Typicality to a White Noise Test 2.1 OOD Detection and the Typicality Argument

Original: 2.1 OOD Detection and the Typicality Argument

Original: It is well-known that model likelihood can potentially be inappropriate for outlier detection, especiallyin high dimensions.
Revised: It is well-known that model likelihood can potentially be inappropriate for outlier detection, especially in high dimensions.

Original: For example, suppose the inliers follow the d -dimensional standard Gaussiandistribution, p in ( x ) ∝ exp( −(cid:107)
Revised: For example, suppose the inliers follow the d -dimensional standard Gaussian distribution, p in ( x ) ∝ exp( −(cid:107)

Original: x (cid:107) 22 / 2) , and the test sample is the origin.
Revised: x (cid:107) 22 / 2) , and the test sample is the origin.

Original: By concentration inequalities,with overwhelming probability an inlier sample will fall onto an annulus with radius √ d (1 ± o (1)) ,the typical set, and thus the test sample could conceivably be classiﬁed as outlier.
Revised: By concentration inequalities, with overwhelming probability an inlier sample will fall onto an annulus with radius √ d (1 ± o (1)) , the typical set, and thus the test sample could conceivably be classiﬁed as outlier.

Original: Yet the (log) pdf ofthe test sample is higher than most inlier samples by O ( d ) .
Revised: Yet the (log) pdf of the test sample is higher than most inlier samples by O ( d ) .

Original: This indicates that the typical set doesnot necessarily coincide with regions of high density, and that to detect outliers we should considerchecking if the input falls into the former set.
Revised: This indicates that the typical set does not necessarily coincide with regions of high density, and that to detect outliers we should consider checking if the input falls into the former set.

Original: We refer to such a test as the typicality test .
Revised: We refer to such a test as the typicality test .

Revised: Remark 2.1.

Revised: The relevance of the typicality argument can be demonstrated through the following variant of the Gaussian example: suppose the inlier distribution is N ( µ, Σ) , where Σ has a fastdecaying eigenspectrum corresponding to latent factors of varying level of importance.

Revised: E.g., in a natural image distribution the leading eigenvectors may correspond to semantic variables, while the trailing eigenvectors may correspond to textural or background information.

Revised: In this case an outlier distribution purely representing background texture may have a covariance spanned by the least signiﬁcant eigenspaces of Σ , with similar eigenvalues; and similar to the above example, its samples usually have higher density when evaluated using p in .

Original: However, the typicality test is not directly applicable to general distributions, since it is difﬁcultto generalize the notion of typical set beyond simple cases such as component-wise independentdistributions, while maintaining a similar concentration property.
Revised: However, the typicality test is not directly applicable to general distributions, since it is difﬁcult to generalize the notion of typical set beyond simple cases such as component-wise independent distributions, while maintaining a similar concentration property.

Original: 1 One appealing proposal thatgeneralizes this idea is to ﬁt a deep latent variable model (LVM) on the inlier dataset using afactorized prior, so that we can transform the inlier distribution back to the prior and invoke thetypicality test in the latent space.
Revised: 2 One appealing proposal that generalizes this idea is to ﬁt a deep latent variable model (LVM) on the inlier dataset using a factorized prior, so that we can transform the inlier distribution back to the prior and invoke the typicality test in the latent space.

Original: This idea has been explored in [3], where the authors conclude thatit is not effective.
Revised: This idea has been explored in [3], where the authors conclude that it is not effective.

Original: One possible explanation is that for such a test to work, we must accurately identifythe LVM, which may be far more difﬁcult than generating visually plausible samples, requiring asigniﬁcantly larger sample size and/or better models.
Revised: One possible explanation is that for such a test to work, we must accurately identify the LVM, which may be far more difﬁcult than generating visually plausible samples, requiring a signiﬁcantly larger sample size and/or better models.

Original: Overall, the idea of typicality has not yet beensuccessfully applied to single-sample outlier detection for general inlier distributions.
Revised: Overall, the idea of typicality has not yet been successfully applied to single-sample outlier detection for general inlier distributions.

Original: 2.2 A White Noise Test for Outlier Detection
Revised: 2.2 A White Noise Test for Outlier Detection

Original: As we focus on the high-dimensional case, it is natural to take a longitudinal view of data, and interpreta d -dimensional random variable x as a sequence of d random variables.
Revised: As we focus on the high-dimensional case, it is natural to take a longitudinal view of data, and interpret a d -dimensional random variable x as a sequence of d random variables.

Original: From this perspective, the aforementioned LVM test essentially transforms x to another sequence T ( x ) , so that when x ∼ p in ,
Revised: From this perspective, the aforementioned LVM test essentially transforms x to another sequence T ( x ) , so that when x ∼ p in ,

Original: T ( x ) is IID.[-
-]
Revised: T ( x ) is IID.[-
-]

Original: Given a new sample x (cid:48) , the test evaluates whether T ( x (cid:48) ) is still IID by checking thevalue of (cid:80) di =1 T i ( x (cid:48) ) 2 .
Revised: Given a new sample x (cid:48) , the test evaluates whether T ( x (cid:48) ) is still IID by checking the value of (cid:80) di =1 T i ( x (cid:48) ) 2 .

Original: The statistical power of the test is supported by concentration properties.
Revised: The statistical power of the test is supported by concentration properties.

Original: Of course IID is a strong property characterizing the lack of any dependency structure in a sequence,and transforming a long sequence back to IID may be an unreasonable objective.
Revised: Of course IID is a strong property characterizing the lack of any dependency structure in a sequence, and transforming a long sequence back to IID may be an unreasonable objective.

Original: Thus it is natural toconsider alternative sequence mappings designed to achieve a weaker criteria, and then subsequentlytest for that criteria.
Revised: Thus it is natural to consider alternative sequence mappings designed to achieve a weaker criteria, and then subsequently test for that criteria.

Original: In the time series literature, there are two such weaker possibilities: the martingaledifference (MD) and white noise (WN).
Revised: In the time series literature, there are two such weaker possibilities: the martingale difference (MD) and white noise (WN).

Original: A sequence x is said to be a MD sequence if E ( x t | x <t ) = 0for all t ; x is said to be WN if for all s (cid:54) =
Revised: A sequence x is said to be a MD sequence if E ( x t | x <t ) =for all t ; x is said to be WN if for all s (cid:54) =

Original: t , Cov( x t , x s ) = 0 ,
Revised: t , Cov( x t , x s ) = 0 ,

Original: Var( x s )
Revised: Var( x s )

Original: = 1 .
Revised: = 1 .

Original: It is thus clear that forsequences with zero mean and unit variance, MD is a weaker property than IID, and WN is weakerthan MD.
Revised: It is thus clear that for sequences with zero mean and unit variance, MD is a weaker property than IID, and WN is weaker than MD.

Original: While IID sequences are automatically MD and WN, we can also construct WN or MD sequencesfrom inlier samples using residuals from autoregressive models per the following:
Revised: While IID sequences are automatically MD and WN, we can also construct WN or MD sequences from inlier samples using residuals from autoregressive models per the following:

Original: Claim 2.1.
Revised: Claim 2.1.

Original: Let ˜ R t ( x ) :
Revised: Let ˜ R t ( x ) :

Original: = x t − E p in ( x t | x <t ) and R ( x ) :=
Revised: = x t − E p in ( x t | x <t ) and R ( x ) :=

Original: ˜ R t ( x ) / (cid:113) Var p in ( ˜ R t ( x )) ; let W t ( x ) :
Revised: ˜ R t ( x ) / (cid:113) Var p in ( ˜ R t ( x )) ; let W t ( x ) :

Original: =x t − (cid:80) t − 1 s =1 a ts x s , where the lower triangular matrix A =
Revised: = x t − (cid:80) t − 1 s =1 a ts x s , where the lower triangular matrix A =

Original: ( a ts ) is the inverse of the Cholesky factorof Cov x ∼ p in ( x ) .
Revised: ( a ts ) is the inverse of the Cholesky factor of Cov x ∼ p in ( x ) .

Original: Assume Var p in ( ˜ R t ) > 0 for all t .
Revised: Assume Var p in ( ˜ R t ) > 0 for all t .

Original: Then when x ∼ p in , ˜ R ( x ) , R ( x ) are both MD,and R ( x ) , W ( x ) are both WN.
Revised: Then when x ∼ p in , ˜ R ( x ) , R ( x ) are both MD, and R ( x ) , W ( x ) are both WN.

Original: The ﬁrst claim above follows from deﬁnition.
Revised: The ﬁrst claim above follows from deﬁnition.

Original: For the second, R is WN because it is MD and has unitvariance.
Revised: For the second, R is WN because it is MD and has unit variance.

Original: Also, W is WN since Cov x ∼ p in [ W t ( x )]
Revised: Also, W is WN since Cov x ∼ p in [ W t ( x )]

Original: = I .
Revised: = I .

Original: The conditional expectations in R can be estimated with deep autoregressive models.
Revised: The conditional expectations in R can be estimated with deep autoregressive models.

Original: For conveniencewe choose to estimate them with existing autoregressive DGMs in literature (e.g. PixelCNN).
Revised: For convenience we choose to estimate them with existing autoregressive DGMs in literature (e.g. PixelCNN).

Original: However,even though we are ﬁtting generative models, we only need to estimate the mean of the autoregressivedistributions { p ( x t | x <t ) } accurately, as opposed to estimating the entire probability density function.
Revised: However, even though we are ﬁtting generative models, we only need to estimate the mean of the autoregressive distributions { p ( x t | x <t ) } accurately, as opposed to estimating the entire probability density function.

Original: For this reason, tests using R should be more robust against estimation errors than tests based onmodel likelihood.
Revised: For this reason, tests using R should be more robust against estimation errors than tests based on model likelihood.

Original: As testing for the MD property is difﬁcult, we choose to test the weaker WN property.
Revised: As testing for the MD property is difﬁcult, we choose to test the weaker WN property.

Original: This can beimplemented using the classical Box-Pierce test statistics [14]
Revised: This can be implemented using the classical Box-Pierce test statistics [15]

Original: [Equation]
Revised: [Equation]

Original: where ˆ ρ l is the l -lag autocorrelation estimate of a test sequence ( T t ( x ))
Revised: where ˆ ρ l is the l -lag autocorrelation estimate of a test sequence ( T t ( x ))

Original: dt =1 .
Revised: dt =1 .

Original: In practice, we can useeither W or R as the test sequence, which are both WN when constructed from inliers.
Revised: In practice, we can use either W or R as the test sequence, which are both WN when constructed from inliers.

Original: When ( T t )has zero mean and unit variance, we have ˆ ρ l = 1 d − l (cid:80) d − l t =1 T t T t + l .
Revised: When ( T t ) has zero mean and unit variance, we have ˆ ρ l = 1 d − l (cid:80) d − l t =1 T t T t + l .

Original: We consider a data point x testmore likely to be outlier when Q BP ( x test ) is larger.
Revised: We consider a data point x test more likely to be outlier when Q BP ( x test ) is larger.

Original: Under the context of hypothesis testing wherea binary decision (whether x test is an outlier) is needed, we can determine the threshold using thedistribution of Q BP evaluated on inlier data.
Revised: Under the context of hypothesis testing where a binary decision (whether x test is an outlier) is needed, we can determine the threshold using the distribution of Q BP evaluated on inlier data.

Original: In high dimensions, formally characterizing the power of a outlier test can be difﬁcult; as illustratedin Section 2.1, it is difﬁcult to even ﬁnd a proper deﬁnition of outlier that is simultaneously practical.
Revised: In high dimensions, formally characterizing the power of a outlier test can be difﬁcult; as illustrated in Section 2.1, it is difﬁcult to even ﬁnd a proper deﬁnition of outlier that is simultaneously practical.

Original: Nonetheless, the following remark provides some intuition on the power of our test, when the testsequence derived from outliers has non-zero autocorrelations.
Revised: Nonetheless, the following remark provides some intuition on the power of our test, when the test sequence derived from outliers has non-zero autocorrelations.

Original: This is a natural assumption forimage data, where the residual sequence from outlier data could contain more unexplained semanticinformation, which subsequently contributes to higher autocorrelation; see Appendix A for empiricalveriﬁcation and further discussion on this matter.
Revised: This is a natural assumption for image data, where the residual sequence from outlier data could contain more unexplained semantic information, which subsequently contributes to higher autocorrelation; see Appendix A for empirical veriﬁcation and further discussion on this matter.

Original: Remark 2.1 (Connection with the concentration-of-measure phenomenon) .
Revised: Remark 2.2 (Connection with the concentration-of-measure phenomenon) .

Original: The power of the BoxPierce test is supported by a concentration-of-measure phenomenon: When { T t ( x ) } is IID Gaussian,[-
-]
Revised: The power of the BoxPierce test is supported by a concentration-of-measure phenomenon: When { T t ( x ) } is IID Gaussian,[-
-]

Original: Q BP will approximately follow a χ 2 L distribution [14], and Q BP /L will concentrate around 1 .
Revised: Q BP will approximately follow a χ 2 L distribution [15], and Q BP /L will concentrate around 1 .

Original: Onthe other hand, if the null hypothesis does not hold and there exists a non-zero ρ l , Q BP /L will be atleast dρ 2 l /L , which is much larger than 1 when d is large.
Revised: On the other hand, if the null hypothesis does not hold and there exists a non-zero ρ l , Q BP /L will be at least dρ 2 l /L , which is much larger than 1 when d is large.

Original: Note that Remark 2.1 provides yet another connection to the typicality test, which is also based onconcentration.
Revised: It should be noted, however, that our test beneﬁts from the concentration phenomenon in a different way comparing to the typicality test. As an example, consider the following outlier distribution, which connects to the colored noise example in [16]: for x ∼ p ood , ( T 1 ( x ) , T 2 ( x )) follow the uniform distribution on the circle centered at origin with radius √ 2 , and T j ( x ) = T j − 2 ( x ) for j > 2 . Then 1 d (cid:80) dj

Original:  In fact, it is easy to verify that our test actually recovers the typicality test when theinlier distribution is component-wise independent.
Revised: =1 T 2 j ( x ) = 1 , and thus the typicality test cannot detect such outliers. In contrast, our test will always detect the lag-2 autocorrelation in T , and, as described above, reject the null hypothesis.

Original: 2.3 Implementation Details
Revised: 2.3 Implementation Details

Original: Incorporating prior knowledge for image data: When applied to image data, the power ofthe proposed test can be improved by incorporating prior knowledge about outlier distributions.
Revised: Incorporating prior knowledge for image data: When applied to image data, the power of the proposed test can be improved by incorporating prior knowledge about outlier distributions.

Original: Speciﬁcally, as the test sequence T ( x ) is obtained by stacking residuals of natural images, ρ l islikely small for the lags l that do not align with ﬁxed offsets along the two spatial dimensions.
Revised: Speciﬁcally, as the test sequence T ( x ) is obtained by stacking residuals of natural images, ρ l is likely small for the lags l that do not align with ﬁxed offsets along the two spatial dimensions.

Original: Asthe corresponding ﬁnite-sample estimates ˆ ρ l are noisy (approximately normal), they constitute asource of independent noise that has a similar scale in both inlier and outlier data, and removing themfrom (1) will increase the gap between the distributions of the test statistics computed from inlierand outlier data, consequently improving the power of our test.
Revised: As the corresponding ﬁnite-sample estimates ˆ ρ l are noisy (approximately normal), they constitute a source of independent noise that has a similar scale in both inlier and outlier data, and removing them from (1) will increase the gap between the distributions of the test statistics computed from inlier and outlier data, consequently improving the power of our test.

Original: For this reason, we modify (1) toonly include lags that correspond to vertical autocorrelations in images.
Revised: For this reason, we modify (1) to only include lags that correspond to vertical autocorrelations in images.

Original: When the data sequence isobtained by stacking an image with channel-last layout (i.e., for x 3( W ( i − 1)+ j )+ c refers to the c -thchannel of the ( i, j ) pixel of a H × W RGB image), we will only include lags that are multiples of3 W .
Revised: When the data sequence is obtained by stacking an image with channel-last layout (i.e., for x 3( W ( i − 1)+ j )+ c refers to the c -th channel of the ( i, j ) pixel of a H × W RGB image), we will only include lags that are multiples of 3 W .

Original: For empirical veriﬁcations and further discussion on this issue, see Appendix A.
Revised: For empirical veriﬁcations and further discussion on this issue, see Appendix A.

Original: Testing on transformed data: Instead of ﬁtting autoregressive models directly in the input space,we may also ﬁt them on some transformed domain, and use the resulting residual for the WNtest.
Revised: Testing on transformed data: Instead of ﬁtting autoregressive models directly in the input space, we may also ﬁt them on some transformed domain, and use the resulting residual for the WN test.

Original: Possible transformations include residuals from VAEs and lower-level latent variables fromhierarchical generative models (e.g. VQ-VAE).
Revised: Possible transformations include residuals from VAEs and lower-level latent variables from hierarchical generative models (e.g. VQ-VAE).

Original: 4 This can be particularly appealing for the test using( W t ) , as linear autoregressive models have limited capacity and cannot effectively remove nonlineardependencies from data, yet the lack of dependency seems important for the Box-Pierce test, assuggested by Remark 2.1.
Revised: 5 This can be particularly appealing for the test using ( W t ) , as linear autoregressive models have limited capacity and cannot effectively remove nonlinear dependencies from data, yet the lack of dependency seems important for the Box-Pierce test, as suggested by Remark 2.2.

Original:  Evaluating the White Noise Test
Revised:  Evaluating the White Noise Test

Original: In this section we evaluate the proposed test, with the goal of better understanding the previousﬁndings in [3].
Revised: In this section we evaluate the proposed test, with the goal of better understanding the previous ﬁndings in [3].

Original: We consider three implementations of our white noise test, which use differentsequences to compute the test statistics (1):
Revised: We consider three implementations of our white noise test, which use different sequences to compute the test statistics (1):

Original: • the residual sequence R , estimated with autoregressive DGMs (denoted as AR-DGM ); • the residual sequence W from a linear AR model, directly ﬁtted on the input space ( Linear ); • the sequence W constructed from a linear model ﬁtted on the space of VAE residuals( VAE+linear ).
Revised: • the residual sequence R , estimated with autoregressive DGMs (denoted as AR-DGM ); • the residual sequence W from a linear AR model, directly ﬁtted on the input space ( Linear ); • the sequence W constructed from a linear model ﬁtted on the space of VAE residuals ( VAE+linear ).

Original: Note that both R and W can be viewed as constructed from generative models: for the sequence W ,the corresponding model is a simple multivariate normal distribution.
Revised: Note that both R and W can be viewed as constructed from generative models: for the sequence W , the corresponding model is a simple multivariate normal distribution.

Original: Therefore, we can always gaininsights from comparing our test to other tests based on the corresponding generative model.
Revised: Therefore, we can always gain insights from comparing our test to other tests based on the corresponding generative model.

Original: 3.1 Evaluation on Standard Image Datasets
Revised: 3.1 Evaluation on Standard Image Datasets

Original: We ﬁrst evaluate our white noise test following the setup in [3], where the outlier data comes fromstandard image datasets, and can be different from inlier data in terms of both low-level details(textures, etc) as well as high-level semantics.
Revised: We ﬁrst evaluate our white noise test following the setup in [3], where the outlier data comes from standard image datasets, and can be different from inlier data in terms of both low-level details (textures, etc) as well as high-level semantics.

Original: In Appendix B we present additional experimentsunder a similar setup, in which we compare with more baselines.
Revised: In Appendix B we present additional experiments under a similar setup, in which we compare with more baselines.

Original: Evaluation Setup: We use CIFAR-10, CelebA, and TinyImageNet images as inliers, and CIFAR10, CelebA and SVHN images as outliers.
Revised: Evaluation Setup: We use CIFAR-10, CelebA, and TinyImageNet images as inliers, and CIFAR10, CelebA and SVHN images as outliers.

Original: All colored images are resized to 32 × 32 and centercropped when necessary.
Revised: All colored images are resized to 32 × 32 and center cropped when necessary.

Original: For deep autoregressive models, we choose PixelSNAIL [15] when theinlier dataset is TinyImageNet, and PixelCNN++ [16] otherwise.
Revised: For deep autoregressive models, we choose PixelSNAIL [17] when the inlier dataset is TinyImageNet, and PixelCNN++ [18] otherwise.

Original: We use the pretrained unconditionalmodels from the respective papers when possible; otherwise we train models using the setups fromthe paper.
Revised: We use the pretrained unconditional models from the respective papers when possible; otherwise we train models using the setups from the paper.

Original: For the VAE-based tests, we use an architecture similar to [17], and vary the latent dimension n z as it may have an inﬂuence on the likelihood-based outlier test.
Revised: For the VAE-based tests, we use an architecture similar to [19], and vary the latent dimension n z as it may have an inﬂuence on the likelihood-based outlier test.

Original: See Appendix C.1 formore details.
Revised: See Appendix C.1 for more details.

Original: We compare our test ( WN ) with three baselines that have been suggested for generative-model-basedoutlier detection: a single-sided likelihood test ( LH ), a two-sided likelihood test ( LH-2S ), and, forthe DGM-related tests, the likelihood-ratio test proposed in [18] ( LR ).
Revised: We compare our test ( WN ) with three baselines that have been suggested for generative-model-based outlier detection: a single-sided likelihood test ( LH ), a two-sided likelihood test ( LH-2S ), and, for the DGM-related tests, the likelihood-ratio test proposed in [20] ( LR ).

Original: The LH test classiﬁes sampleswith lower likelihood as outliers.
Revised: The LH test classiﬁes samples with lower likelihood as outliers.

Original: The LH-2S test classiﬁes samples with model likelihood deviatedfrom the inlier median as outliers.
Revised: The LH-2S test classiﬁes samples with model likelihood deviated from the inlier median as outliers.

Original: It can be viewed as testing if the input falls into the weakly typicalset [12]; 6 while there is no concentration guarantee in the case of general inlier distributions, it isnatural to include such a baseline.
Revised: It can be viewed as testing if the input falls into the weakly typical set [13]; 7 while there is no concentration guarantee in the case of general inlier distributions, it is natural to include such a baseline.

Original: The LR test is a competitive approach to single-sample OODdetection; it conducts a single-sided test using the statistics log p model ( x ) p generic ( x ) , where p generic refers tothe distribution corresponding to some generic image compressor (e.g., PNG).
Revised: The LR test is a competitive approach to single-sample OOD detection; it conducts a single-sided test using the statistics log p model ( x ) p generic ( x ) , where p generic refers to the distribution corresponding to some generic image compressor (e.g., PNG).

Original: Samples with a lowervalue of this statistics is considered outlier.
Revised: Samples with a lower value of this statistics is considered outlier.

Original: The test is based on the assumption that outlier sampleswith a higher model likelihood may have inherently lower complexity, as measured by log p generic .
Revised: The test is based on the assumption that outlier samples with a higher model likelihood may have inherently lower complexity, as measured by log p generic .

Original: The test statistics, having the form of a Bayes factor, and can also be viewed that comparing twocompeting hypotheses ( p model and p generic ) without assuming either is true [20].
Revised: The test statistics, having the form of a Bayes factor, and can also be viewed that comparing two competing hypotheses ( p model and p generic ) without assuming either is true [22]

Original: [Table]
Revised: [Table]

Original: Results and Discussion: We compare the distribution of the test statistics on the inlier test dataand outlier datasets

Original: [Table]
Revised: [Table]

Original:  and report the AUROC values.
Revised: Results and Discussion: We compare the distribution of the test statistics on the inlier test data and outlier datasets, and report the AUROC values.

Original: The results are shown in Table [Table] where weobserve that our WN proposal outperforms all the others in terms of the average ranking across testingconditions; see rightmost column.
Revised: The results are shown in Table 1, where we observe that our WN proposal outperforms all the others in terms of the average ranking across testing conditions; see rightmost column.

Original: (We have deferred to Appendix C.1 the results of likelihood-basedtests based on multivariate normal models ﬁtted on VAE residuals, as those tests did not work well.)
Revised: (We have deferred to Appendix C.1 the results of likelihood-based tests based on multivariate normal models ﬁtted on VAE residuals, as those tests did not work well.)

Original: Drilling further into details, we can see that our WN test generally outperforms the likelihood-basedtests, and the single-side likelihood test exhibits pathological behaviors.
Revised: Drilling further into details, we can see that our WN test generally outperforms the likelihood-based tests, and the single-side likelihood test exhibits pathological behaviors.

Original: This happens across allchoices of generative models, including the simple Gaussian model corresponding to the linear test.
Revised: This happens across all choices of generative models, including the simple Gaussian model corresponding to the linear test.

Original: Therefore, it is reasonable to doubt whether the previously observed failures of likelihood-based testsshould be attributed to some undesirable properties of DGMs.
Revised: Therefore, it is reasonable to doubt whether the previously observed failures of likelihood-based tests should be attributed to some undesirable properties of DGMs.

Original: Alternatively, those results may bebetter explained by the counter-intuitive properties of high-dimensional probability, as in Section 2.1.
Revised: Alternatively, those results may be better explained by the counter-intuitive properties of high-dimensional probability, as in Section 2.1.

Original: Furthermore, the fact that we can always construct a principled test statistics out of generative modelssuggests that these models have in some sense calibrated behavior on such outliers.
Revised: Furthermore, the fact that we can always construct a principled test statistics out of generative models suggests that these models have in some sense calibrated behavior on such outliers.

Original: In other words,under these settings the models do know what they don’t know.
Revised: In other words, under these settings the models do know what they don’t know.

Original: Our result is to be compared withthe recent discovery that EBMs assign lower likelihood to outliers under this setting [5, 6], whichnaturally leads to the question of whether a calibrated DGM should always have a similar behavior.
Revised: Our result is to be compared with the recent discovery that EBMs assign lower likelihood to outliers under this setting [5, 6], which naturally leads to the question of whether a calibrated DGM should always have a similar behavior.

Original: However, our ﬁndings are not necessarily inconsistent with theirs, as we explain in Section 4.
Revised: However, our ﬁndings are not necessarily inconsistent with theirs, as we explain in Section 4.

Original: Comparison between our test and the LR test is more nuanced, as the latter is also competitive inmany cases.
Revised: Comparison between our test and the LR test is more nuanced, as the latter is also competitive in many cases.

Original: Still, the LR test consistently produces a slightly higher average rank, and also has twocases of notable failures.
Revised: Still, the LR test consistently produces a slightly higher average rank, and also has two cases of notable failures.

Original: Finally, note that the simple linear generative model, especially when combined with the WN test,works well in most cases.
Revised: Finally, note that the simple linear generative model, especially when combined with the WN test, works well in most cases.

Original: This challenges the intuition that the inﬂexibility of a linear model wouldhamper outlier-detection performance, and has two-fold implications.
Revised: This challenges the intuition that the inﬂexibility of a linear model would hamper outlier-detection performance, and has two-fold implications.

Original: First, these results indicatethat the linear white-noise test could be useful in practice, as it is easy to implement, and does nothave unexpected failures like the likelihood tests.
Revised: First, these results indicate that the linear white-noise test could be useful in practice, as it is easy to implement, and does not have unexpected failures like the likelihood tests.

Original: Hence, it could be applied as a cheap, ﬁrst test in adetection pipeline.
Revised: Hence, it could be applied as a cheap, ﬁrst test in a detection pipeline.

Original: And secondly, the success of the linear test shows that the current benchmarksleave a lot to be desired, since it implies that the differences between the inlier and outlier distributionsbeing exploited for outlier detection are mostly low-level.
Revised: And secondly, the success of the linear test shows that the current benchmarks leave a lot to be desired, since it implies that the differences between the inlier and outlier distributions being exploited for outlier detection are mostly low-level.

Original: Consequently, it remains unclear if thesebenchmarks are adequate for showcasing tests that are sensitive to semantic differences.
Revised: Consequently, it remains unclear if these benchmarks are adequate for showcasing tests that are sensitive to semantic differences.

Original: Such asemantics-oriented evaluation is arguably more important for downstream applications.
Revised: Such a semantics-oriented evaluation is arguably more important for downstream applications.

Original: Moreover, itbetter reﬂects the ability of DGMs to learn high-level semantics from data, as was the intent of [3].
Revised: Moreover, it better reﬂects the ability of DGMs to learn high-level semantics from data, as was the intent of [3].

Original: To address this issue, in the following subsection we conduct additional experiments that are morefocused on semantics.
Revised: To address this issue, in the following subsection we conduct additional experiments that are more focused on semantics.

Original: 3.2 Semantics-Oriented Evaluation
Revised: 3.2 Semantics-Oriented Evaluation

Original: [Figure]

Original: In this section we evaluate the OOD tests in scenarios where the inlier and outlier distributions havedifferent semantics, but the inﬂuence from background or textual differences is minimized.
Revised: In this section we evaluate the OOD tests in scenarios where the inlier and outlier distributions have different semantics, but the inﬂuence from background or textual differences is minimized.

Original: Weconsider two setups:
Revised: We consider two setups

Original:  • CIFAR , in which we use CIFAR-10 images as inliers and a subset of CIFAR-100 as outliers.
Revised: : • CIFAR , in which we use CIFAR-10 images as inliers and a subset of CIFAR-100 as outliers.

Original: In this setup the inlier and outlier distributions have signiﬁcantly different semantics, as wehave removed from CIFAR-100 all classes that overlap with CIFAR-10, namely, non-insect creatures and vehicles.
Revised: In this setup the inlier and outlier distributions have signiﬁcantly different semantics, as we have removed from CIFAR-100 all classes that overlap with CIFAR-10, namely, non-insect creatures and vehicles.

Original: Furthermore, this setup also reduces textual differences contributed by inconsistent data collection processes; note that both CIFAR datasets have been createdfrom the 80 Million Tiny Images dataset [21].
Revised: Furthermore, this setup also reduces textual differences contributed by inconsistent data collection processes; note that both CIFAR datasets have been created from the 80 Million Tiny Images dataset [23].

Original: • Synthetic , in which we further reduce the background and textual differences between imageclasses by using synthesized images from BigGAN [22].
Revised: • Synthetic , in which we further reduce the background and textual differences between image classes by using synthesized images from BigGAN [24].

Original: The outliers are class-conditionalsamples corresponding to two semantically different ImageNet classes; the inlier distributionis obtained by interpolating between these two classes using the GAN model.
Revised: The outliers are class-conditional samples corresponding to two semantically different ImageNet classes; the inlier distribution is obtained by interpolating between these two classes using the GAN model.

Original: In this case,the semantic difference between inlier and outlier distributions is smaller, although in mostcases it is still noticeable, as shown in Figure 1.
Revised: In this case, the semantic difference between inlier and outlier distributions is smaller, although in most cases it is still noticeable.

Original: We construct three benchmarks under this setting.
Revised: We construct three benchmarks under this setting.

Original: Detailed settings andmore sample images are postponed to Appendix C.
Revised: Detailed settings and sample images are postponed to Appendix C.2.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: The results are summarized in Table 2, with full results for the synthetic experiments deferred to
Revised: The results are summarized in Table 2, with full results for the synthetic experiments deferred to

Original: Appendix C.2.
Revised: Appendix C.2.

Original: In the CIFAR setup, none of the tests that are based on the AR DGM or the vanilla
Revised: In the CIFAR setup, none of the tests that are based on the AR DGM or the vanilla

Original: Gaussian model works well, which is consistent with the common belief that these models cannot capture the high-level semantics.
Revised: Gaussian model works well, which is consistent with the common belief that these models cannot capture the high-level semantics.

Original: When using VAEs, the WN test works well.
Revised: When using VAEs, the WN test works well.

Original: This experimentreafﬁrms that DGMs such as VAEs are able to distinguish between distributions with signiﬁcantlydifferent semantics, even though they may assign similar likelihood to samples from both distributions.
Revised: This experiment reafﬁrms that DGMs such as VAEs are able to distinguish between distributions with signiﬁcantly different semantics, even though they may assign similar likelihood to samples from both distributions.

Original: However, as we move to the synthetic setup where the semantic difference is smaller but still evident,the outcome becomes quite different.
Revised: However, as we move to the synthetic setup where the semantic difference is smaller but still evident, the outcome becomes quite different.

Original: The LH test performs much better, and our test no longerconsistently outperforms the others.
Revised: The LH test performs much better, and our test no longer consistently outperforms the others.

Original: It is also interesting to note that the LR test does not work wellon the second synthetic setup (see Appendix C.2), and completely fails to distinguish between inliersand outliers when using an autoregressive DGM.
Revised: It is also interesting to note that the LR test does not work well on the second synthetic setup (see Appendix C.2), and completely fails to distinguish between inliers and outliers when using an autoregressive DGM.

Original: To understand this failure, we plot the distributionsof model likelihood and test statistics in Appendix C.2.
Revised: To understand this failure, we plot the distributions of model likelihood and test statistics in Appendix C.2.

Original: We can see that the outlier distribution has aslightly higher complexity as measured the generic image compressor, contrary to the assumption in[18] that the lower input complexity of outliers causes the failure of likelihood-based OOD test.
Revised: We can see that the outlier distribution has a slightly higher complexity as measured the generic image compressor, contrary to the assumption in [20] that the lower input complexity of outliers causes the failure of likelihood-based OOD test.

Original: The difference in outcome between these experiments and Section 3.1 demonstrates the difﬁcultyin developing a universally effective OOD test.
Revised: The difference in outcome between these experiments and Section 3.1 demonstrates the difﬁculty in developing a universally effective OOD test.

Original: It is thus possible that in the purely unsupervisedsetting we have investigated, OOD tests are best developed on a problem-dependent basis.
Revised: It is thus possible that in the purely unsupervised setting we have investigated, OOD tests are best developed on a problem-dependent basis.

Original: Comparedwith Section 3.1, we can also see that the previous evaluation setups do not adequately evaluate theability of each test to measure semantic differences.
Revised: Compared with Section 3.1, we can also see that the previous evaluation setups do not adequately evaluate the ability of each test to measure semantic differences.

Original: For this purpose, our approach may be moreappropriate.
Revised: For this purpose, our approach may be more appropriate.

Original:  On the Difﬁculty of Density Estimation in OOD Regions
Revised:  On the Difﬁculty of Density Estimation in OOD Regions

Original: While DGMs such as GANs, VAEs, autoregressive models, and ﬂow-based models tend to assignhigher likelihoods to certain OOD images, high-capacity energy-based models have been shown attimes to have the opposite behavior [5, 6].
Revised: While DGMs such as GANs, VAEs, autoregressive models, and ﬂow-based models tend to assign higher likelihoods to certain OOD images, high-capacity energy-based models have been shown at times to have the opposite behavior [5, 6].

Original: This observation naturally leads to the question of whethercalibrated generative models trained on natural image datasets should always assign lower likelihoodto such outliers.
Revised: This observation naturally leads to the question of whether calibrated generative models trained on natural image datasets should always assign lower likelihood to such outliers.

Original: In this section, we argue that such a question is unlikely to have a clear-cut answer,by showing that given the relatively small sample size of typical image datasets compared to the highdimensionality of data, density estimation on OOD regions is intrinsically difﬁcult, and even modelssuch as EBMs can make mistakes.
Revised: In this section, we argue that such a question is unlikely to have a clear-cut answer, by showing that given the relatively small sample size of typical image datasets compared to the high dimensionality of data, density estimation on OOD regions is intrinsically difﬁcult, and even models such as EBMs can make mistakes.

Original: Speciﬁcally, we train a PixelCNN++ and the high-capacity EBM in [5] on samples generated by a
Revised: Speciﬁcally, we train a PixelCNN++ and the high-capacity EBM in [5] on samples generated by a

Original: VAE.
Revised: VAE.

Original: Since by design we have access to (lower bounds of) the true log probability density of theinlier distribution, we can check if a test model’s density estimation in OOD regions is correct, simplyby comparing it to the ground truth.
Revised: Since by design we have access to (lower bounds of) the true log probability density of the inlier distribution, we can check if a test model’s density estimation in OOD regions is correct, simply by comparing it to the ground truth.

Original: Our ground truth VAE has the same architecture as in Section 3, with n z = 64 ; training is conductedon CIFAR-10.
Revised: Our ground truth VAE has the same architecture as in Section 3, with n z = 64 ; training is conducted on CIFAR-10.

Original: The DGMs to be tested are trained using 80000 samples from the VAE, under thesame setup as in the original papers.
Revised: The DGMs to be tested are trained using 80000 samples from the VAE, under the same setup as in the original papers.

Original: See Appendix C.3 for details.
Revised: See Appendix C.3 for details.

Original: We generate outliers by settinghalf of the latent code in the VAE to zero.
Revised: We generate outliers by setting half of the latent code in the VAE to zero.

Original: Such outliers are likely to have a higher density under theground truth model, per the reasoning from Section 2.1.
Revised: Such outliers are likely to have a higher density under the ground truth model, per the reasoning from Section 2.1.

Original: Therefore, a DGM that correctly estimatesthe ground-truth data pdf should also assign higher likelihood to them.
Revised: Therefore, a DGM that correctly estimates the ground-truth data pdf should also assign higher likelihood to them.

Original: The distributions of density estimates are shown in Figure 2.
Revised: The distributions of density estimates are shown in Figure 1.

Original: We can see that while both the EBM and PixelCNN++ models being tested assign a higher relative likelihood to the outliers (note that theabsolute likelihoods between different models are not comparable because of different scaling andoffset factors), the inlier and outlier density estimates from the EBM overlap signiﬁcantly (middlemodels [ of such differences can plot) as compared to analogous overlap within the ground-truth VAE (left plot).
Revised: We can see that while both the EBM and PixelCNN++ models being tested assign a higher relative likelihood to the outliers (note that the absolute likelihoods between different models are not comparable because of different scaling and offset factors), the inlier and outlier density estimates from the EBM overlap signiﬁcantly (middle plot) as compared to analogous overlap within the ground-truth VAE (left plot).

Original: Such behaviormay be attributed to the inductive bias of the EBM, which has a stronger inﬂuence than data on theestimated pdf in OOD regions given the relatively small sample size.
Revised: Such behavior may be attributed to the inductive bias of the EBM, which has a stronger inﬂuence than data on the estimated pdf in OOD regions given the relatively small sample size.

Original: While we conjecture that VAEs or deep AR models can exhibit similar failures due to a different typeof inductive bias, we cannot reverse the above experiment and train these models on EBM samples,as sampling from EBMs rely on ad hoc processes such as premature termination of MCMC chains[5, 6, 24].
Revised: While we conjecture that VAEs or deep AR models can exhibit similar failures due to a different type of inductive bias, we cannot reverse the above experiment and train these models on EBM samples, as sampling from EBMs rely on ad hoc processes such as premature termination of MCMC chains [5, 6, 26].

Original: Nonetheless, our experiment has demonstrated the intrinsic difﬁculty of density estimationin OOD regions under the ﬁnite-sample, high-dimensional setting.
Revised: Nonetheless, our experiment has demonstrated the intrinsic difﬁculty of density estimation in OOD regions under the ﬁnite-sample, high-dimensional setting.

Original: For this reason, it is difﬁcult todraw a deﬁnitive conclusion as to whether real-world outliers should be assigned higher likelihoods,and alternative explanations, such as the typicality argument in Section 2, deserve more attention.
Revised: For this reason, it is difﬁcult to draw a deﬁnitive conclusion as to whether real-world outliers should be assigned higher likelihoods, and alternative explanations, such as the typicality argument in Section 2, deserve more attention.

Original: The hardness of density estimation in OOD regions also suggests that OOD tests based on DGMlikelihood should be used with caution, as is also suggested by the results in Section 3.1.
Revised: The hardness of density estimation in OOD regions also suggests that OOD tests based on DGM likelihood should be used with caution, as is also suggested by the results in Section 3.1.

Original:  Related Work
Revised:  Related Work

Original: Several works have explored the use of DGMs in outlier detection under settings similar to [3], someof which also provided possible explanations to the ﬁndings in [3].
Revised: Several works have explored the use of DGMs in outlier detection under settings similar to [3], some of which also provided possible explanations to the ﬁndings in [3].

Original: For example, [11] presents aheuristic test using the Watanebe-Akaike Information Criterion; however, the efﬁcacy of this testremains poorly understood.
Revised: For example, [11] presents a heuristic test using the Watanebe-Akaike Information Criterion; however, the efﬁcacy of this test remains poorly understood.

Original: As another alternative, [25] proposes to compute the likelihood ratiobetween the inlier model and a background model, based on the intuition that background can be aconfounding factor in the likelihood test.
Revised: As another alternative, [12] proposes to compute the likelihood ratio between the inlier model and a background model, based on the intuition that background can be a confounding factor in the likelihood test.

Original: In Appendix B we present evaluations for the two tests,showing that they do not always work across all settings.
Revised: In Appendix B we present evaluations for the two tests, showing that they do not always work across all settings.

Original: In Section 3 we have introduced thework of [18], and demonstrated that its assumption does not always hold.
Revised: In Section 3 we have introduced the work of [20], and demonstrated that its assumption does not always hold.

Original: In summary then, to datethere has not been a comprehensive explanation of the peculiar behavior of generative models onsemantically different outliers, although previous works can be illuminating and practically useful incertain scenarios.
Revised: In summary then, to date there has not been a comprehensive explanation of the peculiar behavior of generative models on semantically different outliers, although previous works can be illuminating and practically useful in certain scenarios.

Original: For the general problem of high-dimensional outlier detection, methods have also been developedunder different settings.
Revised: For the general problem of high-dimensional outlier detection, methods have also been developed under different settings.

Original: For example, [19] proposes a typicality test assuming input contains a batchof IID samples, while [4] assumes a few outlier samples are available before testing.
Revised: For example, [21] proposes a typicality test assuming input contains a batch of IID samples, while [4] assumes a few outlier samples are available before testing.

Original: There is alsowork on outlier detection in supervised learning tasks, where auxiliary label information is available;see, e.g. [26–32].
Revised: There is also work on outlier detection in supervised learning tasks, where auxiliary label information is available; see, e.g. [27–33].

Original: Finally, it is worth mentioning the formulation of atypicality [33], as motivated by the possiblemismatch between the typical set and the high-density regions.
Revised: Finally, it is worth mentioning the formulation of atypicality [16], as motivated by the possible mismatch between the typical set and the high-density regions.

Original: The atypicality test considers a testsequence to be OOD when there exists an alternative model leading to a smaller description length[34].
Revised: The atypicality test considers a test sequence to be OOD when there exists an alternative model leading to a smaller description length [34].

Revised: Our test shares a similar spirit in the use of autocorrelation estimation, which can be viewed as ﬁtting a linear alternative model.

Original: However, their choice to estimate p ( x t | x <t ) for test data x  becomes problematic when x cannotbe viewed as a stationary process, or with a large hypothesis space such aswith DGMs.
Revised: However, our test works on the residual sequence from a ﬂexible generative model, which beneﬁts from information in the inlier distribution; in contrast, they opt to estimate p ( x t | x <t ) directly on a test point x . This becomes problematic when x cannot be viewed as a stationary process, and prevents the use of ﬂexible models such as DGMs.

Original:  Discussion
Revised:  Discussion

Original: The recent discovery that DGMs may assign higher likelihood to natural image outliers casts intodoubt the calibration of such models.
Revised: The recent discovery that DGMs may assign higher likelihood to natural image outliers casts into doubt the calibration of such models.

Original: In this work, we present a possible explanation based on an
Revised: In this work, we present a possible explanation based on an

Original: OOD test that generalizes the notion of typicality.
Revised: OOD test that generalizes the notion of typicality.

Original: In evaluations we have found that our test iseffective under the previously used benchmarks, and that such peculiar behaviors of model likelihoodare not restricted to DGMs.
Revised: In evaluations we have found that our test is effective under the previously used benchmarks, and that such peculiar behaviors of model likelihood are not restricted to DGMs.

Original: We have also demonstrated that certain DGMs cannot accurately estimatepdfs at OOD locations, even if at times they may correctly differentiate outliers.
Revised: We have also demonstrated that certain DGMs cannot accurately estimate pdfs at OOD locations, even if at times they may correctly differentiate outliers.

Original: These ﬁndingssuggest that it may be premature to judge the merits of a model by its (in)ability to assign lowerlikelihood to outliers.
Revised: These ﬁndings suggest that it may be premature to judge the merits of a model by its (in)ability to assign lower likelihood to outliers.

Original: Further investigation of the behavior of DGMs on outliers will undoubtedly continue to provideuseful insights.
Revised: Further investigation of the behavior of DGMs on outliers will undoubtedly continue to provide useful insights.

Original: However, our analyses suggest a change of practice in such investigations, suchas considering alternatives to simply the model likelihood as our proposed test has exempliﬁed.
Revised: However, our analyses suggest a change of practice in such investigations, such as considering alternatives to simply the model likelihood as our proposed test has exempliﬁed.

Original: Likewise, the observation that a simple linear test performs well under current evaluation settings alsosuggests that care should be taken in the design and diversity of benchmark datasets, e.g., inclusionof at least some cases where low-level textures cannot be exclusively relied on.
Revised: Likewise, the observation that a simple linear test performs well under current evaluation settings also suggests that care should be taken in the design and diversity of benchmark datasets, e.g., inclusion of at least some cases where low-level textures cannot be exclusively relied on.

Original: And ﬁnally, from the perspective of unsupervised outlier detection, our experiments also revealed theintrinsic difﬁculty in designing universally effective tests.
Revised: And ﬁnally, from the perspective of unsupervised outlier detection, our experiments also revealed the intrinsic difﬁculty in designing universally effective tests.

Original: It is thus possible that future OOD tests arebest developed on a problem-dependent basis, with prior knowledge of potential outlier distributionstaken into account.
Revised: It is thus possible that future OOD tests are best developed on a problem-dependent basis, with prior knowledge of potential outlier distributions taken into account.

Original: [25] provides an example of such practice.
Revised: [12] provides an example of such practice.

Original: Broader Impact

Original: This paper explores the nuances of applying DGMs to outlier detection, with the goal of understandingthe limitations of current approaches as well as practical workarounds.

Original: From the perspective offundamental research into existing machine learning and data mining techniques, we believe thatthis contribution realistically has little potential downside.

Original: Additionally, given the pernicious rolethat outliers play in numerous application domains, e.g., fraud, computer intrusion, etc., betterpreventative measures can certainly play a positive role.

Original: That being said, it is of course alwayspossible to envision scenarios whereby an outlier detection system could inadvertently introducebias that unfairly penalizes a marginalized group, e g., in processing loan applications.

Original: Even so, it isour hope that the analysis herein could more plausibly be applied to exposing and mitigating suchalgorithmic biases.

Original: AppendixA On the Assumptions and Efﬁcacy of the White Noise Test
Revised: Appendix A On the Assumptions and Efﬁcacy of the White Noise Test

Original: In this section we provide visualizations to better understand the statistical power of our test, and toverify the claims in Section 2.3.
Revised: In this section we provide visualizations to better understand the statistical power of our test, and to verify the claims in Section 2.3.

Original: We ﬁrst plot samples of the residual sequence R in Figure 3, 8 under varying choices of inlier andoutlier distributions.
Revised: We ﬁrst plot samples of the residual sequence R in Figure 2, 9 under varying choices of inlier and outlier distributions.

Original: We can see that R constructed from outlier images generally include a higherproportion of unexplained semantic information: comparing the CelebA residual in Fig.3(a) (secondcolumn) where the model is trained on CIFAR-10, to Fig.3(b)
Revised: We can see that R constructed from outlier images generally include a higher proportion of unexplained semantic information: comparing the CelebA residual in Fig.2(a)

Original:  (ﬁrst column) where CelebA is inlier,we can see that the facial structure in CelebA residual is more evident when the model is trainedon CIFAR-10.
Revised: (second column) where the model is trained on CIFAR-10, to Fig.2(b) (ﬁrst column) where CelebA is inlier, we can see that the facial structure in CelebA residual is more evident when the model is trained on CIFAR-10.

Original: Similarly, comparing the CIFAR-10 residual from both models, we can see that thestructure of the vehicle (e.g. front window and car frame) is more evident when the model is trainedon CelebA. As the residual sequences constructed from outliers tend to have more natural image-likestructures, they will also have stronger spatial autocorrelations, compared with residuals from inliersamples that should in principle be white noise.
Revised: Similarly, comparing the CIFAR-10 residual from both models, we can see that the structure of the vehicle (e.g. front window and car frame) is more evident when the model is trained on CelebA. As the residual sequences constructed from outliers tend to have more natural image-like structures, they will also have stronger spatial autocorrelations, compared with residuals from inlier samples that should in principle be white noise.

Original: Note that while the residual sequences constructed from inliers also contain unexplained semanticinformation, this is due to estimation error of the deep AR model, and should not happen should wehave access to the ground truth model, as we have shown in Section 2.2.
Revised: Note that while the residual sequences constructed from inliers also contain unexplained semantic information, this is due to estimation error of the deep AR model, and should not happen should we have access to the ground truth model, as we have shown in Section 2.2.

Original: Moreover, the estimationerror should have a small impact on the efﬁcacy of the white noise test, as it is very easy to learn thecorrect linear autocorrelation structure of the inlier distribution, and thus the deviation of R from WNis usually small, as we show in Figure 4 right.
Revised: Moreover, the estimation error should have a small impact on the efﬁcacy of the white noise test, as it is very easy to learn the correct linear autocorrelation structure of the inlier distribution, and thus the deviation of R from WN is usually small, as we show in Figure 3 right.

Original: [Figure]
Revised: [Figure]

Original: The left 3 columns are generated from a PixelCNN++ trained on CIFAR-10;

Original: the

Original: 3 columns corresponds to CelebA.

Original: We now turn to the veriﬁcation of our prior belief about the autocorrelation structure in T ( x test ) ,when x test comes from the outlier distribution.
Revised: We now turn to the veriﬁcation of our prior belief about the autocorrelation structure in T ( x test ) , when x test comes from the outlier distribution.

Original: Speciﬁcally, we plot the average ACFs on inlier andoutlier data in Figure 4.
Revised: Speciﬁcally, we plot the average ACFs on inlier and outlier data in Figure 3.

Original: We can see that the ACF estimates on outlier residuals peaks at lags thatare multiples of 96 , which corresponds to the vertical spatial autocorrelations in 32 × 32 × 3 images.
Revised: We can see that the ACF estimates on outlier residuals peaks at lags that are multiples of 96 , which corresponds to the vertical spatial autocorrelations in 32 × 32 × 3 images.

Original: Moreover, on inlier and outlier distributions, the ACF estimates at other lags have approximatelyequal variances.
Revised: Moreover, on inlier and outlier distributions, the ACF estimates at other lags have approximately equal variances.

Original: When aggregated, these estimates will constitute a noticeable source of noise whichreduces the gap between the distributions of inlier and outlier test statistics, and thus excluding themfrom the statistics will improve the power of the WN test.
Revised: When aggregated, these estimates will constitute a noticeable source of noise which reduces the gap between the distributions of inlier and outlier test statistics, and thus excluding them from the statistics will improve the power of the WN test.

Original: Finally, we remark that it is also possible use spatial correlations directly in the construction of teststatistics.
Revised: Finally, we remark that it is also possible use spatial correlations directly in the construction of test statistics.

Original: However, our main focus in this work is to understand previous ﬁndings in generativeoutlier detection (instead of improving the state-of-the-art of OOD tests), and our choice to includeonly the vertical spatial autocorrelations is good enough for this purpose.
Revised: However, our main focus in this work is to understand previous ﬁndings in generative outlier detection (instead of improving the state-of-the-art of OOD tests), and our choice to include only the vertical spatial autocorrelations is good enough for this purpose.

Original: B More Experiments on Standard Image Datasets
Revised: B More Experiments on Standard Image Datasets

Original: In this section we conduct additional experiments, and evaluate a variety of generative outlier detectionmethods under a common setting.
Revised: In this section we conduct additional experiments, and evaluate a variety of generative outlier detection methods under a common setting.

Original: As we will see, while several tests are in general more competitivethan others, no single test achieves the best performance across all settings.
Revised: As we will see, while several tests are in general more competitive than others, no single test achieves the best performance across all settings.

Original: This experimentstrengthens our argument in the main text that unsupervised OOD tests should be developed on aproblem-dependent basis.
Revised: This experiment strengthens our argument in the main text that unsupervised OOD tests should be developed on a problem-dependent basis.

Original: Evaluation Setup: We use CIFAR-10 as inlier data.
Revised: Evaluation Setup: We use CIFAR-10 as inlier data.

Original: For outliers we consider two setups.
Revised: For outliers we consider two setups.

Original: Theﬁrst setup is taken from [18], and consists of 9 generic image datasets and 2 synthetic datasets,const and random; see Appendix A in [18] for details.
Revised: The ﬁrst setup is taken from [20], and consists of 9 generic image datasets and 2 synthetic datasets, const and random; see Appendix A in [20] for details.

Original: The second setup controls for low-leveldifferences by using the CIFAR-100 subset constructed in Section 3.2.
Revised: The second setup controls for low-level differences by using the CIFAR-100 subset constructed in Section 3.2.

Original: The tests to be evaluatedinclude those considered in Section 3.1, as well as the WAIC test [11] and the background likelihoodratio (BLR) test [25].
Revised: The tests to be evaluated include those considered in Section 3.1, as well as the WAIC test [11] and the background likelihood ratio (BLR) test [12].

Original: We base these tests on two DGMs: the VAE-512 model used in Section 3.1, anda smaller-capacity PixelCNN++ model as in [25].
Revised: We base these tests on two DGMs: the VAE-512 model used in Section 3.1, and a smaller-capacity PixelCNN++ model as in [12].

Original: For the BLR test, a noise level of the backgroundmodel needs to be determined.
Revised: For the BLR test, a noise level of the background model needs to be determined.

Original: Following the recommendations of the authors, we search for theoptimal parameter in the range of { 0 . 1 , 0 . 2 , 0 . 3 } using the grayscaled CIFAR-10 dataset as outlier.
Revised: Following the recommendations of the authors, we search for the optimal parameter in the range of { 0 . 1 , 0 . 2 , 0 . 3 } using the grayscaled CIFAR-10 dataset as outlier.

Original: We found the optimal noise level to be 0 . 1 , which is consistent with [25].
Revised: We found the optimal noise level to be 0 . 1 , which is consistent with [12].

Original: Results: Results are shown in Table 3-4.
Revised: Results: Results are shown in Table 3-4.

Original: When using VAEs, neither of the newly added baselinesare very competitive, suggesting that these methods are more prone to model misspeciﬁcation.
Revised: When using VAEs, neither of the newly added baselines are very competitive, suggesting that these methods are more prone to model misspeciﬁcation.

Original: Notably, the WAIC test does not work with SVHN as outlier.
Revised: Notably, the WAIC test does not work with SVHN as outlier.

Original: This is also observed in [25, 19] usingdifferent generative models (autoregressive and ﬂow-based models, respectively).
Revised: This is also observed in [12, 21] using different generative models (autoregressive and ﬂow-based models, respectively).

Original: For this reason wedrop it in the PixelCNN++ experiment.
Revised: For this reason we drop it in the PixelCNN++ experiment.

Original: When we switch to PixelCNN++, the BLR test performs much better under the setting of [18].
Revised: When we switch to PixelCNN++, the BLR test performs much better under the setting of [20].

Original: However, in either case it does not work well with the subset-of-CIFAR-100 dataset, despite thedataset’s clear semantic difference from the inlier dataset.
Revised: However, in either case it does not work well with the subset-of-CIFAR-100 dataset, despite the dataset’s clear semantic difference from the inlier dataset.

Original: Such results are not surprising since thedifference in background or low-level details is much smaller for CIFAR-100 compared with theother datasets, as we have discussed in Section 3.2.
Revised: Such results are not surprising since the difference in background or low-level details is much smaller for CIFAR-100 compared with the other datasets, as we have discussed in Section 3.2.

Original: Again, the difference in outcome between thetwo different settings demonstrates the difﬁculty of constructing universally effective OOD tests inthe unsupervised setup.
Revised: Again, the difference in outcome between the two different settings demonstrates the difﬁculty of constructing universally effective OOD tests in the unsupervised setup.

Original: C Experiment Details and Additional Results  
Revised: C Experiment Details and Additional Results C.1 Details for Section 3.

Original: C.1 Details for Section 3.

Original: Experiment Setup: For the AR-DGM experiments, we use the pretrained unconditional modelsfrom ofﬁcial repositories for CIFAR-10 and TinyImageNet.
Revised: Experiment Setup: For the AR-DGM experiments, we use the pretrained unconditional models from ofﬁcial repositories for CIFAR-10 and TinyImageNet.

Original: For CelebA we train a PixelCNN++model using the authors’ setup for unconditional CIFAR-10 generation.
Revised: For CelebA we train a PixelCNN++ model using the authors’ setup for unconditional CIFAR-10 generation.

Original: Both PixelCNN++ and
Revised: Both PixelCNN++ and

Original: [Table] [Table]
Revised: [Table] [Table]

Original: PixelSNAIL use the discretized mixture-of-logistics (DMOL) likelihood parameteriation.
Revised: PixelSNAIL use the discretized mixture-of-logistics (DMOL) likelihood parameteriation.

Original: To calculateits expectation, we ﬁrst calculate the expectation of the continuous mixture of logistics distribution,and then clip the result to the range of [0 , 1] .
Revised: To calculate its expectation, we ﬁrst calculate the expectation of the continuous mixture of logistics distribution, and then clip the result to the range of [0 , 1] .

Original: This is needed because the deﬁnition of the DMOLlikelihood include a similar truncation [16]: extra probability mass for the interval (1 , + ∞ ) (or( −∞ , 0) ) are assigned to the discretization bin [1 − 1 / 256 , 1] (or [0 , 1 / 256] , respectively), so that thedistribution is always supported on [0 , 1] .
Revised: This is needed because the deﬁnition of the DMOL likelihood include a similar truncation [18]: extra probability mass for the interval (1 , + ∞ ) (or ( −∞ , 0) ) are assigned to the discretization bin [1 − 1 / 256 , 1] (or [0 , 1 / 256] , respectively), so that the distribution is always supported on [0 , 1] .

Original: For the VAE experiments, we use the discretized logistics likelihood as the observation model.
Revised: For the VAE experiments, we use the discretized logistics likelihood as the observation model.

Original: Thenetwork architecture is adapted from [17]; we vary the capacity of the model by increasing thenumber of ﬁlters in convolutional layers by k times, where k may be in { 1 , 2 , 4 , 8 } .
Revised: The network architecture is adapted from [19]; we vary the capacity of the model by increasing the number of ﬁlters in convolutional layers by k times, where k may be in { 1 , 2 , 4 , 8 } .

Original: We train forat most 8 × 10 5 iterations using a learning rate of 10 − 4 , and perform early stopping based on thevalidation ELBO.
Revised: We train for at most 8 × 10 5 iterations using a learning rate of 10 − 4 , and perform early stopping based on the validation ELBO.

Original: We choose k to maximize validation ELBO.
Revised: We choose k to maximize validation ELBO.

Original: This leads to k = 1 for CIFAR-10, 4for CelebA and 8 for TinyImageNet.
Revised: This leads to k = 1 for CIFAR-10,for CelebA and 8 for TinyImageNet.

Original: This step is needed, because when k is further increased, thereconstruction error will start to have different distributions between training and held-out set.
Revised: This step is needed, because when k is further increased, the reconstruction error will start to have different distributions between training and held-out set.

Original: Sucha difference would be undesirable for all tests, as they will start to ﬁnd false differences betweenthe inlier training set and the test set.
Revised: Such a difference would be undesirable for all tests, as they will start to ﬁnd false differences between the inlier training set and the test set.

Original: Note that this difference is not due to overﬁtting, as we haveperformed early stopping based on validation ELBO; instead, it is simply due to the fact that themodel is exposed to training samples and not validation samples, and the gap appears very early intraining.
Revised: Note that this difference is not due to overﬁtting, as we have performed early stopping based on validation ELBO; instead, it is simply due to the fact that the model is exposed to training samples and not validation samples, and the gap appears very early in training.

Original: We use ELBO to approximate model likelihood in likelihood-related tests.
Revised: We use ELBO to approximate model likelihood in likelihood-related tests.

Original: The discrepancybetween ELBO and true model likelihood is likely to have little impact on test performance, since wehave also experimented with IWAE 100 which led to very similar results.
Revised: The discrepancy between ELBO and true model likelihood is likely to have little impact on test performance, since we have also experimented with IWAE 100 which led to very similar results.

Original: We compare the distributions of the test statistics evaluated on the inlier test set and outlier test set, andreport the AUROC value.
Revised: We compare the distributions of the test statistics evaluated on the inlier test set and outlier test set, and report the AUROC value.

Original: We veriﬁed that the four tests used in this section do not falsely distinguishbetween inlier training samples and test samples: the AUROC value for such a comparison is alwaysin the range of (0 . 42 , 0 . 53) .
Revised: We veriﬁed that the four tests used in this section do not falsely distinguish between inlier training samples and test samples: the AUROC value for such a comparison is always in the range of (0 . 42 , 0 . 53) .

Original: For outlier datasets with more than 50000 test samples, we sub-sample50000 images for evaluation.
Revised: For outlier datasets with more than 50000 test samples, we sub-sample 50000 images for evaluation.

Original: Using the formula in [35], we can thus show that the maximum possible95% conﬁdence interval for the AUROC values is ± 0 . 011 .
Revised: Using the formula in [35], we can thus show that the maximum possible 95% conﬁdence interval for the AUROC values is ± 0 . 011 .

Original: For a description of the four datasets usedin this section, please refer to, e.g., Table 3 in [18].
Revised: For a description of the four datasets used in this section, please refer to, e.g., Table 3 in [20].

Original: Choice of L and Sensitivity: For our test, we use L = 1200 when computing the Box-Piercestatistics (1).
Revised: Choice of L and Sensitivity: For our test, we use L = 1200 when computing the Box-Pierce statistics (1).

Original: This is because while in principle we should include all lags that are known a priori tobe informative, in practice we only have d − l samples to estimate ˆ ρ l , so the most distant lags canbe difﬁcult to estimate.
Revised: This is because while in principle we should include all lags that are known a priori to be informative, in practice we only have d − l samples to estimate ˆ ρ l , so the most distant lags can be difﬁcult to estimate.

Original: Nonetheless, the impact of L on the test outcome is relatively small: as isshown in Figure 5, using different L does not lead to qualitatively different outcome.
Revised: Nonetheless, the impact of L on the test outcome is relatively small: as is shown in Figure 4, using different L does not lead to qualitatively different outcome.

Original: We also notethat our purpose in the experiments is not to build new state-of-the-art in OOD detection, but is to usethe proposed test to validate our explanation to previous ﬁndings.
Revised: We also note that our purpose in the experiments is not to build new state-of-the-art in OOD detection, but is to use the proposed test to validate our explanation to previous ﬁndings.

Original: Still, if it is desirable to furtherimprove the performance of the test, we can consider tuning L on “validation outlier datasets” that isknown a priori to be similar to the outliers that will be encountered in practice, as is done in e.g. [25]. Inlier is CIFAR-10.
Revised: Still, if it is desirable to further improve the performance of the test, we can consider tuning L on “validation outlier datasets” that is known a priori to be similar to the outliers that will be encountered in practice, as is done in e.g. [12].

Original: Results for the Normal Likelihood Test on VAE Residuals:
Revised: Results for the Normal Likelihood Test on VAE Residuals:

Original: In Table 5 we present results for thelikelihood tests using a multivariate normal model ﬁtted on VAE residual, denoted with a preﬁx of“LN”.
Revised: In Table 5 we present results for the likelihood tests using a multivariate normal model ﬁtted on VAE residual, denoted with a preﬁx of “LN”.

Original: We also consider both single-side and two-side tests.
Revised: We also consider both single-side and two-side tests.

Original: Overall the performance is similar to
Revised: Overall the performance is similar to

Original: DGM likelihood, and the single-side likelihood test still manifests catastrophic failures.
Revised: DGM likelihood, and the single-side likelihood test still manifests catastrophic failures.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]

Original: C.2 Details for Section [-3.
-] 
Revised: C.2 Details for Section [-3.
-]3.

Original: The CIFAR Experiment: We use the trained models from Section 3.1.
Revised: The CIFAR Experiment: We use the trained models from Section 3.1.

Original: We remove from CIFAR100 the superclasses 1,2,9,12-17,19,20.
Revised: We remove from CIFAR100 the superclasses 1,2,9,12-17,19,20.

Original: For reference, the class names of CIFAR-10 and CIFAR-100can be found in https://www.cs.toronto.edu/~kriz/cifar.html .
Revised: For reference, the class names of CIFAR-10 and CIFARcan be found in https://www.cs.toronto.edu/~kriz/cifar.html .

Original: The Synthetic Experiments: We use a pretrained BigGAN model on ImageNet 128 × 128 , 10and down-sample the generated images to 32 × 32 .
Revised: The Synthetic Experiments: We use a pretrained BigGAN model on ImageNet 128 × 128 ,and down-sample the generated images to 32 × 32 .

Original: To generate the outliers, recall the BigGANgenerator takes as input a noise vector z ∈ R 128 and the one-hot class encoding vector c ∈ R 1000 .
Revised: To generate the outliers, recall the BigGAN generator takes as input a noise vector z ∈ R 128 and the one-hot class encoding vector c ∈ R 1000 .

Original: Therefore, we interpolate between two classes i and j by setting c k = 0 .
Revised: Therefore, we interpolate between two classes i and j by setting c k = 0 .

Original: 5 · 1 k ∈{ i,j } .
Revised: 5 · 1 k ∈{ i,j } .

Original: There aretwo tunable parameters in our generation process: the truncation parameter σ that determines thetruncated normal prior, and a crop parameter τ .
Revised: There are two tunable parameters in our generation process: the truncation parameter σ that determines the truncated normal prior, and a crop parameter τ .

Original: Before down-sampling the generated samples, weapply center-cropping to retain a proportion of (1 − 2 τ ) 2 pixels, to reduce the amount of details lostin the down-sampling process.
Revised: Before down-sampling the generated samples, we apply center-cropping to retain a proportion of (1 − 2 τ ) 2 pixels, to reduce the amount of details lost in the down-sampling process.

Original: The classes and generation parameters used are listed in Table 6; theyare hand-picked to ensure the background is similar in inlier and outlier classes.
Revised: The classes and generation parameters used are listed in Table 6; they are hand-picked to ensure the background is similar in inlier and outlier classes.

Original: In each setup wegenerate 200000 samples and use 80% for training.
Revised: In each setup we generate 200000 samples and use 80% for training.

Original: The VAEs are trained using the same setting as in Section 3.1.
Revised: The VAEs are trained using the same setting as in Section 3.1.

Original: For PixelCNN++ we use thehyperparameters of the unconditional CIFAR-10 experiment in the original paper.
Revised: For PixelCNN++ we use the hyperparameters of the unconditional CIFAR-10 experiment in the original paper.

Original: As the syntheticdatasets contain more samples, we train for 80 epochs.
Revised: As the synthetic datasets contain more samples, we train for 80 epochs.

Original: The full AUROC values for the synthetic experiments are shown in Table 7.
Revised: The full AUROC values for the synthetic experiments are shown in Table 7.

Original: We plot the distributionsof various statistics related to the LR tests using AR-DGM in the second synthetic experiment in
Revised: We plot the distributions of various statistics related to the LR tests using AR-DGM in the second synthetic experiment in

Original: Figure 6.
Revised: Figure 5.

Original: We also plot additional inlier and outlier samples in Figure 7.
Revised: We also plot additional inlier and outlier samples in Figure 6.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: C.3 Details for Section4
Revised: C.3 Details for Section

Original: The ground truth VAE has the same architecture as in Section 3.1, but with a continuous normallikelihood.
Revised: The ground truth VAE has the same architecture as in Section 3.1, but with a continuous normal likelihood.

Original: We use n z = 64 .
Revised: We use n z = 64 .

Original: The VAE (log) likelihood is lower bounded by IWAE 200 .
Revised: The VAE (log) likelihood is lower bounded by IWAE 200 .

Original: For EBMand PixelCNN++, we use the authors’ hyperparameters and training setup for the unconditional
Revised: For EBM and PixelCNN++, we use the authors’ hyperparameters and training setup for the unconditional

Original: CIFAR-10 experiments.
Revised: CIFAR-10 experiments.

Original: After training, we veriﬁed that the distributions of energy values of trainingand held-out samples have small differences, so the models do not appear to overﬁt.
Revised: After training, we veriﬁed that the distributions of energy values of training and held-out samples have small differences, so the models do not appear to overﬁt.

Original: As the OOD test results in [5, 6] are obtained with conditional models, we perform the single-sidedlikelihood test with the unconditional model (trained on the real CIFAR-10 dataset) to check if itsbehavior on the SVHN dataset is similar to the conditional model.
Revised: As the OOD test results in [5, 6] are obtained with conditional models, we perform the single-sided likelihood test with the unconditional model (trained on the real CIFAR-10 dataset) to check if its behavior on the SVHN dataset is similar to the conditional model.

Original: The AUROC value from thesingle-side likelihood test is 0.529, meaning that the EBM assigns similar or lower likelihood to
Revised: The AUROC value from the single-side likelihood test is 0.529, meaning that the EBM assigns similar or lower likelihood to

Original: [Figure]
Revised: [Figure]

Original: SVHN compared with the inliers.
Revised: SVHN compared with the inliers.

Original: This is still signiﬁcantly different from the results using othergenerative models, justifying our use of an unconditional model.
Revised: This is still signiﬁcantly different from the results using other generative models, justifying our use of an unconditional model.