Original: I NPUT C OMPLEXITY AND O UT - OF - DISTRIBUTION D ETECTION WITH L IKELIHOOD - BASED G ENERATIVE M ODELS
Revised: I NPUT C OMPLEXITY AND O UT - OF - DISTRIBUTION D ETECTION WITH L IKELIHOOD - BASED G ENERATIVE M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Likelihood-based generative models are a promising resource to detect out-ofdistribution (OOD) inputs which could compromise the robustness or reliability of a machine learning system.
Revised: Likelihood-based generative models are a promising resource to detect out-ofdistribution (OOD) inputs which could compromise the robustness or reliability of a machine learning system.

Original: However, likelihoods derived from such models have been shown to be problematic for detecting certain types of inputs that signiﬁcantly differ from training data.
Revised: However, likelihoods derived from such models have been shown to be problematic for detecting certain types of inputs that signiﬁcantly differ from training data.

Original: In this paper, we pose that this problem is due to the excessive inﬂuence that input complexity has in generative models’ likelihoods.
Revised: In this paper, we pose that this problem is due to the excessive inﬂuence that input complexity has in generative models’ likelihoods.

Original: We report a set of experiments supporting this hypothesis, and use an estimate of input complexity to derive an efﬁcient and parameter-free OOD score, which can be seen as a likelihood-ratio, akin to Bayesian model comparison.
Revised: We report a set of experiments supporting this hypothesis, and use an estimate of input complexity to derive an efﬁcient and parameter-free OOD score, which can be seen as a likelihood-ratio, akin to Bayesian model comparison.

Original: We ﬁnd such score to perform comparably to, or even better than, existing
Revised: We ﬁnd such score to perform comparably to, or even better than, existing

Original: OOD detection approaches under a wide range of data sets, models,  and complexity estimates.
Revised: OOD detection approaches under a wide range of data sets, models, model sizes, and complexity estimates.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Assessing whether input data is novel or signiﬁcantly different than the one used in training is critical for real-world machine learning applications.
Revised: Assessing whether input data is novel or signiﬁcantly different than the one used in training is critical for real-world machine learning applications.

Original: Such data are known as out-of-distribution (OOD) inputs, and detecting them should facilitate safe and reliable model operation.
Revised: Such data are known as out-of-distribution (OOD) inputs, and detecting them should facilitate safe and reliable model operation.

Original: This is particularly necessary for deep neural network classiﬁers, which can be easily fooled by OOD data (Nguyen et al., 2015).
Revised: This is particularly necessary for deep neural network classiﬁers, which can be easily fooled by OOD data (Nguyen et al., 2015).

Original: Several approaches have been proposed for OOD detection on top of or within a neural network classiﬁer (Hendrycks & Gimpel, 2017; Lakshminarayanan et al., 2017; Liang et al., 2018;
Revised: Several approaches have been proposed for OOD detection on top of or within a neural network classiﬁer (Hendrycks & Gimpel, 2017; Lakshminarayanan et al., 2017; Liang et al., 2018;

Original: Lee et al., 2018).
Revised: Lee et al., 2018).

Original: Nonetheless, OOD detection is not limited to classiﬁcation tasks nor to labeled data sets.
Revised: Nonetheless, OOD detection is not limited to classiﬁcation tasks nor to labeled data sets.

Original: Two examples of that are novelty detection from an unlabeled data set and next-frame prediction from video sequences.
Revised: Two examples of that are novelty detection from an unlabeled data set and next-frame prediction from video sequences.

Original: A rather obvious strategy to perform OOD detection in the absence of labels (and even in the presence of them) is to learn a density model M that approximates the true distribution p ∗ ( X ) of training inputs x ∈ X (Bishop, 1994).
Revised: A rather obvious strategy to perform OOD detection in the absence of labels (and even in the presence of them) is to learn a density model M that approximates the true distribution p ∗ ( X ) of training inputs x ∈ X (Bishop, 1994).

Original: Then, if such approximation is good enough, that is, p ( x |M )
Revised: Then, if such approximation is good enough, that is, p ( x |M )

Original: ≈ p ∗ ( x ) ,
Revised: ≈ p ∗ ( x ) ,

Original: OOD inputs should yield a low likelihood under model M .
Revised: OOD inputs should yield a low likelihood under model M .

Original: With complex data like audio or images, this strategy was long thought to be unattainable due to the difﬁculty of learning a sufﬁciently good model.
Revised: With complex data like audio or images, this strategy was long thought to be unattainable due to the difﬁculty of learning a sufﬁciently good model.

Original: However, with current approaches, we start having generative models that are able to learn good approximations of the density conveyed by those complex data.
Revised: However, with current approaches, we start having generative models that are able to learn good approximations of the density conveyed by those complex data.

Original: Autoregressive and invertible models such as PixelCNN++ (Salimans et al., 2017) and Glow (Kingma & Dhariwal, 2018) perform well in this regard and, in addition, can approximate p ( x |M ) with arbitrary accuracy.
Revised: Autoregressive and invertible models such as PixelCNN++ (Salimans et al., 2017) and Glow (Kingma & Dhariwal, 2018) perform well in this regard and, in addition, can approximate p ( x |M ) with arbitrary accuracy.

Original: Recent works, however, have shown that likelihoods derived from generative models fail to distinguish between training data and some OOD input types (Choi et al., 2018;
Revised: Recent works, however, have shown that likelihoods derived from generative models fail to distinguish between training data and some OOD input types (Choi et al., 2018;

Original: Nalisnick et al., 2019a;
Revised: Nalisnick et al., 2019a;

Original: Hendrycks et al., 2019).
Revised: Hendrycks et al., 2019).

Original: This occurs for different likelihood-based generative models, and even when inputs are unrelated to training data or have totally different semantics.
Revised: This occurs for different likelihood-based generative models, and even when inputs are unrelated to training data or have totally different semantics.

Original: For instance, when trained on CIFAR10, generative models report higher likelihoods for SVHN than for CIFAR10 itself (Fig. 1; data descriptions are available in Appendix A).
Revised: For instance, when trained on CIFAR10, generative models report higher likelihoods for SVHN than for CIFAR10 itself (Fig. 1; data descriptions are available in Appendix A).

Original: Intriguingly, this behavior is not consistent across data sets, as other ones correctly tend to produce likelihoods lower than the ones of the training data (see the example of TrafﬁcSign in Fig. 1).
Revised: Intriguingly, this behavior is not consistent across data sets, as other ones correctly tend to produce likelihoods lower than the ones of the training data (see the example of TrafﬁcSign in Fig. 1).

Original: A number of explanations have been suggested for the root cause of this behavior (Choi et al., 2018;
Revised: A number of explanations have been suggested for the root cause of this behavior (Choi et al., 2018;

Original: Nalisnick et al., 2019a;
Revised: Nalisnick et al., 2019a;

Original: Ren et al., 2019) but, to date, a full understanding of the phenomenon remains elusive.
Revised: Ren et al., 2019) but, to date, a full understanding of the phenomenon remains elusive.

Original: [Figure]
Revised: [Figure]

Original: In this paper, we shed light to the above phenomenon, showing that likelihoods computed from generative models exhibit a strong bias towards the complexity of the corresponding inputs.
Revised: In this paper, we shed light to the above phenomenon, showing that likelihoods computed from generative models exhibit a strong bias towards the complexity of the corresponding inputs.

Original: We ﬁnd that qualitatively complex images tend to produce the lowest likelihoods, and that simple images always yield the highest ones.
Revised: We ﬁnd that qualitatively complex images tend to produce the lowest likelihoods, and that simple images always yield the highest ones.

Original: In fact, we show a clear negative correlation between quantitative estimates of complexity and the likelihood of generative models.
Revised: In fact, we show a clear negative correlation between quantitative estimates of complexity and the likelihood of generative models.

Original: In the second part of the paper, we propose to leverage such estimates of complexity to detect OOD inputs.
Revised: In the second part of the paper, we propose to leverage such estimates of complexity to detect OOD inputs.

Original: To do so, we introduce a widely-applicable OOD score for individual inputs that corresponds, conceptually, to a likelihoodratio test statistic.
Revised: To do so, we introduce a widely-applicable OOD score for individual inputs that corresponds, conceptually, to a likelihoodratio test statistic.

Original: We show that such score turns likelihood-based generative models into practical and effective OOD detectors, with performances comparable to, or even better than the state-of-theart.
Revised: We show that such score turns likelihood-based generative models into practical and effective OOD detectors, with performances comparable to, or even better than the state-of-theart.

Original: We base our experiments on an extensive collection of alternatives, including a pool of 12 data sets, two conceptually-different generative models, increasing model sizes, and three variants of complexity estimates.
Revised: We base our experiments on an extensive collection of alternatives, including a pool of 12 data sets, two conceptually-different generative models, increasing model sizes, and three variants of complexity estimates.

Original:  C OMPLEXITY B IAS IN L IKELIHOOD - BASED G ENERATIVE M ODELS
Revised:  C OMPLEXITY B IAS IN L IKELIHOOD - BASED G ENERATIVE M ODELS

Original: From now on, we shall consider the log-likelihood of an input x given a model M : (cid:96) M ( x ) = log 2 p ( x |M ) .
Revised: From now on, we shall consider the log-likelihood of an input x given a model M : (cid:96) M ( x ) = log 2 p ( x |M ) .

Original: Following common practice in evaluating generative models, negative log-likelihoods − (cid:96) M will be expressed in bits per dimension (Theis et al., 2016), where dimension corresponds to the total size of x (we resize all images to 3 × 32 × 32 pixels).
Revised: Following common practice in evaluating generative models, negative log-likelihoods − (cid:96) M will be expressed in bits per dimension (Theis et al., 2016), where dimension corresponds to the total size of x (we resize all images to 3 × 32 × 32 pixels).

Original: Note that the qualitative behavior of log-likelihoods is the same as likelihoods: ideally, OOD inputs should have a low (cid:96) M , while in-distribution data should have a larger (cid:96) M .
Revised: Note that the qualitative behavior of log-likelihoods is the same as likelihoods: ideally, OOD inputs should have a low (cid:96) M , while in-distribution data should have a larger (cid:96) M .

Original: Most literature compares likelihoods of a given model for a few data sets.
Revised: Most literature compares likelihoods of a given model for a few data sets.

Original: However, if we consider several different data sets at once and study their likelihoods, we can get some insight.
Revised: However, if we consider several different data sets at once and study their likelihoods, we can get some insight.

Original: In Fig.
Revised: In Fig.

Original: 2, we show the log-likelihood distributions for the considered data sets (Appendix A), computed with a Glow model trained on CIFAR10.
Revised: 2, we show the log-likelihood distributions for the considered data sets (Appendix A), computed with a Glow model trained on CIFAR10.

Original: We observe that the data set with a higher log-likelihood is
Revised: We observe that the data set with a higher log-likelihood is

Original: Constant, a data set of constant-color images, followed by Omniglot, MNIST, and FashionMNIST; all of those featuring gray-scale images with a large presence of empty black background.
Revised: Constant, a data set of constant-color images, followed by Omniglot, MNIST, and FashionMNIST; all of those featuring gray-scale images with a large presence of empty black background.

Original: On the other side of the spectrum, we observe that the data set with a lower log-likelihood is Noise, a data set of uniform random images, followed by TrafﬁcSign and TinyImageNet; both featuring colorful images with non-trivial background.
Revised: On the other side of the spectrum, we observe that the data set with a lower log-likelihood is Noise, a data set of uniform random images, followed by TrafﬁcSign and TinyImageNet; both featuring colorful images with non-trivial background.

Original: Such ordering is perhaps more clear by looking at the average log-likelihood of each data set (Appendix D).
Revised: Such ordering is perhaps more clear by looking at the average log-likelihood of each data set (Appendix D).

Original: If we think about the visual complexity of the images in those data sets, it would seem that log-likelihoods tend to grow when images become simpler and with less information or content.
Revised: If we think about the visual complexity of the images in those data sets, it would seem that log-likelihoods tend to grow when images become simpler and with less information or content.

Original: To further conﬁrm the previous observation, we design a controlled experiment where we can set different decreasing levels of image complexity.
Revised: To further conﬁrm the previous observation, we design a controlled experiment where we can set different decreasing levels of image complexity.

Original: We train a generative model with some data set, as before, but now compute likelihoods of progressively simpler inputs.
Revised: We train a generative model with some data set, as before, but now compute likelihoods of progressively simpler inputs.

Original: Such inputs are obtained by average-pooling the uniform random Noise images by factors of 1, 2, 4, 8, 16, and 32, and re-scaling back the images to the original size by nearest-neighbor up-sampling.
Revised: Such inputs are obtained by average-pooling the uniform random Noise images by factors of 1, 2, 4, 8, 16, and 32, and re-scaling back the images to the original size by nearest-neighbor up-sampling.

Original: Intuitively, a noise image with a pooling size of 1 (no pooling) has the highest complexity, while a noise image with a pooling of 32 (constant-color image) has the lowest complexity.
Revised: Intuitively, a noise image with a pooling size of 1 (no pooling) has the highest complexity, while a noise image with a pooling of 32 (constant-color image) has the lowest complexity.

Original: Pooling factors from 2 to 16 then account for
Revised: Pooling factors from 2 to 16 then account for

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: intermediate, decreasing levels of complexity.
Revised: intermediate, decreasing levels of complexity.

Original: The result of the experiment is a progressive growing of the log-likelihood (cid:96) M (Fig. 3).
Revised: The result of the experiment is a progressive growing of the log-likelihood (cid:96) M (Fig. 3).

Original: Given that the only difference between data is the pooling factor, we can infer that image complexity plays a major role in generative models’ likelihoods.
Revised: Given that the only difference between data is the pooling factor, we can infer that image complexity plays a major role in generative models’ likelihoods.

Original: Until now, we have consciously avoided a quantitative deﬁnition of complexity.
Revised: Until now, we have consciously avoided a quantitative deﬁnition of complexity.

Original: However, to further study the observed phenomenon
Revised: However, to further study the observed phenomenon

Original: [Figure]
Revised: [Figure]

Original: and despite the difﬁculty in quantifying the multiple aspects that affect the complexity of an input (cf. Lloyd, 2001), we have to adopt one.
Revised: and despite the difﬁculty in quantifying the multiple aspects that affect the complexity of an input (cf. Lloyd, 2001), we have to adopt one.

Original: A sensible choice would be to exploit the notion of Kolmogorov complexity (Kolmogorov, 1963) which, unfortunately, is noncomputable.
Revised: A sensible choice would be to exploit the notion of Kolmogorov complexity (Kolmogorov, 1963) which, unfortunately, is noncomputable.

Original: In such cases, we have to deal with it by calculating an upper bound using a lossless compression algorithm
Revised: In such cases, we have to deal with it by calculating an upper bound using a lossless compression algorithm

Original: (Cover & Thomas, 2006).
Revised: (Cover & Thomas, 2006).

Original: Given a set of inputs x coded with the same bit depth, the normalized size of their compressed versions, L ( x ) (in bits per dimension), can be considered a reasonable estimate of their complexity.
Revised: Given a set of inputs x coded with the same bit depth, the normalized size of their compressed versions, L ( x ) (in bits per dimension), can be considered a reasonable estimate of their complexity.

Original: That is, given the same coding depth, a highly complex input will require more bits per dimension, while a less complex one will be compressed with fewer bits per dimension.
Revised: That is, given the same coding depth, a highly complex input will require more bits per dimension, while a less complex one will be compressed with fewer bits per dimension.

Original: For images, we can use PNG, JPEG2000, or FLIF compressors (Appendix C).
Revised: For images, we can use PNG, JPEG2000, or FLIF compressors (Appendix C).

Original: For other data types such as audio or text, other lossless compressors should be available to produce a similar estimate.
Revised: For other data types such as audio or text, other lossless compressors should be available to produce a similar estimate.

Original: If we study the relation between generative models’ likelihoods and our complexity estimates, we observe that there is a clear negative correlation (Fig. 4).
Revised: If we study the relation between generative models’ likelihoods and our complexity estimates, we observe that there is a clear negative correlation (Fig. 4).

Original: Considering all data sets, we ﬁnd Pearson’s correlation coefﬁcients below − 0.75 for models trained on FashionMNIST, and below − 0.9 for models trained on CIFAR10, independently of the compressor used (Appendix D).
Revised: Considering all data sets, we ﬁnd Pearson’s correlation coefﬁcients below − 0.75 for models trained on FashionMNIST, and below − 0.9 for models trained on CIFAR10, independently of the compressor used (Appendix D).

Original: Such signiﬁcant correlations, all of them with inﬁnitesimal p-values, indicate that likelihood-based measures are highly inﬂuenced by the complexity of the input image, and that this concept accounts for most of their variance.
Revised: Such signiﬁcant correlations, all of them with inﬁnitesimal p-values, indicate that likelihood-based measures are highly inﬂuenced by the complexity of the input image, and that this concept accounts for most of their variance.

Original: In fact, such strong correlations suggest that one may replace the computed likelihood
Revised: In fact, such strong correlations suggest that one may replace the computed likelihood

Original: [Figure]
Revised: [Figure]

Original: values for the negative of the complexity estimate and obtain almost the same result (Appendix D).
Revised: values for the negative of the complexity estimate and obtain almost the same result (Appendix D).

Original: This implies that, in terms of detecting OOD inputs, a complexity estimate would perform as well (or bad) as the likelihoods computed from our generative models.
Revised: This implies that, in terms of detecting OOD inputs, a complexity estimate would perform as well (or bad) as the likelihoods computed from our generative models.

Original:  T ESTING O UT - OF - DISTRIBUTION I NPUTS 3. D EFINITION
Revised:  T ESTING O UT - OF - DISTRIBUTION I NPUTS 3. D EFINITION

Original: As complexity seems to account for most of the variability in generative models’ likelihoods, we propose to compensate for it when testing for possible OOD inputs.
Revised: As complexity seems to account for most of the variability in generative models’ likelihoods, we propose to compensate for it when testing for possible OOD inputs.

Original: Given that both negative loglikelihoods − (cid:96) M ( x ) and the complexity estimate L ( x ) are expressed in bits per dimension (Sec. 2), we can express our OOD score as a subtraction between the two:
Revised: Given that both negative loglikelihoods − (cid:96) M ( x ) and the complexity estimate L ( x ) are expressed in bits per dimension (Sec. 2), we can express our OOD score as a subtraction between the two:

Original: [Equation]
Revised: [Equation]

Original: Notice that, since we use negative log-likelihoods, the higher the S , the more OOD the input x will be (see below).
Revised: Notice that, since we use negative log-likelihoods, the higher the S , the more OOD the input x will be (see below).

Original: 3. I NTERPRETATION : O CCAM ’ S R AZOR AND THE O UT - OF - DISTRIBUTION P ROBLEM
Revised: 3. I NTERPRETATION : O CCAM ’ S R AZOR AND THE O UT - OF - DISTRIBUTION P ROBLEM

Original: Interestingly, S can be interpreted as a likelihood-ratio test statistic.
Revised: Interestingly, S can be interpreted as a likelihood-ratio test statistic.

Original: For that, we take the point of view of Bayesian model comparison or minimum description length principle
Revised: For that, we take the point of view of Bayesian model comparison or minimum description length principle

Original: (MacKay, 2003).
Revised: (MacKay, 2003).

Original: We can think of a compressor M 0 as a universal model, adjusted for all possible inputs and general enough so that it is not biased towards a particular type of data semantics.
Revised: We can think of a compressor M 0 as a universal model, adjusted for all possible inputs and general enough so that it is not biased towards a particular type of data semantics.

Original: Considering the probabilistic model associated with the size of the output produced by the lossless compressor (MacKay, 2003), we have
Revised: Considering the probabilistic model associated with the size of the output produced by the lossless compressor, we have

Original: [Equation]
Revised: [Equation]

Original: and, correspondingly,
Revised: and, correspondingly,

Original: [Equation]
Revised: [Equation]

Original: In Bayesian model comparison, we are interested in comparing the posterior probabilities of different models in light of data X .
Revised: In Bayesian model comparison, we are interested in comparing the posterior probabilities of different models in light of data X .

Original: In our setting, the trained generative model M is a ‘simpler’ version of the universal model M 0 , targeted to a speciﬁc semantics or data type.
Revised: In our setting, the trained generative model M is a ‘simpler’ version of the universal model M 0 , targeted to a speciﬁc semantics or data type.

Original: With it, one aims to approximate the marginal likelihood (or model evidence) for x ∈ X , which integrates out all model parameters:
Revised: With it, one aims to approximate the marginal likelihood (or model evidence) for x ∈ X , which integrates out all model parameters:

Original: [Equation]
Revised: [Equation]

Original: This integral is intractable, but current generative models can approximate p ( x |M ) with arbitrary accuracy (Kingma & Dhariwal, 2018).
Revised: This integral is intractable, but current generative models can approximate p ( x |M ) with arbitrary accuracy (Kingma & Dhariwal, 2018).

Original: Choosing between one or another model is then reduced to a simple likelihood ratio:
Revised: Choosing between one or another model is then reduced to a simple likelihood ratio:

Original: [Equation]
Revised: [Equation]

Original: For uniform priors p ( M 0 ) = p ( M ) = 1 / 2 , this ratio is reduced to
Revised: For uniform priors p ( M 0 ) = p ( M ) = 1 / 2 , this ratio is reduced to

Original: [Equation]
Revised: [Equation]

Original: which, using Eq. 2 for the last term, becomes Eq.
Revised: which, using Eq. 2 for the last term, becomes Eq.

Original: The ratio S accommodates the Occam’s razor principle.
Revised: The ratio S accommodates the Occam’s razor principle.

Original: Consider simple inputs that can be easily compressed by M 0 using a few bits, and that are not present in the training of M .
Revised: Consider simple inputs that can be easily compressed by M 0 using a few bits, and that are not present in the training of M .

Original: These cases have a high probability under M 0 , effectively correcting the abnormal high likelihood given by the learned model M .
Revised: These cases have a high probability under M 0 , effectively correcting the abnormal high likelihood given by the learned model M .

Original: The same effect will occur with complex inputs that are not present in the training data.
Revised: The same effect will occur with complex inputs that are not present in the training data.

Original: In these cases, both likelihoods will be low, but the universal lossless compressor Mwill predict those better than the learned model M .
Revised: In these cases, both likelihoods will be low, but the universal lossless compressor Mwill predict those better than the learned model M .

Original: The two situations will lead to large values of S .
Revised: The two situations will lead to large values of S .

Original: In contrast, inputs that belong to the data used to train the generative model M will always be better predicted by M than by M 0 , resulting in lower values of S . 
Revised: In contrast, inputs that belong to the data used to train the generative model M will always be better predicted by M than by M 0 , resulting in lower values of S . 3. U SING S IN P RACTICE

Revised: Given a training set X of in-distribution samples and the corresponding scores S ( x ) for each x ∈ X , we foresee a number of strategies to perform OOD detection in practice for new instances z .

Revised: The ﬁrst and more straightforward one is to use S ( z ) as it is, just as a score, to perform OOD ranking.

Revised: This can be useful to monitor the top-k, potentially more problematic instances z in a new set of unlabeled data Z .

Revised: The second strategy is to interpret S ( z ) as the corresponding Bayes factor in

Original: In practice, if testing data is available, we can optimize a threshold value for S  or tune it for a desired percentage of false positives and/or negatives.
Revised: Eq. 3, and directly assign z to be OOD for S ( z ) > 0 , or in-distribution otherwise (cf. MacKay, 2003). The decision is then taken with stronger evidence for higher absolute values of S ( z ) . A third strategy is to consider the empirical or null distribution of S for the full training set, S ( X ) .

Revised: We could then choose an appropriate quantile as threshold, adopting the notion of frequentist hypothesis testing (see for instance Nalisnick et al., 2019b).

Revised: Finally, if ground truth OOD data Y is available, a fourth strategy is to optimize a threshold value for S ( z ) .

Revised: Using X and Y , we can choose a threshold that targets a desired percentage of false positives or negatives.

Revised: The choice of a speciﬁc strategy will depend on the characteristics of the particular application under consideration.

Revised: In this work, we prefer to keep our evaluation generic and to not adopt any speciﬁc thresholding strategy (that is, we use S directly, as a score).

Revised: This also allows us to compare with the majority of reported values from the existing literature, which use the AUROC measure (see below).

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Ren et al.
Revised: Ren et al.

Original: (2019) have recently proposed the use of likelihood-ratio tests for OOD detection.
Revised: (2019) have recently proposed the use of likelihood-ratio tests for OOD detection.

Original: They posit that “background statistics” (for instance, the number of zeros in the background of MNISTlike images) are the source of abnormal likelihoods, and propose to exploit them by learning a background model which is trained on random surrogates of input data.
Revised: They posit that “background statistics” (for instance, the number of zeros in the background of MNISTlike images) are the source of abnormal likelihoods, and propose to exploit them by learning a background model which is trained on random surrogates of input data.

Original: Such surrogates are generated according to a Bernoulli distribution, and an L2 regularization term is added to the background model, which implies that the approach has two hyper-parameters.
Revised: Such surrogates are generated according to a Bernoulli distribution, and an L2 regularization term is added to the background model, which implies that the approach has two hyper-parameters.

Original: Moreover, both the background model and the model trained using in-distribution data need to capture the background information equally well.
Revised: Moreover, both the background model and the model trained using in-distribution data need to capture the background information equally well.

Original: In contrast to their method, our method does not require additional training nor extra conditions on a speciﬁc background model for every type of training data.
Revised: In contrast to their method, our method does not require additional training nor extra conditions on a speciﬁc background model for every type of training data.

Original: Choi et al.
Revised: Choi et al.

Original: (2018) and Nalisnick et al.
Revised: (2018) and Nalisnick et al.

Original: (2019b) suggest that typicality is the culprit for likelihoodbased generative models not being able to detect OOD inputs.
Revised: (2019b) suggest that typicality is the culprit for likelihoodbased generative models not being able to detect OOD inputs.

Original: While Choi et al.
Revised: While Choi et al.

Original: (2018) do not explicitly address typicality, their estimate of the Watanabe-Akaike information criterion using ensembles of generative models performs well in practice.
Revised: (2018) do not explicitly address typicality, their estimate of the Watanabe-Akaike information criterion using ensembles of generative models performs well in practice.

Original: Nalisnick et al.
Revised: Nalisnick et al.

Original: (2019b) propose an explicit test for typicality employing a Monte Carlo estimate of the empirical entropy, which limits their approach to batches of inputs of the same type.
Revised: (2019b) propose an explicit test for typicality employing a Monte Carlo estimate of the empirical entropy, which limits their approach to batches of inputs of the same type.

Original: The works of Høst-Madsen et al.
Revised: The works of Høst-Madsen et al.

Original: and Sabeti & Høst-Madsen (2019) combine the concepts of typicality and minimum description length to perform novelty detection.
Revised: and Sabeti & Høst-Madsen (2019) combine the concepts of typicality and minimum description length to perform novelty detection.

Original: Although concepts are similar to the ones employed here, their focus is mainly on bit sequences.
Revised: Although concepts are similar to the ones employed here, their focus is mainly on bit sequences.

Original: They consider atypical sequences those that can be described (coded) with fewer bits in itself rather than using the (optimum) code for typical sequences.
Revised: They consider atypical sequences those that can be described (coded) with fewer bits in itself rather than using the (optimum) code for typical sequences.

Original: We ﬁnd their implementation to rely on strong parametric assumptions, which makes it difﬁcult to generalize to generative or other machine learning models.
Revised: We ﬁnd their implementation to rely on strong parametric assumptions, which makes it difﬁcult to generalize to generative or other machine learning models.

Original: A number of methods have been proposed to perform OOD detection under a classiﬁcation-based framework (Hendrycks & Gimpel, 2017;
Revised: A number of methods have been proposed to perform OOD detection under a classiﬁcation-based framework (Hendrycks & Gimpel, 2017;

Original: Lakshminarayanan et al., 2017; Liang et al., 2018;
Revised: Lakshminarayanan et al., 2017; Liang et al., 2018;

Original: Alemi et al., 2018; Lee et al., 2018; Hendrycks et al., 2019).
Revised: Alemi et al., 2018; Lee et al., 2018; Hendrycks et al., 2019).

Original: Although achieving promising results, these methods do not generally apply to the more general case of non-labeled or self-supervised data.
Revised: Although achieving promising results, these methods do not generally apply to the more general case of non-labeled or self-supervised data.

Original: The method of Hendrycks et al.
Revised: The method of Hendrycks et al.

Original: (2019) extends to such cases by leveraging generative models, but nonetheless makes use of auxiliary, outlier data to learn to distinguish OOD inputs.[Table]
Revised: (2019) extends to such cases by leveraging generative models, but nonetheless makes use of auxiliary, outlier data to learn to distinguish OOD inputs.

Original:  R ESULTS
Revised:  R ESULTS

Original: We now study how S performs on the OOD detection task.
Revised: We now study how S performs on the OOD detection task.

Original: For that, we train a generative model M on the train partition of a given data set and compute scores for such partition and the test partition of a different data set.
Revised: For that, we train a generative model M on the train partition of a given data set and compute scores for such partition and the test partition of a different data set.

Original: With both sets of scores, we then calculate the area under the receiver operating characteristic curve (AUROC), which is a common evaluation measure for the OOD detection task (Hendrycks et al., 2019) and for classiﬁcation tasks in general (Fawcett, 2005).
Revised: With both sets of scores, we then calculate the area under the receiver operating characteristic curve (AUROC), which is a common evaluation measure for the OOD detection task (Hendrycks et al., 2019) and for classiﬁcation tasks in general. Note that AUROC represents a good performance summary across different score thresholds (Fawcett, 2005).

Original: First of all, we want to assess the improvement of S over log-likelihoods alone ( − (cid:96) M ).
Revised: First of all, we want to assess the improvement of S over log-likelihoods alone ( − (cid:96) M ).

Original: When considering likelihoods from generative models trained on CIFAR10, the problematic results reported by previous works become clearly apparent (Table 1).
Revised: When considering likelihoods from generative models trained on CIFAR10, the problematic results reported by previous works become clearly apparent (Table 1).

Original: The unintuitive higher likelihoods for SVHN observed in Sec.
Revised: The unintuitive higher likelihoods for SVHN observed in Sec.

Original: 1 now translate into a poor AUROC below 0.1.
Revised: 1 now translate into a poor AUROC below 0.1.

Original: This not only happens for SVHN, but also for Constant, Omniglot, MNIST, and FashionMNIST data sets, for which we observed consistently higher likelihoods than CIFAR10 in Sec. 2.
Revised: This not only happens for SVHN, but also for Constant, Omniglot, MNIST, and FashionMNIST data sets, for which we observed consistently higher likelihoods than CIFAR10 in Sec. 2.

Original: Likelihoods for the other data sets yield
Revised: Likelihoods for the other data sets yield

Original: AUROCs above the random baseline of 0.5, but none above 0.67.
Revised: AUROCs above the random baseline of 0.5, but none above 0.67.

Original: The only exception is the Noise data set, which is perfectly distinguishable from CIFAR10 using likelihood alone.
Revised: The only exception is the Noise data set, which is perfectly distinguishable from CIFAR10 using likelihood alone.

Original: For completeness, we include the AUROC values when trying to perform OOD with the test partition of CIFAR10.
Revised: For completeness, we include the AUROC values when trying to perform OOD with the test partition of CIFAR10.

Original: We see those are close to the ideal value of 0.5, showing that, as expected, the reported measures do not generally consider those samples to be OOD.
Revised: We see those are close to the ideal value of 0.5, showing that, as expected, the reported measures do not generally consider those samples to be OOD.

Original: We now look at the AUROCs obtained with S (Table 1).
Revised: We now look at the AUROCs obtained with S (Table 1).

Original: We see that, not only results are reversed for less complex datasets like MNIST or SVHN, but also that all AUROCs for the rest of the data sets improve as well.
Revised: We see that, not only results are reversed for less complex datasets like MNIST or SVHN, but also that all AUROCs for the rest of the data sets improve as well.

Original: The only exception to the last assertion among all studied combinations is the combination of TinyImageNet with PixelCNN++ and FLIF (see Appendix D for other combinations).
Revised: The only exception to the last assertion among all studied combinations is the combination of TinyImageNet with PixelCNN++ and FLIF (see Appendix D for other combinations).

Original: In general, we obtain AUROCs above 0.7, with many of them approaching 0.9 or 1.
Revised: In general, we obtain AUROCs above 0.7, with many of them approaching 0.9 or 1.

Original: Thus, we can conclude that S clearly improves over likelihoods alone in the OOD detection task, and that
Revised: Thus, we can conclude that S clearly improves over likelihoods alone in the OOD detection task, and that

Original: S is able to revert the situation with intuitively less complex data sets that were previously yielding a low AUROC.
Revised: S is able to revert the situation with intuitively less complex data sets that were previously yielding a low AUROC.

Original: We also study how the training set, the choice of compressor/generative model, or the size of the model affects the performance of S (Appendix D).
Revised: We also study how the training set, the choice of compressor/generative model, or the size of the model affects the performance of S (Appendix D).

Original: In terms of models and compressors, we do not observe a large difference between the considered combinations, except for a few isolated cases whose investigation we defer for future work.
Revised: In terms of models and compressors, we do not observe a large difference between the considered combinations, except for a few isolated cases whose investigation we defer for future work.

Original: In terms of model size, we do observe a tendency to provide better discrimination with increasing size.
Revised: In terms of model size, we do observe a tendency to provide better discrimination with increasing size.

Original: In terms of data sets, we ﬁnd the OOD detection task to be easier with FashionMNIST than with CIFAR10.
Revised: In terms of data sets, we ﬁnd the OOD detection task to be easier with FashionMNIST than with CIFAR10.

Original: We assume that this is due to the ease of the generative model to learn and approximate the density conveyed by the data.
Revised: We assume that this is due to the ease of the generative model to learn and approximate the density conveyed by the data.

Original: A similar but less marked trend is also observed for compressors, with better compressors yielding slightly improved
Revised: A similar but less marked trend is also observed for compressors, with better compressors yielding slightly improved

Revised: AUROCs than other, in principle, less powerful ones.

Revised: A takeaway from all these observations would be that using larger generative models and better compressors will yield a more reliable S and a

Original: [Table]
Revised: [Table]

Revised: Trained on:

Revised: FashionMNIST

Revised: CIFAR

Revised: OOD data:

Original: [Table]
Revised: [Table]

Revised: Classiﬁer

Original: S using Glow and FLIF (ours)
Revised: [Table] based approaches ODIN (Liang et al., 2018) a,

Original: [Table]
Revised: [Table]

Original: S
Revised: Alemi [Table] [Table] Mahalanobis (

Revised: [Table]

Revised: [Table]

Revised: Outlier exposure (Hendrycks

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Generative-based approaches

Revised: [Table]

Revised: Outlier

Revised: [Table]

Revised: Typicality

Revised: [Table]

Revised: [Table]

Revised: Glow and FLIF

Original: [Table]
Revised: [Table]

Original: AUROCs than other, in principle, less powerful ones.
Revised: ours) [Table] S [Table]

Revised: PixelCNN+

Original: A takeaway from all these observations would be that using larger generative models and better compressors will yield a more reliable S and a better AUROC.
Revised: [Table] and FLIF (ours) [Table] better AUROC.

Original: The conducted experiments support that, but a more in-depth analysis should be carried out to further conﬁrm this hypothesis.
Revised: The conducted experiments support that, but a more in-depth analysis should be carried out to further conﬁrm this hypothesis.

Original: Finally, we want to assess how S compares to previous approaches in the literature.
Revised: Finally, we want to assess how S compares to previous approaches in the literature.

Original: For that, we compile a number of reported AUROCs for both classiﬁer- and generative-based approaches and compare them with S .
Revised: For that, we compile a number of reported AUROCs for both classiﬁer- and generative-based approaches and compare them with S .

Original: Note that classiﬁer-based approaches, as mentioned in Sec. 1, are less applicable than generative-based ones.
Revised: Note that classiﬁer-based approaches, as mentioned in Sec. 1, are less applicable than generative-based ones.

Original: In addition, as they exploit label information, they might have an advantage over generative-based approaches in terms of performance (some also exploit external or outlier data; Sec. 4).
Revised: In addition, as they exploit label information, they might have an advantage over generative-based approaches in terms of performance (some also exploit external or outlier data; Sec. 4).

Original: We observe that S is competitive with both classiﬁer- and existing generative-based approaches (Table 2).
Revised: We observe that S is competitive with both classiﬁer- and existing generative-based approaches (Table 2).

Original: When training with FashionMNIST, S achieves the best scores among all considered approaches.
Revised: When training with FashionMNIST, S achieves the best scores among all considered approaches.

Original: The results with further test sets are also encouraging, with almost all AUROCs approaching 1 (Appendix D).
Revised: The results with further test sets are also encouraging, with almost all AUROCs ap- proaching 1 (Appendix D).

Original: When training with CIFAR10, S achieves similar or better performance than existing approaches.
Revised: When training with CIFAR10, S achieves similar or better performance than existing approaches.

Original: Noticeably, within generative-based approaches, S is only outperformed in two occasions by the same approach, WAIC, which uses ensembles of generative models (Sec. 4).
Revised: Noticeably, within generative-based approaches, S is only outperformed in two occasions by the same approach, WAIC, which uses ensembles of generative models (Sec. 4).

Original: On the one hand, it would be interesting to see how S could perform when using ensembles of models and compressors to produce better estimates of − (cid:96) M and L , respectively.
Revised: On the one hand, it would be interesting to see how S could perform when using ensembles of models and compressors to produce better estimates of − (cid:96) M and L , respectively.

Original: On the other hand, however, the use of a single generative model together with a single fast compression library makes
Revised: On the other hand, however, the use of a single generative model together with a single fast compression library makes

Original: S an efﬁcient alternative compared to WAIC and some other existing approaches.
Revised: S an efﬁcient alternative compared to WAIC and some other existing approaches.

Original: It is also worth noting that many existing approaches have a number of hyper-parameters that need to be tuned, sometimes with the help of outlier or additional data.
Revised: It is also worth noting that many existing approaches have a number of hyper-parameters that need to be tuned, sometimes with the help of outlier or additional data.

Original: In contrast, S is a parameter-free measure, which makes it easy to use and deploy.
Revised: In contrast, S is a parameter-free measure, which makes it easy to use and deploy.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We illustrate a fundamental insight with regard to the use of generative models’ likelihoods for the task of detecting OOD data.
Revised: We illustrate a fundamental insight with regard to the use of generative models’ likelihoods for the task of detecting OOD data.

Original: We show that input complexity has a strong effect in those likelihoods, and pose that it is the main culprit for the puzzling results of using generative models’ likelihoods for OOD detection.
Revised: We show that input complexity has a strong effect in those likelihoods, and pose that it is the main culprit for the puzzling results of using generative models’ likelihoods for OOD detection.

Original: In addition, we show that an estimate of input complexity can be used to compensate standard negative log-likelihoods in order to produce an efﬁcient and reliable OOD score.
Revised: In addition, we show that an estimate of input complexity can be used to compensate standard negative log-likelihoods in order to produce an efﬁcient and reliable OOD score.

Original: We also offer an interpretation of our score as a likelihood-ratio akin to Bayesian model comparison.
Revised: We also offer an interpretation of our score as a likelihood-ratio akin to Bayesian model comparison.

Original: Such score performs comparably to, or even better than several state-of-the-art approaches, with results that are consistent across a range of data sets, models, model sizes, and compression algorithms.
Revised: Such score performs comparably to, or even better than several state-of-the-art approaches, with results that are consistent across a range of data sets, models, model sizes, and compression algorithms.

Original: The proposed score has no hyper-parameters besides the deﬁnition of a generative model and a compression algorithm, which makes it easy to employ in a variety of practical problems and situations.
Revised: The proposed score has no hyper-parameters besides the deﬁnition of a generative model and a compression algorithm, which makes it easy to employ in a variety of practical problems and situations.

Original: B M ODELS AND T RAINING
Revised: B M ODELS AND T RAINING

Original: The results of this paper are obtained using two generative models of different nature: one autoregressive model and one invertible model.
Revised: The results of this paper are obtained using two generative models of different nature: one autoregressive model and one invertible model.

Original: As autoregressive model we choose PixelCNN++ (Salimans et al., 2017), which has been shown to obtain very good results in terms of likelihood for image data.
Revised: As autoregressive model we choose PixelCNN++ (Salimans et al., 2017), which has been shown to obtain very good results in terms of likelihood for image data.

Original: As invertible model we choose Glow (Kingma & Dhariwal, 2018), which is also capable of inferring exact log-likelihoods using large stacks of bijective transformations.
Revised: As invertible model we choose Glow (Kingma & Dhariwal, 2018), which is also capable of inferring exact log-likelihoods using large stacks of bijective transformations.

Original: We implement the
Revised: We implement the

Original: Glow model using the default conﬁguration of the original implementation 1 , except that we zeropad and do not use ActNorm inside the coupling network.
Revised: Glow model using the default conﬁguration of the original implementation 1 , except that we zeropad and do not use ActNorm inside the coupling network.

Original: The model has 3 blocks of 32 ﬂows, using an afﬁne coupling with an squeezing factor of 2.
Revised: The model has 3 blocks of 32 ﬂows, using an afﬁne coupling with an squeezing factor of 2.

Original: As for PixelCNN++, we set 5 residual blocks per stage, with 80 ﬁlters and 10 logistic components in the mixture.
Revised: As for PixelCNN++, we set 5 residual blocks per stage, with 80 ﬁlters and 10 logistic components in the mixture.

Original: The non-linearity of the residual layers corresponds to an exponential linear unit 2 .
Revised: The non-linearity of the residual layers corresponds to an exponential linear unit 2 .

Original: We train both Glow and PixelCNN++ using the Adam optimizer with an initial learning rate of 10 − 4 .
Revised: We train both Glow and PixelCNN++ using the Adam optimizer with an initial learning rate of 10 − 4 .

Original: We reduce this initial value by a factor of 1 / 5 every time that the validation loss does not decrease during 5 consecutive epochs.
Revised: We reduce this initial value by a factor of 1 / 5 every time that the validation loss does not decrease during 5 consecutive epochs.

Original: The training ﬁnishes when the learning rate is reduced by factor of 1 / 100 .
Revised: The training ﬁnishes when the learning rate is reduced by factor of 1 / 100 .

Original: The batch size of both models is set to 50.
Revised: The batch size of both models is set to 50.

Original: The ﬁnal model weights are the ones yielding the best validation loss.
Revised: The ﬁnal model weights are the ones yielding the best validation loss.

Original: The likelihoods obtained in validation with both Glow and PixelCNN++ match the ones reported in the literature for CIFAR10 (Kingma & Dhariwal, 2018; Salimans et al., 2017).
Revised: The likelihoods obtained in validation with both Glow and PixelCNN++ match the ones reported in the literature for CIFAR10 (Kingma & Dhariwal, 2018; Salimans et al., 2017).

Original: We also make sure that the generated images are of comparable quality to the ones shown in those references.
Revised: We also make sure that the generated images are of comparable quality to the ones shown in those references.

Original: We use PyTorch version 1.2.0 (Paszke et al., 2017).
Revised: We use PyTorch version 1.2.0 (Paszke et al., 2017).

Original: All models have been trained with a single
Revised: All models have been trained with a single

Original: NVIDIA GeForce GTX 1080Ti GPU.
Revised: NVIDIA GeForce GTX 1080Ti GPU.

Original: Training takes some hours under that setting.
Revised: Training takes some hours under that setting.

Original: C C OMPRESSORS AND C OMPLEXITY E STIMATE
Revised: C C OMPRESSORS AND C OMPLEXITY E STIMATE

Original: We explore three different options to compress input images.
Revised: We explore three different options to compress input images.

Original: As a mandatory condition, they need to provide lossless compression.
Revised: As a mandatory condition, they need to provide lossless compression.

Original: The ﬁrst format that we consider is PNG, and old-classic format which is globally used and well-known.
Revised: The ﬁrst format that we consider is PNG, and old-classic format which is globally used and well-known.

Original: We use OpenCV 3 to compress from raw Numpy matrices, with compression set to the maximum possible level.
Revised: We use OpenCV 3 to compress from raw Numpy matrices, with compression set to the maximum possible level.

Original: The second format that we consider is
Revised: The second format that we consider is

Original: JPEG2000.
Revised: JPEG2000.

Original: Although not as globally known as the previous one, it is a more modern format with several new generation features such as progressive decoding.
Revised: Although not as globally known as the previous one, it is a more modern format with several new generation features such as progressive decoding.

Original: Again, we use the default OpenCV implementation to obtain the size of an image using this compression algorithm.
Revised: Again, we use the default OpenCV implementation to obtain the size of an image using this compression algorithm.

Original: The third format that we consider is FLIF, the most modern algorithm of the list.
Revised: The third format that we consider is FLIF, the most modern algorithm of the list.

Original: According to its website 4 , it promises to generate up to 53% smaller ﬁles than JPEG2000.
Revised: According to its website 4 , it promises to generate up to 53% smaller ﬁles than JPEG2000.

Original: We use the publicly-available compressor implementation in their website.
Revised: We use the publicly-available compressor implementation in their website.

Original: We do not include header sizes in the measurement of the resulting bits per dimension.
Revised: We do not include header sizes in the measurement of the resulting bits per dimension.

Original: To compute our complexity estimate L ( x ) , we compress the input x with one of the compressors C above.
Revised: To compute our complexity estimate L ( x ) , we compress the input x with one of the compressors C above.

Original: With that, we obtain a string of bits C ( x ) .
Revised: With that, we obtain a string of bits C ( x ) .

Original: The length of it, | C ( x ) | , is normalized by the size or dimensionality of x , which we denote by d , to obtain the complexity estimate:
Revised: The length of it, | C ( x ) | , is normalized by the size or dimensionality of x , which we denote by d , to obtain the complexity estimate:

Original: [Equation]
Revised: [Equation]

Original: We also experimented with an improved version of L ,
Revised: We also experimented with an improved version of L ,

Original: [Equation]
Revised: [Equation]

Original: where L i corresponds to different compression schemes.
Revised: where L i corresponds to different compression schemes.

Original: This forces S to work always with the best compressor for every x .
Revised: This forces S to work always with the best compressor for every x .

Original: In our case, as FLIF was almost always the best compressor, we did not observe a clear difference between using L or L (cid:48) .
Revised: In our case, as FLIF was almost always the best compressor, we did not observe a clear difference between using L or L (cid:48) .

Original: However, in cases where it is not clear which compressor to use or cases in which we do not have a clear best/winner, L (cid:48) could be of use.
Revised: However, in cases where it is not clear which compressor to use or cases in which we do not have a clear best/winner, L (cid:48) could be of use.

Original: D A DDITIONAL R ESULTS
Revised: D A DDITIONAL R ESULTS

Original: The additional results mentioned in the main paper are the following:
Revised: The additional results mentioned in the main paper are the following:

Original: • In Table 4, we report the average log-likelihood (cid:96) M for every data set.
Revised: • In Table 4, we report the average log-likelihood (cid:96) M for every data set.

Original: We sort data sets from highest to lowest log-likelihood.
Revised: We sort data sets from highest to lowest log-likelihood.

Original: • In Table 5, we report the global Pearson’s correlation coefﬁcient for different models, train sets, and compressors.
Revised: • In Table 5, we report the global Pearson’s correlation coefﬁcient for different models, train sets, and compressors.

Original: Due to the large sample size, Scipy version 1.2.1 reports a p-value of 0 in all cases.
Revised: Due to the large sample size, Scipy version 1.2.1 reports a p-value of 0 in all cases.

Original: • In Table 6, we report the AUROC values obtained from log-likelihoods (cid:96) M , complexity estimates L , a simple two-tail test T taking into account lower and higher log-likelihoods, T = | (cid:96) M − (cid:96) M | , and the proposed score S .
Revised: • In Table 6, we report the AUROC values obtained from log-likelihoods (cid:96) M , complexity estimates L , a simple two-tail test T taking into account lower and higher log-likelihoods, T = | (cid:96) M − (cid:96) M | , and the proposed score S .

Original: • In Table 7, we report the AUROC values obtained from S across different Glow model sizes, using a PNG compressor.
Revised: • In Table 7, we report the AUROC values obtained from S across different Glow model sizes, using a PNG compressor.

Original: • In Table 8, we report the AUROC values obtained from S across different data sets, models, and compressors.
Revised: • In Table 8, we report the AUROC values obtained from S across different data sets, models, and compressors.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: SVHN 0.
Revised: SVHN 0.

Original: CIFAR 0.
Revised: CIFAR 0.

Original: CelebA 0.
Revised: CelebA 0.

Original: FaceScrub 0.
Revised: FaceScrub 0.

Original: TinyImageNet 0.
Revised: TinyImageNet 0.

Original: TrafﬁcSign 0.
Revised: TrafﬁcSign 0.

Original: Noise 1.
Revised: Noise 1.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]