Original: A UTOREGRESSIVE G ENERATIVE M ODELING WITH N OISE C ONDITIONAL M AXIMUM L IKELIHOOD E STIMATION
Revised: A UTOREGRESSIVE G ENERATIVE M ODELING WITH N OISE C ONDITIONAL M AXIMUM L IKELIHOOD E STIMATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce a simple modification to the standard maximum likelihood estimation (MLE) framework.
Revised: We introduce a simple modification to the standard maximum likelihood estimation (MLE) framework.

Original: Rather than maximizing a single unconditional likelihood of the data under the model, we maximize a family of noise conditional likelihoods consisting of the data perturbed by a continuum of noise levels.
Revised: Rather than maximizing a single unconditional likelihood of the data under the model, we maximize a family of noise conditional likelihoods consisting of the data perturbed by a continuum of noise levels.

Original: We find that models trained this way are more robust to noise, obtain higher test likelihoods, and generate higher quality images.
Revised: We find that models trained this way are more robust to noise, obtain higher test likelihoods, and generate higher quality images.

Original: They can also be sampled from via a novel score-based sampling scheme which combats the classical covariate shift problem that occurs during sample generation in autoregressive models.
Revised: They can also be sampled from via a novel score-based sampling scheme which combats the classical covariate shift problem that occurs during sample generation in autoregressive models.

Original: Applying this augmentation to autoregressive image models, we obtain 3.32 bits per dimension on the ImageNet 64x64 dataset, and substantially improve the quality of generated samples in terms of the Frechet Inception distance (FID) — from 37.50 to 12.09 on the CIFAR-10 dataset.
Revised: Applying this augmentation to autoregressive image models, we obtain 3.32 bits per dimension on the ImageNet 64x64 dataset, and substantially improve the quality of generated samples in terms of the Frechet Inception distance (FID) — from 37.50 to 12.09 on the CIFAR-10 dataset.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Likelihood maximization models, i.e. , models trained by maximizing log-likelihood, are a leading class of modern generative models.
Revised: Maximum likelihood estimation (MLE) is arguably the gold standard for probabilistic model fitting, and serves as the de facto method for parameter estimation in countless statistical problems

Original: Of these, autoregressive models boast state-of-the-art performance in many domains, including images Child et al.
Revised: Bishop (2006), across a variety of fields. Estimators obtained via MLE enjoy a number of theoretical guarantees, including consistency, efficiency, asymptotic normality, and equivariance to model reparameterizations Van der Vaart (2000). In the field of density estimation and generative modeling, MLE models have played a key role, where autoregressive models and normalizing flows have boasted competitive performance in a bevy of domains, including images Child et al.

Original: (2019), text Vaswani et al.
Revised: , text Vaswani et al.

Original: (2017),and audio Oord et al.
Revised: (2017), audio Oord et al.

Revised: (2016), and tabular data Papamakarios et al.

Original: However, while log-likelihood is broadly agreed upon as one of the most rigorous metrics for goodness-of-fit in statistical and generative modeling, models with high likelihoods do not necessarily produce samples of high visual quality.
Revised: However, while log-likelihood is broadly agreed upon as one of the most rigorous metrics for goodness-of-fit in statistical and generative modeling, models with high likelihoods do not necessarily produce samples of high visual quality.

Original: This phenomenon has been discussed at length by
Revised: This phenomenon has been discussed at length by

Original: Theis et al.
Revised: Theis et al.

Original: ; Husz´ar (2015), and corroborated in empirical studies Grover et al.
Revised: ; Husz´ar (2015), and corroborated in empirical studies Grover et al.

Original: ; Kim et al.
Revised: ; Kim et al.

Original: Autoregressive models suffer an additional affliction: they have notoriously unstable dynamics during sample generation Bengio et al.
Revised: Autoregressive models suffer an additional affliction: they have notoriously unstable dynamics during sample generation Bengio et al.

Original:  Lamb et al.
Revised: ; Lamb et al.

Original: (2016) due to their sequential sampling algorithm, which can cause errors to compound across time steps.
Revised: (2016) due to their sequential sampling algorithm, which can cause errors to compound across time steps.

Original: Such errors cannot usually be corrected ex post facto due to the autoregressive structure of the model, and can substantially affect downstream steps as we find that model likelihoods are highly sensitive to even the most minor of perturbations.
Revised: Such errors cannot usually be corrected ex post facto due to the autoregressive structure of the model, and can substantially affect downstream steps as we find that model likelihoods are highly sensitive to even the most minor of perturbations.

Original: Score-based diffusion models Song et al.
Revised: Score-based diffusion models Song et al.

Revised: (2020b)

Original:  Ho et al.
Revised: ; Ho et al.

Original: (2020) offer a different perspective on the data generation process.
Revised: (2020) offer a different perspective on the data generation process.

Original: Even though sampling is also sequential, diffusion models are more robust to perturbations because, in essence, they are trained as denoising functions Ho et al.
Revised: Even though sampling is also sequential, diffusion models are more robust to perturbations because, in essence, they are trained as denoising functions Ho et al.

Original: Moreover, the update direction in each step is unconstrained (unlike token-wise autoregressive models, which can only update one token at a time, and only once), meaning the model can correct errors from previous steps.
Revised: Moreover, the update direction in each step is unconstrained (unlike token-wise autoregressive models, which can only update one token at a time, and only once), meaning the model can correct errors from previous steps.

Original: However,diffusion models are poor likelihoodmodels, as they cannot be trained via maximum likelihood 1 , and density evaluationsare slow and have no closed form, requiring ODE/SDE solvers involving hundreds to thousands of calls to the underlying network.
Revised: However, likelihood evaluations have no closed form, requiring ODE/SDE solvers and hundreds to thousands of calls to the underlying network, and rendering the models incapable of being trained via MLE. Moreover, diffusion models do not inherit any of the asymptotic guarantees Hyv¨arinen & Dayan (2005); Song et al. (2020a) of score matching, even though they are thusly trained 1 . These details make diffusion models theoretically inferior and less viable for many downstream tasks that involve the likelihood, such as anomaly and out-of-distribution (OOD) detection Ren et al.

Revised: adversarial defense Song et al.

Revised: (2017), among others.

Original: Thus we wonder: is there a conceptual middle ground?
Revised: Thus we wonder: is there a conceptual middle ground?

Original: In this paper, we offer such a framework.
Revised: In this paper, we offer such a framework.

Original: We further analyze the likelihood-sample quality mismatch in autoregressive models, and propose techniques inspired by diffusion models to alleviate it.
Revised: We further analyze the likelihood-sample quality mismatch in autoregressive models, and propose techniques inspired by diffusion models to alleviate it.

Original: In particular, we leverage the fact that the score function is naturally learned as a byproduct of maximum likelihood estimation.
Revised: In particular, we leverage the fact that the score function is naturally learned as a byproduct of maximum likelihood estimation.

Original: This allows a novel two-part sampling strategy with noisy sampling and score-based refinement.
Revised: This allows a novel two-part sampling strategy with noisy sampling and score-based refinement.

Original: Our contributions are threefold.
Revised: Our contributions are threefold.

Original: We investigate the pitfalls of training and inference under the log-likelihood maximization scheme, particularly regarding sensitivity to noise corruptions.
Revised: We investigate the pitfalls of training and inference under the maximum likelihood estimation framework, particularly regarding sensitivity to noise corruptions.

Original: We propose a simple sanity test for checking the robustness of likelihood models to minor perturbations, and find that many models fail this test.
Revised: We propose a simple sanity test for checking the robustness of likelihood models to minor perturbations, and find that many models fail this test.

Original: We introduce a novel framework for the training and sampling of likelihood maximization models that significantly improves the noise-robustness and generated sample quality.
Revised: We introduce a novel framework for the training and sampling of MLE models that significantly improves the noise-robustness and generated sample quality.

Original: As a result, we obtain a model that can generate samples at a quality approaching that of diffusion models, without losing the maximum likelihood framework and O (1) likelihood evaluation speed of likelihood maximization models.
Revised: As a result, we obtain a model that can generate samples at a quality approaching that of diffusion models, without losing the maximum likelihood framework and O (1) likelihood evaluation speed of MLE models.

Original: The paper is structured as follows.

Original: We will cover background and related work (Section 2), further investigate shortcomings of vanilla maximum likelihood (Section 3), present the noise-conditional maximum likelihood (NCML) framework (Section 4), and finally evaluate the robustness and performance of models trained under this framework (Section 5).

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original: Let our dataset X consist of i.i.d.
Revised: Let our dataset X consist of i.i.d.

Original: samples drawn from an unknown target density x ∼ p data ( x ) .
Revised: samples drawn from an unknown target density x ∼ p data ( x ) .

Original: The goal of likelihood-based generative modeling is to approximate p data via a parametric model p θ , where samples x ∼ p θ can be easily obtained.
Revised: The goal of likelihood-based generative modeling is to approximate p data via a parametric model p θ , where samples x ∼ p θ can be easily obtained.

Original: 2. B ACKGROUND
Revised: 2. B ACKGROUND

Original: We first discuss fundamental techniques for estimating and sampling from p θ in generative modeling.
Revised: We first discuss fundamental techniques for estimating and sampling from p θ in generative modeling.

Original: Maximum Likelihood Estimation (MLE)
Revised: Maximum Likelihood Estimation (MLE)

Original: The standard MLE framework consists in solving
Revised: The standard MLE framework consists in solving

Original: [Equation]
Revised: [Equation]

Original: When p θ is autoregressive, the likelihood of each sample can be further decomposed by the probabilistic chain rule,
Revised: When p θ is autoregressive, the likelihood of each sample can be further decomposed by the probabilistic chain rule,

Original: i.e. , p θ ( x ) =
Revised: i.e. , p θ ( x ) =

Original: p θ ( x 1 ) p θ ( x 2 | x 1 i ) . . .
Revised: p θ ( x 1 ) p θ ( x 2 | x 1 i ) . . .

Original: p θ ( x d | x d − 1 . . .
Revised: p θ ( x d | x d − 1 . . .

Original: x 1 ) where x k denotes the k th dimension of x .
Revised: x 1 ) where x k denotes the k th dimension of x .

Original: Likelihood models draw samples x ∼ p θ one of two ways.
Revised: Likelihood models draw samples x ∼ p θ one of two ways.

Original: Normalizing flows Dinh et al.
Revised: Normalizing flows Dinh et al.

Original: Rezende & Mohamed (2015) apply a series of invertible transformations to a latent variable z ∈ R d ∼ p prior .
Revised: Rezende & Mohamed (2015); Grathwohl et al. (2018) apply a series of invertible transformations to a latent variable z ∈ R d ∼ p prior .

Original: On the other hand, autoregressive models sample x sequentially and coordinate-wise by drawing from each conditional likelihood.
Revised: On the other hand, autoregressive models sample x sequentially and coordinate-wise by drawing from each conditional likelihood.

Original: Score-based Modeling An alternative to MLE is score matching
Revised: Score-based Modeling An alternative to MLE is score matching

Original: Hyv¨arinen & Dayan (2005), i.e.
Revised: Hyv¨arinen & Dayan (2005), i.e.

Original: [Equation]
Revised: [Equation]

Original: where ∇ x  p ( x ) is also known as the (Stein) score function.
Revised: where ∇ x log p ( x ) is also known as the (Stein) score function.

Original: Most score-based models sidestep the estimation p θ and directly estimate the score.
Revised: Most score-based models sidestep the estimation p θ and directly estimate the score.

Original: Sampling from p θ can then be achieved via annealed
Revised: Sampling from p θ can then be achieved via annealed

Original: Langevin dynamics Song & Ermon (2019),
Revised: Langevin dynamics Song & Ermon (2019),

Original: variational denoising Ho et al.
Revised: variational denoising Ho et al.

Original: (2020), or reversing a diffusion process Song et al. 
Revised: (2020), or reversing a diffusion process Song et al. (2020b).

Original: Of these, diffusion models Song et al.
Revised: Of these, diffusion models Song et al.

Original: (2020) provide the most general framework for score-based sampling, as well as a means to recover p θ by solving an ordinary differential equation (ODE).
Revised: (2020b) provide the most general framework for score-based sampling, as well as a means to recover p θ by solving an ordinary differential equation (ODE).

Original: Here, each data point is modeled as a diffusion process, and can be seen as a function x : [0 , T ] → R d such that x (0) ∼ p data and x ( T ) ∼ p prior .
Revised: Here, each data point is modeled as a diffusion process, and can be seen as a function x : [0 , T ] → R d such that x (0) ∼ p data and x ( T ) ∼ p prior .

Original: The forward diffusion process is an Ito stochastic differential equation (SDE)
Revised: The forward diffusion process is an Ito stochastic differential equation (SDE)

Original: [Equation]
Revised: [Equation]

Original: for some drift and diffusion terms f and g , where w is the standard Wiener process.
Revised: for some drift and diffusion terms f and g , where w is the standard Wiener process.

Original: By a central result in Anderson (1982), this diffusion can be tractably reversed, and is moreover also a diffusion process whose SDE
Revised: By a central result in Anderson (1982), this diffusion can be tractably reversed, and is moreover also a diffusion process whose SDE

Original: [Equation]
Revised: [Equation]

Original: depends again on f , g , and additionally the noise-conditional score function, where ¯ w is a backwards
Revised: depends again on f , g , and additionally the noise-conditional score function, where ¯ w is a backwards

Original: Wiener process.
Revised: Wiener process.

Original: Sampling then consists of drawing x ( T ) ∼ p prior , and solving the reverse diffusion process.
Revised: Sampling then consists of drawing x ( T ) ∼ p prior , and solving the reverse diffusion process.

Original: A Hybrid Framework?

Original: MLE and score-based models each have advantages and drawbacks when measured against the other.

Original: Score-based models exhibit unrivaled generative capabilities, producing images of state-of-the-art sample quality that beat out even GANs Dhariwal & Nichol (2021).

Original: However, they cannot be trained by maximum likelihood, and therefore do not have the same asymptotic guarantees as MLE ( e.g. , efficiency, functional equivariance, sufficiency).

Original: Moreover, likelihoods have no closed-form expression, thus requiring approximate ODE/SDE solvers and hundreds to thousands of function evaluations to evaluate.

Original: MLE models, on the other hand, are best-in-class in terms of model log-likelihood and log-likelihood computation speed — which, outside of its inherent value, has proven to be useful in many downstream tasks, including anomaly and out-of-distribution (OOD) detection Ren et al.

Original: adversarial defense Song et al.

Original: (2017), among others.

Original: Additionally, they enjoy the guarantees that come with maximum likelihood estimation.

Original: However, they are well-known have subpar generative sample quality, which we further investigate in Section 3.

Original: These shortcomings raise the natural question of whether there exist frameworks that bridge the gap between the methods.

Original: In this paper, we explore this question.

Original: 2. R ELATED W ORK
Revised: 2. R ELATED W ORK

Original: A number of other works combine score- and energy-based modeling with autoregressive architectures.
Revised: A number of other works combine score- and energy-based modeling with autoregressive architectures.

Original: Hoogeboom et al.
Revised: Hoogeboom et al.

Original: (2021) propose an order-agnostic autoregressive model for simulatingUnder a discrete diffusions via a variational lower bound.
Revised: (2021) propose an order-agnostic autoregressive model for simulating discrete diffusions via a variational lower bound.

Original: Meng et al.
Revised: Meng et al.

Original: (2020) use unnormalized autoregressive models to learn distributions in an augmented score-matching framework.
Revised: (2020) use unnormalized autoregressive models to learn distributions in an augmented score-matching framework.

Original: Nash & Durkan (2019) design an energy-based model with an autoregressive structure such that the normalizing constant can be estimated coordinate-wise via importance sampling. 
Revised: Nash & Durkan (2019) design an energy-based model with an autoregressive structure such that the normalizing constant can be estimated coordinate-wise via importance sampling. Similar to our approach, Meng et al.

Revised: (2021) divide the data generation process into a noisy sampling step and a denoising step, but their formulation introduces latent variables and does not support adaptive refinement strategies.

Original: However, each approach relinquishes the ability to compute exact likelihoods in their framework, which is one of the motivating advantages of using autoregressive structures.
Revised: Ultimately, each approach relinquishes the ability to compute exact likelihoods in their framework, which is one of the motivating advantages of using autoregressive structures.

Original: A closely related vein of research explores alternative training and inference strategies so as to improve sampling stability in autoregressive models.
Revised: A closely related vein of research explores alternative training and inference strategies so as to improve sampling stability in autoregressive models.

Original: Bengio et al.
Revised: Bengio et al.

Original: (2015) propose training models with a mixture of true and generated samples over time, where the proportion of generated samples gradually grows to take up the majority of training sequences.
Revised: (2015) propose training models with a mixture of true and generated samples over time, where the proportion of generated samples gradually grows to take up the majority of training sequences.

Original: This is initially promising, except
Revised: This is initially promising, except

Original: Husz´ar (2015) note that this technique is biased and not guaranteed to produce solutions that converge on the true distribution.
Revised: Husz´ar (2015) note that this technique is biased and not guaranteed to produce solutions that converge on the true distribution.

Original: Lamb et al.
Revised: Lamb et al.

Original: (2016) subsequently suggest incorporating an adversarial  loss provided by a discriminator that ”teaches” the model to produce more realistic samples over multiple sampling steps.
Revised: (2016) subsequently suggest incorporating an adversarial differences are nearly to the human eye, yet cause drastic deviations in average log-likelihood for standard likelihood models (Section 3 and Table 1). loss provided by a discriminator that ”teaches” the model to produce more realistic samples over multiple sampling steps.

Original: Both techniques again depart from the maximum likelihood framework.
Revised: Both techniques again depart from the maximum likelihood framework.

Original:  Perhaps most similar to our approach is Jayaram & Thickstun (2021), who also suggest sampling via the score function with Langevin dynamics, but they crucially do not train with noise, which Song & Ermon (2019) found to be essential for stable sampling in a Langevin algorithm.
Revised: Jun et al. (2020) considers alternative forms of data augmentation, which Kingma et al. (2021) show can be complementarily combined with noise conditional perturbations for training. Perhaps most similar to our approach is Jayaram & Thickstun (2021), who also suggest sampling via the score function with Langevin dynamics, but they crucially do not train with noise, which Song & Ermon (2019) found to be essential for stable sampling in a Langevin algorithm.

Original: As a result, they are not able to sample images unconditionally.
Revised: As a result, they are not able to sample images unconditionally.

Original: Ultimately, our approach uniquely provides a principled and generalized framework for modeling stochastic processes (including diffusions) that retains the asymptotic guarantees of maximum likelihood estimation while producing samples of superior quality.
Revised: Ultimately, our approach uniquely provides a principled and generalized framework for modeling stochastic processes (including diffusions) that retains the asymptotic guarantees of maximum likelihood estimation while producing samples of superior quality.

Original:  T HE P ITFALLS OF M AXIMUM L IKELIHOOD 
Revised:  T HE P ITFALLS OF M AXIMUM L IKELIHOOD E STIMATION

Original: We first show that density models trained to maximize the standard log-likelihood are surprisingly sensitive to minor perturbations.
Revised: We first show that density models trained to maximize the standard log-likelihood are surprisingly sensitive to minor perturbations.

Original: We then discuss why this is bad for generative modeling performance.
Revised: We then discuss why this is bad for generative modeling performance.

Original: 3. A S IMPLE S ANITY T EST
Revised: 3. A S IMPLE S ANITY T EST

Original: Consider the class of minimally corrupted probability densities we call p π , where
Revised: Consider the class of minimally corrupted probability densities we call p π , where

Original: [Equation]
Revised: [Equation]

Original: Here, ∗ denotes the convolution operator, and p mult { a,b,c } ( α,β,γ ) is the density a d -dimensional multi- nomial distribution taking on a , b , and c with probabilities α , β , and γ , respectively.
Revised: Here, ∗ denotes the convolution operator, and p mult { a,b,c } ( α,β,γ ) is the density a d -dimensional multinomial distribution taking on a , b , and c with probabilities α , β , and γ , respectively.

Original: p π is minimally corrupted in the sense that, if p data is an integer-discretized distribution (say, 8-bit images, or 16bit digital audio signals), p π describes the distribution of points in p data that have had their least significant bit incremented or decremented with probability π .
Revised: p π is minimally corrupted in the sense that, if p data is an integer-discretized distribution (say, 8-bit images, or 16-bit digital audio signals), p π describes the distribution of points in p data that have been perturbed by adding or subtracting 1 bit with probability π .

Original: In 8-bit images, the difference between samples drawn from p π and p data is almost imperceptible to the human eye, even for π = 1 (see Fig 2).
Revised: In 8-bit images, the difference between samples drawn from p π and p data is almost imperceptible to the human eye, even for π = 1 (see Fig 2).

Original: However, for likelihood models, this perturbation drastically increases the negative log-likelihood of the data under the model (see Table 1), to the point that it significantly undermines (if not outright nullifies)
Revised: However, for likelihood models, this perturbation drastically increases the negative log-likelihood of the data under the model (see Table 1), to the point that it significantly undermines (if not outright nullifies)

Original: any recent advances in density estimation.
Revised: any recent advances in density estimation.

Original: This basic inconsistency suggests that the learned density of many standard likelihood models is brittle and overly emphasizes bit-level statistics that have little influence on the inherent content of the image.
Revised: This basic inconsistency suggests that the learned density of many standard likelihood models is brittle and overly emphasizes bit-level statistics that have little influence on the inherent content of the image.

Original: 3. W HY W E S HOULD C ARE
Revised: 3. W HY W E S HOULD C ARE

Original: We provide three reasons for why failing this test is problematic, especially for autoregressive models.
Revised: We provide three reasons for why failing this test is problematic, especially for autoregressive models.

Original: First, noise is natural — and being less robust to noise also means being a poorer fit to natural data, especially in the ways that matter to the end user.
Revised: First, noise is natural — and being less robust to noise also means being a poorer fit to natural data, especially in the ways that matter to the end user.

Original: Outside of the log-likelihood, measures of generative success in generative models fall under two categories: qualitative assessments ( e.g. , the no-reference perceptual quality assessment Wang et al.
Revised: Outside of the log-likelihood, measures of generative success in generative models fall under two categories: qualitative assessments ( e.g. , the no-reference perceptual quality assessment Wang et al.

Original: or ’eyeballing’ it) and quantitative heuristics ( e.g. , computing statistics of hidden activations of pretrained CNNs Salimans et al.
Revised: or ’eyeballing’ it) and quantitative heuristics ( e.g. , computing statistics of hidden activations of pretrained CNNs Salimans et al.

Original: (2016); and π = bottom) .

Original: The differences are nearly indistinguishable to the human eye, yet cause drastic deviations in average log-likelihood for standard likelihood models (Section 3 and Table 1).

Original: Heusel et al.
Revised: Heusel et al.

Original: ; Sajjadi et al.
Revised: ; Sajjadi et al.

Original: Both strategies either rely directly on the human visual system, or are known to be closely related to it G¨uc¸l¨u & van Gerven (2015); Yamins et al.
Revised: Both strategies either rely directly on the human visual system, or are known to be closely related to it G¨uc¸l¨u & van Gerven (2015); Yamins et al.

Original: Khaligh-Razavi & Kriegeskorte (2014);
Revised: Khaligh-Razavi & Kriegeskorte (2014);

Original: Eickenberg et al.
Revised: Eickenberg et al.

Original: ; Cichy et al.
Revised: ; Cichy et al.

Original: Therefore, implicit in the use of these criteria is assuming the existence of a human (or human-like) model of images q human , where q human ≈ p data Husz´ar (2015).
Revised: Therefore, implicit in the use of these criteria is assuming the existence of a human (or human-like) model of images q human , where q human ≈ p data Husz´ar (2015).

Original: This assumption, as well as the fact that we find samples from p π nearly indistinguishable from p data , whereas p θ finds them very different, suggests that p data ≈ q human ̸
Revised: This assumption, as well as the fact that we find samples from p π nearly indistinguishable from p data , whereas p θ finds them very different, suggests that p data ≈ q human ̸

Original: = p θ .
Revised: = p θ .

Original: Second, it suggests that we may need to re-evaluate our current research trajectory in likelihood modeling on images.
Revised: Second, it suggests that we may need to re-evaluate our current research trajectory in likelihood modeling on images.

Original: Perhaps more surprising than failing the test is the fact all likelihood models fail the test more or less equally.
Revised: Perhaps more surprising than failing the test is the fact all likelihood models fail the test more or less equally.

Original: As touched upon in Section 3.1, Table 1 shows that recent gains in autoregressive likelihood modeling are largely diminished when evaluating the average model likelihood of p π for π = 0 .
Revised: As touched upon in Section 3.1, Table 1 shows that recent gains in autoregressive likelihood modeling are largely diminished when evaluating the average model likelihood of p π for π = 0 .

Original: 5 and π = 1 .
Revised: 5 and π = 1 .

Original: For example, the difference in likelihood between one of the first autoregressive likelihood models evaluated on CIFAR-10 Salimans et al.
Revised: For example, the difference in likelihood between one of the first autoregressive likelihood models evaluated on CIFAR-10 Salimans et al.

Original: (2017) and the current state of the art Child et al.
Revised: (2017) and the current state of the art Child et al.

Original: (2019) is 0.12 bits per dim (bpd).
Revised: (2019) is 0.12 bits per dim (bpd).

Original: However, under our perturbation, the difference is 0.02 bpd, which is more than a 6x reduction.
Revised: However, under our perturbation, the difference is 0.02 bpd, which is more than a 6x reduction.

Original: Similarly, the difference between the stateof-the-art and normalizing flow models, which are known to be inferior to autoregressive models in terms of likelihood, is also significantly reduced.
Revised: Similarly, the difference between the stateof-the-art and normalizing flow models, which are known to be inferior to autoregressive models in terms of likelihood, is also significantly reduced.

Original: This indicates that much of the progress in recent years leans heavily towards improvements in modeling the least significant bit, which ultimately bears little significance to the content of the image.
Revised: This indicates that much of the progress in recent years leans heavily towards improvements in modeling the least significant bit, which ultimately bears little significance to the content of the image.

Original: Third, generative sample quality suffers.
Revised: Third, generative sample quality suffers.

Original: This holds for general likelihood models, given what we argue in the first point — namely p θ is very different from q human .
Revised: This holds for general likelihood models, given what we argue in the first point — namely p θ is very different from q human .

Original: However, noise-sensitivity is doubly problematic in autoregressive models.
Revised: However, noise-sensitivity is doubly problematic in autoregressive models.

Original: Due to the sequential nature of autoregressive sampling and the fact that models are presented only with data from the true distribution during training, autoregressive models are already known to be poorly-equipped to handle the sequences they encounter during sampling Bengio et al.
Revised: Due to the sequential nature of autoregressive sampling and the fact that models are presented only with data from the true distribution during training, autoregressive models are already known to be poorly-equipped to handle the sequences they encounter during sampling Bengio et al.

Original: Any sampling error introduces a shift in distribution that can affect the model’s predicted likelihood of downstream tokens.
Revised: Any sampling error introduces a shift in distribution that can affect the model’s predicted likelihood of downstream tokens.

Original: This will increase the risk of mis-sampling the next token, which, in turn, further affects the downstream likelihoods.
Revised: This will increase the risk of mis-sampling the next token, which, in turn, further affects the downstream likelihoods.

Original: This is related to the well-known covariate shift phenomenon Shimodaira (2000).
Revised: This is related to the well-known covariate shift phenomenon Shimodaira (2000).

Original: Sensitivity to minor perturbations only exacerbates the problem, and Table 1 shows that in vanilla autoregressive likelihood models, mis-sampling pixels by even a single bit can cause drastic changes to the overall likelihood.
Revised: Sensitivity to minor perturbations only exacerbates the problem, and Table 1 shows that in vanilla autoregressive likelihood models, mis-sampling pixels by even a single bit can cause drastic changes to the overall likelihood.

Original: This can explain why such models commonly produce nonsensical results (Fig 1).
Revised: This can explain why such models commonly produce nonsensical results (Fig 1).

Original: For these reasons, we find that improving noise-robustness is of central importance to likelihoodbased generative modeling, and especially likelihood-based autoregressive generative models.
Revised: For these reasons, we find that improving noise-robustness is of central importance to likelihoodbased generative modeling, and especially likelihood-based autoregressive generative models.

Original:  N OISE C ONDITIONAL M AXIMUM L IKELIHOOD
Revised:  N OISE C ONDITIONAL M AXIMUM L IKELIHOOD

Original: To alleviate the problems discussed in Section 3, we propose a simple modification to the standard objective in maximum likelihood estimation.
Revised: To alleviate the problems discussed in Section 3, we propose a simple modification to the standard objective in maximum likelihood estimation.

Original: Rather than evaluating a single likelihood as in the vanilla formulation, we consider a family of noise conditional likelihoods
Revised: Rather than evaluating a single likelihood as in the vanilla formulation, we consider a family of noise conditional likelihoods

Original: [Equation]
Revised: [Equation]

Original: (6) where p t is a stochastic process indexed by noise scales t modeling a noise-corrupting process on p data , and µ is a distribution over such scales.
Revised: (6) where p t is a stochastic process indexed by noise scales t modeling a noise-corrupting process on p data , and µ is a distribution over such scales.

Original: We call this the noise conditional maximum likelihood (NCML) framework.
Revised: We call this the noise conditional maximum likelihood (NCML) framework.

Original: In general, Eq 6 is an all-purpose plug-in objective that can be used with any likelihood model adapted to accept a noise conditioning vector 2 .
Revised: In general, Eq 6 is an all-purpose plug-in objective that can be used with any likelihood model adapted to accept a noise conditioning vector 2 .

Original: We now explore various perspectives of NCML that may help with reasoning about this framework.
Revised: We now explore various perspectives of NCML that may help with reasoning about this framework.

Original: NCML as Regularized Maximum Likelihood When the set of noise scales t ∈ T is finitely large, a natural and mathematically equivalent formulation of NCML is as a form of data- and modeldependent regularization of the standard maximum likelihood estimation objective:
Revised: NCML as Regularized Maximum Likelihood When the set of noise scales t ∈ T is finitely large, a natural and mathematically equivalent formulation of NCML is as a form of data- and model- dependent regularization of the standard maximum likelihood estimation objective:

Original: [Equation]
Revised: [Equation]

Original: where T comprises the set of nonzero noise scales and λ t :
Revised: where T comprises the set of nonzero noise scales and λ t :

Original: = µ ( t ) /µ (0) .
Revised: = µ ( t ) /µ (0) .

Original: Clearly, the standard likelihood can be considered a special case of our proposed method where λ t = 0 for all t ∈ T .
Revised: Clearly, the standard likelihood can be considered a special case of our proposed method where λ t = 0 for all t ∈ T .

Original: Furthermore, since the NCML framework can simply be seen as formulating |T | simultaneous and separate MLE problems, it retains all the statistical properties of standard MLE.
Revised: Furthermore, since the NCML framework can simply be seen as formulating |T | simultaneous and separate MLE problems, it retains all the statistical properties of standard MLE.

Original: Of course, any form of regularization introduces bias to the model framework.
Revised: Of course, any form of regularization introduces bias to the model framework.

Original: Whereas L0/L1/L regularizations bias towards solutions of minimal or sparse weight norm, our experiments suggest that NCML biases towards solutions that are less sensitive to noisy perturbations (see Section 4.1).
Revised: Whereas L0/L1/L regularizations bias towards solutions of minimal or sparse weight norm, our experiments suggest that NCML biases towards solutions that are less sensitive to noisy perturbations (see Section 4.1).

Original: NCML as a Diffusion Model Letting t be the time index of a diffusion process, our approach becomes closely related to score-based diffusion models Song et al.
Revised: NCML as a Diffusion Model Letting t be the time index of a diffusion process, our approach becomes closely related to score-based diffusion models Song et al.

Original: (2020), albeit with two crucial differences.
Revised: (2020b), albeit with two crucial differences.

Original: First, instead of merely estimating the noise-conditional score ∇ x log p t ( x ) for t ∈ T , we directly estimate p t itself.
Revised: First, instead of merely estimating the noise-conditional score ∇ x log p t ( x ) for t ∈ T , we directly estimate p t itself.

Original: However, ∇ x log p t ( x ) is still learned as a by-product of NCML, as it is the derivative of the log of the learned quantity.
Revised: However, ∇ x log p t ( x ) is still learned as a by-product of NCML, as it is the derivative of the log of the learned quantity.

Original: We may then access our approximated score via standard backpropagation techniques.
Revised: We may then access our approximated score via standard backpropagation techniques.

Original: Therefore, like diffusion models, we can draw samples via Langevin dynamics.
Revised: Therefore, like diffusion models, we can draw samples via Langevin dynamics.

Original: This provides an alternative strategy for sampling from p θ,t , which we explore in 4.2.
Revised: This provides an alternative strategy for sampling from p θ,t , which we explore in 4.2.

Original: Second, we need not design our diffusion so that p T approximates the limiting stationary distribution of the process.
Revised: Second, we need not design our diffusion so that p T approximates the limiting stationary distribution of the process.

Original: This is necessary in diffusion models as the limiting prior is the only tractableThough a it distribution to initialize the sampling algorithm with.
Revised: This is necessary in diffusion models as the limiting prior is the only tractable distribution to initialize the sampling algorithm with.

Original: Since we have learned the density itself for all t ∈ T , we may initialize from any point of the diffusion, which increases the flexibility of the sampling strategy, and can drastically reduce the steps required to solve the reverse diffusion.
Revised: Since we have learned the density itself for all t ∈ T , we may initialize from any point of the diffusion, which increases the flexibility of the sampling strategy, and can drastically reduce the steps required to solve the reverse diffusion.

Original: For our models, we consider the three diffusion processes proposed in Song et al.
Revised: For our models, we consider the three diffusion processes proposed in Song et al.

Original: (2020): variance exploding (VE), variance preserving (VP), or sub-variance preserving (sub-VP), and choose µ to be the uniform distribution over T .
Revised: (2020b): variance exploding (VE), variance preserving (VP), or sub-variance preserving (sub-VP), and choose µ to be the uniform distribution over T .

Original: Due to space constraints, we refer to the aforementioned paper for more details on these SDEs.
Revised: Due to space constraints, we refer to the aforementioned paper for more details on these SDEs.

Original: 4. N OISE - ROBUSTNESS OF NCML M ODELS
Revised: 4. N OISE - ROBUSTNESS OF NCML M ODELS

Original: We find that models trained via NCML are more robust to noise.
Revised: We find that models trained via NCML are more robust to noise.

Original: Surprisingly, this is not only the case for noise conditions t > 0 , where the model is exposed to noisy samples.
Revised: Surprisingly, this is not only the case for noise conditions t > 0 , where the model is exposed to noisy samples.

Original: Indeed, even in the t = 0 condition, noise-robustness is improved (Table 1), and our models surpass state-ofthe-art likelihood models on minimally perturbed data (as defined in Eq 5) in terms of average log-likelihood.
Revised: Indeed, even in the t = 0 condition, noise-robustness is improved (Table 1), and our models surpass state-ofthe-art likelihood models on minimally perturbed data (as defined in Eq 5) in terms of average log-likelihood.

Original: This is somewhat unexpected: by passing the noise condition, the model should theoretically be able to separate the NCML loss into distinct problems at each noise scale.
Revised: This is somewhat unexpected: by passing the noise condition, the model should theoretically be able to separate the NCML loss into distinct problems at each noise scale.

Original: If this occurs, then the t = 0 case should simplify to a vanilla MLE problem, where behavior would not differ from standard likelihood models.
Revised: If this occurs, then the t = 0 case should simplify to a vanilla MLE problem, where behavior would not differ from standard likelihood models.

Original: Yet this is not the case.
Revised: Yet this is not the case.

Original: We believe that this lends credence to the regularization perspective provided in 4.
Revised: We believe that this lends credence to the regularization perspective provided in 4.

Original: In essence, noise robustness is explicitly enforced for t > 0 .
Revised: In essence, noise robustness is explicitly enforced for t > 0 .

Original: For t = 0 , NCML leverages the limited capacity of the underlying network to implicitly impose robustness.
Revised: For t = 0 , NCML leverages the limited capacity of the underlying network to implicitly impose robustness.

Original: To quantify the noise-robustness at each t , we define the simple measure ∆ log p as the absolute difference between the negative model log likelihood (as measured in bits per dimension) evaluated on p data minus that on p π , i.e. ,
Revised: To quantify the noise-robustness at each t , we define the simple measure ∆ log p as the absolute difference between the negative model log likelihood (as measured in bits per dimension) evaluated on p data minus that on p π , i.e. ,

Original: [Equation]
Revised: [Equation]

Original: The left and middle graphs in Fig 4 show ∆ log p as a function of t for π = 0 .
Revised: The left and middle graphs in Fig 4 show ∆ log p as a function of t for π = 0 .

Original: 5 and π = 1 , respectively.
Revised: 5 and π = 1 , respectively.

Original: Here, we can clearly see that noise robustness increases with increasing t .
Revised: Here, we can clearly see that noise robustness increases with increasing t .

Original: Moreover, the regularization effect of NCML enforces greater robustness than competing models even at t = 0 , as seen by the dotted line showing the next lowest ∆ log p , indicating that NCML enforces some degree of noise-robustness as regularization.
Revised: Moreover, the regularization effect of NCML enforces greater robustness than competing models even at t = 0 , as seen by the dotted line showing the next lowest ∆ log p , indicating that NCML enforces some degree of noise-robustness as regularization.

Original: As one may expect, the correlation between noise-robustness and t follows closely to the noise schedules of VP, sub-VP, and VE SDEs in [0 , T ] respectively.
Revised: As one may expect, the correlation between noise-robustness and t follows closely to the noise schedules of VP, sub-VP, and VE SDEs in [0 , T ] respectively.

Original: This can be seen in the rightmost plot of Figure 4, which shows the average absolute perturbation per pixel of each SDE over time.
Revised: This can be seen in the rightmost plot of Figure 4, which shows the average absolute perturbation per pixel of each SDE over time.

Original: The dotted line represents 1 , i.e. , the absolute perturbation of corrupted images in our sanity test p π .
Revised: The dotted line represents 1 , i.e. , the absolute perturbation of corrupted images in our sanity test p π .

Original: The point at which each model attains robustness to p π corrupted noise is more or less the same time the noise schedule begins to perturb each pixel by at least one bit, on average.
Revised: The point at which each model attains robustness to p π corrupted noise is more or less the same time the noise schedule begins to perturb each pixel by at least one bit, on average.

Original: The increased noise-robustness of NCML-trained models at larger t motivates our improved autoregressive sampling algorithm, which we introduce below.
Revised: The increased noise-robustness of NCML-trained models at larger t motivates our improved autoregressive sampling algorithm, which we introduce below.

Original: 4. S AMPLING WITH A UTOREGRESSIVE NCML M ODELS
Revised: 4. S AMPLING WITH A UTOREGRESSIVE NCML M ODELS

Original: Our framework allows for two sampling strategies.
Revised: Our framework allows for two sampling strategies.

Original: The first is to draw directly from the noise-free distribution p θ , 0 , in which case the conditional likelihood simplifies to a standard (unconditional) likelihood, and sampling is identical to that for a vanilla autoregressive model.[Table] PixelCNN+ [Table]
Revised: The first is to draw directly from the noise-free distribution p θ , 0 , in which case the conditional likelihood simplifies to a standard (unconditional) likelihood, and sampling is identical to that for a vanilla autoregressive model.

Original: PixelSNAIL

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: NCPN

Original: [Table]

Original: NCML

Original: [Table]

Original: [Table]

Original: NCPN (NCML

Original: [Table]

Original: VP)

Original: [Table]

Original: π = 0 is equivalent to NLL of the original data.

Original: However, as discussed in Section 3, this strategy is unstable and tends to quickly accumulate errors.
Revised: However, as discussed in Section 3, this strategy is unstable and tends to quickly accumulate errors.

Original: This motivates an alternative two-part sampling strategy, which involves drawing from p θ,t for t >(the noisy sampling phase), then solving a reverse diffusion process back to t = 

Revised: This motivates an alternative two-part sampling strategy, which involves drawing from p θ,t for t >(the noisy sampling phase), then solving a reverse diffusion process back to t =

Original: (the score-based refinement phase).
Revised: (the score-based refinement phase).

Original: The tractability of the latter is due to the fact that NCML-trained models learn the score as a byproduct of likelihood estimation.
Revised: The tractability of the latter is due to the fact that NCML-trained models learn the score as a byproduct of likelihood estimation.

Original: This is identical to the sampling procedure in scorebased diffusion models Song et al.
Revised: This is identical to the sampling procedure in score-based diffusion models Song et al.

Original: (2020), except for the key difference that we need not initialize with the prior distribution, as we can sample from any t ∈ T . 
Revised: (2020b), except for the key difference that we need not initialize with the prior distribution, as we can sample from any t ∈ T . [Table]

Revised: π = 0 is equivalent to NLL of the original data.

Original: This two-part strategy has several benefits.
Revised: This two-part strategy has several benefits.

Original: First, we saw in Section 4.1 that NCML-trained models are more robust to noise at higher t , which improves stability during sampling.
Revised: First, we saw in Section 4.1 that NCML-trained models are more robust to noise at higher t , which improves stability during sampling.

Original: This is reflected in the improved FID of NCML-trained NCPNs compared to ML- ( i.e. , maximum likelihood-) trained
Revised: This is reflected in the improved FID of NCML-trained NCPNs compared to ML- ( i.e. , maximum likelihood-) trained

Original: NCPNs in Table 1.
Revised: NCPNs in Table 1.

Original: Second, the refinement phase allows the model to make fine-tuned adjustments to the sample, which can further improve quality.
Revised: Second, the refinement phase allows the model to make fine-tuned adjustments to the sample, which can further improve quality.

Original: This is not possible in standard autoregressive models by construction.
Revised: This is not possible in standard autoregressive models by construction.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We demonstrate that incorporating noise in the maximum likelihood framework provides significant improvements in terms ofboth density estimation and sample generation.
Revised: We demonstrate that incorporating noise in the maximum likelihood framework provides significant improvements in terms of density estimation, sample generation, and anomaly detection.

Original: In all experiments, we fix p t to be one of the variance exploding (VE), variance preserving (VP), or sub-variance preserving (sub-VP) SDEs, and µ to be the uniform distribution over t ∈ T .
Revised: In all experiments, we fix p t to be one of the variance exploding (VE), variance preserving (VP), or sub-variance preserving (sub-VP) SDEs, and µ to be the uniform distribution over t ∈ T .

Original: For our architecture, we introduce the noise conditional pixel network (NCPN), which consists of a PixelCNN++ Salimans et al.
Revised: For our architecture, we introduce the noise conditional pixel network (NCPN), which consists of a PixelCNN++ Salimans et al.

Original: (2017) backbone with added attention layers.
Revised: (2017) backbone with added attention layers.

Revised: More experimental details can be found in A.1.

Original: Unconditional Modeling We evaluate our models on minimally perturbed transformations (see
Revised: Unconditional Modeling We evaluate our models on minimally perturbed transformations (see

Original: Section 3.1) of unconditional CIFAR-10 and ImageNet 64x64 for π ∈ { 0 , 12 , 1 } , where we note that p π = p data when π = 0 .
Revised: Section 3.1) of unconditional CIFAR-10 and ImageNet 64x64 for π ∈ { 0 , 12 , 1 } , where we note that p π = p data when π = 0 .

Original: All noise conditional models, i.e. , ours, VDM Kingma et al.
Revised: All noise conditional models, i.e. , ours, VDM Kingma et al.

Original: (2021), and ScoreFlow Song et al.
Revised: (2021), and ScoreFlow Song et al.

Original: (2021), are evaluated at t = 0 .
Revised: (2021), are evaluated at t = 0 .

Original: We show our results in Table 1.
Revised: We show our results in Table 1.

Original: We additionally evaluate our model on the CelebA 64x64 dataset; autoregressive comparisons on this dataset are limited, so we defer results to the appendix.
Revised: We additionally evaluate our model on the CelebA 64x64 dataset; autoregressive comparisons on this dataset are limited, so we defer results to the appendix.

Original: On the standard unperturbed dataset π = 0 , our models attain competitive likelihoods on CIFARand state-of-the-art likelihoods on ImageNet 64x64.
Revised: On the standard unperturbed dataset π = 0 , our models attain competitive likelihoods on CIFARand state-of-the-art likelihoods on ImageNet 64x64.

Original: On all perturbed datasets, our models achieve state-of-the-art likelihoods.
Revised: On all perturbed datasets, our models achieve state-of-the-art likelihoods.

Original: Furthermore, we significantly improve on the state-of-the-art in terms of sample quality among likelihood maximization models on CIFAR-10, from 37.50 to 12.09 in terms of FID.
Revised: Furthermore, we significantly improve on the state-of-the-art in terms of sample quality among MLE models on CIFAR-10, from 37.50 to 12.09 in terms of FID.

Original: In general, our results indicate that the NCML framework improves generative modeling performance of the underlying models in terms of both test log-likelihoods and sample quality.
Revised: In general, our results indicate that the NCML framework improves generative modeling performance of the underlying models in terms of both test log-likelihoods and sample quality.

Original: Class-conditional Modeling We find that NCML-trained models exhibit stable sampling even in class-conditional generation.
Revised: Class-conditional Modeling We find that NCML-trained models exhibit stable sampling even in class-conditional generation.

Original: For this experiment, we train a class-conditional model under our framework on the CIFAR-10 dataset.
Revised: For this experiment, we train a class-conditional model under our framework on the CIFAR-10 dataset.

Original: We show our results in Figure 5, where each row shows data sampled from a different class of CIFAR-10.
Revised: We show our results in Figure 5, where each row shows data sampled from a different class of CIFAR-10.

Original: Image Completion We further examine NCML-trained models in a controllable generation context through the image completion task, which involves conditioning an autoregressive model on the first half of an image, and drawing the second half from the modeled conditional distribution.
Revised: Image Completion We further examine NCML-trained models in a controllable generation context through the image completion task, which involves conditioning an autoregressive model on the first half of an image, and drawing the second half from the modeled conditional distribution.

Original: We again use the CIFAR-10 dataset, and compare models trained under our framework against those trained with MLE.
Revised: We again use the CIFAR-10 dataset, and compare models trained under our framework against those trained with MLE.

Original: All images are taken from the test set to minimize data memorization.
Revised: All images are taken from the test set to minimize data memorization.

Original: Our results are in figure 5 (right).
Revised: Our results are in figure 5 (right).

Original: The leftmost and rightmost four columns are outputs generated by an MLE-trained model and an NCML-trained model, respectively, with the middle column depicting the masked input to the sampling algorithm.
Revised: The leftmost and rightmost four columns are outputs generated by an MLE-trained model and an NCML-trained model, respectively, with the middle column depicting the masked input to the sampling algorithm.

Original: Both models use the same architecture.
Revised: Both models use the same architecture.

Original: Our model demonstrates improved stability over the course of sampling, and produces completed images with greater realism and fidelity, while maintaining a high diversity of sampled trajectories. 
Revised: Our model demonstrates improved stability over the course of sampling, and produces completed images with greater realism and fidelity, while maintaining a high diversity of sampled trajectories. [Figure]

Revised: Out-of-distribution (OOD) Detection We show that robustness to minimally perturbed data confers benefits that extend beyond enhanced sample quality, including marked improvements in anomaly detection.

Revised: In this task, the network is trained on in-distribution points, then tasked to produce a statistic that successfully differentiates in-distribution points from out-of-distribution points.

Original: C ONCLUSION AND F URTHER W ORK
Revised: Following Du & Mordatch (2019); Meng et al. (2020), we train models on CIFAR-10, and let the OOD points be SVHN Netzer et al.

Revised: (2011), constant uniform, and random uniform images.

Revised: We compare the Area

Revised: Under the Receiving Operator Curve (AUROC) of our model statistic to vanilla PixelCNN++ Salimans et al.

Revised: (2017), GLOW Kingma & Dhariwal (2018), EBM Du & Mordatch (2019), and AR-CSM

Revised: Meng et al.

Revised: (2020) models, and see that our model performs equally or better in every respect (Table 2).

Revised:  C ONCLUSION AND F URTHER W ORK

Original: We proposed a simple sanity test for checking the robustness of likelihoods to visually imperceptible levels of noise, and found that most models are highly sensitive to perturbations under this test.
Revised: We proposed a simple sanity test for checking the robustness of likelihoods to visually imperceptible levels of noise, and found that most models are highly sensitive to perturbations under this test.

Original: We argue that this is further evidence of a fundamental disconnect between likelihoods and other sample quality metrics.
Revised: We argue that this is further evidence of a fundamental disconnect between likelihoods and other sample quality metrics.

Original: To alleviate this issue, we developed a novel framework for training likelihood models that combines autoregressive and diffusion models in a principled manner.
Revised: To alleviate this issue, we developed a novel framework for training likelihood models that combines autoregressive and diffusion models in a principled manner.

Original: Finally, we find that models trained under this setting have substantial improvements in both training and sampling.
Revised: Finally, we find that models trained under this setting have substantial improvements in both training and evaluation.

Original: While models trained under the NCML framework show greater invariance to imperceptible noise, they are by no means robust, indicating that the underlying model still differs significantly from the theoretical human model q human proposed in Husz´ar (2015).
Revised: While models trained under the NCML framework show greater invariance to imperceptible noise, they are by no means robust, indicating that the underlying model still differs significantly from the theoretical human model q human proposed in Husz´ar (2015).

Original: We hope that further research can help close this gap, and furnish us with a more intuitive grasp on the maximum likelihood as a framework for assessing goodness-of-fit in generative models.
Revised: We hope that further research can help close this gap, and furnish us with a more intuitive grasp on the maximum likelihood as a framework for assessing goodness-of-fit in generative models A A PPENDIX A. A DDITIONAL E XPERIMENTAL D ETAILS

Revised: Our proposed NCPN architecture consists of the PixelCNN++ backbone Salimans et al.

Revised: (2017) with axial attention layers Ho et al.

Revised: (2019b)

Revised: after each residual block.

Revised: We retain the hyperparameters of PixelCNN++, changing only the dropout on the CIFAR-10 dataset (from 0.5 to 0.25), which we reduced due to the regularization properties of NCML.

Revised: For the axial attention layers, we useheads and skip connection rescaling as in Song et al.

Revised: (2020b).

Revised: Finally, we add noise conditioning to each residual block via a Gaussian Fourier Projection layer, much like Ho et al.

Revised: ; Song et al.

Revised: (2020b).

Revised: For our NCML-trained models, the diffusion times of the VE, VP, and sub-VP SDEs were chosen to be T = 0 .

Revised: 5 , T = 0 .

Revised: 1 , and T = 0 .

Revised: 025 , respectively.

Revised: The values are selected such that the standard deviation of the per-pixel differences between samples in p data and their noised counterparts in p T was ≈ 10 bits.

Revised: We suspect that further improvements can be made to the empirical results if these numbers were chosen more judiciously.

Revised: All NCPN models were trained on RTX 2080 Ti GPUs for 500,000 iterations.

Revised: This is approximately 1.5 weeks of training.

Revised: We use the same NCPN architecture and hyperparameters across all datasets (except for dropout, which is set to 0.25 on CIFAR-10 and 0.00 on ImageNet 64x64 and CelebA 64x64).

Revised: All NCPN models have 73M parameters.

Revised: A.1. D ENSITY E STIMATION AND G ENERATIVE M ODELING E XPERIMENTS

Revised: For experiments on CIFAR-10 and ImageNet 64x64, we compare against Kingma et al.

Revised: Song et al.

Revised: ; Child (2020); Ho et al. (2019a); Grci ´ c et al.

Revised: ; Salimans et al.

Revised: ; Chen et al.

Revised: ; Child et al.

Revised: Some results could not be included due to the irreproducibility of the techniques.

Revised: There is limited existing work on likelihood-based modeling on CelebA 64x64, so we do not provide comparisons, however the performance of our model is summarized in Table 3.

Revised: A.1. O UT - OF - DISTRIBUTION D ETECTION E XPERIMENTS

Revised: We directly use the CIFAR-10 models trained in A.1.1 and thus retain all hyperparameters from the previous experiment.

Revised: All NCPN are evaluated with the time condition t = 0 .

Revised: Judicious choice of the statistic is important for the performance of the model.

Revised: For example, Du & Mordatch (2019) use the unnormalized energy function log( Z · p ( x )) where Z = (cid:82) exp f ( x ) is the partition function, and

Revised: Meng et al.

Revised: (2020) use (cid:80) di

Revised: =1 [ ∇ log p ( x )]

Revised: i , where p ( x ) i denotes the i th coordinate of the score.

Original: {+We use ||∇ p ( x ) || 
+}
Revised: {+We use ||∇ p ( x ) || 
+}

Revised: = | p ( x ) | ∗ ||∇ log p ( x ) || 2 .

Revised: Additionally, we show that performance on the OOD detection task is closely associated with robustness to minimally perturbed data (Table 4), in the sense that improved robustness to p π is associated with improved performance on OOD detection.

Revised: One possible explanation for this correlation is that robustness to visually inconsequential high frequency content forces the model to rely on more intrinsic features of the image to assign probabilities, which allows the model to better classify OOD images.

Revised: A. A DDITIONAL F IGURES

Revised: [Table]