Score-Based Generative Classiﬁers
Abstract
The tremendous success of generative models in recent years raises the question whether they can also be used to perform classiﬁcation.
Generative models have been used as adversarially robust classiﬁers on simple datasets such as MNIST, but this robustness has not been observed on more complex datasets like CIFAR-10.
Additionally, on natural image datasets, previous results have suggested a trade-off between the likelihood of the data and classiﬁcation accuracy.
In this work, we investigate score-based generative models as classiﬁers for natural images.
We show that these models not only obtain competitive likelihood values but simultaneously achieve state-of-the-art classiﬁcation accuracy for generative classiﬁers on CIFAR10.
Nevertheless, we ﬁnd that these models are only slightly, if at all, more robust than discriminative baseline models on out-of-distribution tasks based on common image corruptions.
Similarly and contrary to prior results, we ﬁnd that score-based are prone to worst-case distribution shifts in the form of adversarial perturbations.
Our work highlights that score-based generative models are closing the gap in classiﬁcation accuracy compared to standard discriminative models.
While they do not yet deliver on the promise of adversarial and out-of-domain robustness, they provide a different approach to classiﬁcation that warrants further research.
Introduction
[Figure]
There exist two fundamentally distinct ways of performing classiﬁcation.
The standard way is to train a classiﬁer which discriminates between classes by modeling the conditional probability p p y | x q of labels y given some input x .
These models are called discriminative classiﬁers and have been extremely successful in supervised learning [24, 22].
The alternative is to model the conditional likelihood p p x | y q of an image for each label and predict the label that maximizes this likelihood [39].
This approach is much less common because it requires solving the more complicated task of learning conditional generative models of the data; these models are referred to as generative classiﬁers
A line of research has been discussing the advantages and disadvantages between using discriminative or generative models for classiﬁcation [39, 33, 36, 2, 26].
They found generative models to be more data efﬁcient, but discriminative models to be asymptotically better.
However, these analyses were mostly limited to shallow models [but see 12] and performance evaluation within the same data domain.
On MNIST [24] and SVHN [32], previous results have demonstrated that generative classiﬁers are robust even to worst-case distribution shifts in the form of adversarial perturbations [40, 17, 35].
Furthermore, it was found that generative classiﬁers have the highest alignment with humans on controversial stimuli [11].
However, so far, these results have not transferred to any natural datasets of higher visual complexity [25, 8, 1, 27].
Fetaya et al.
[8] argue that ‘obtaining strong classiﬁcation accuracy without harming likelihood estimation is still a challenging problem’.
This is empirically supported in their paper as well as a number of related works [13, 1, 27] who all demonstrate a trade-off between good likelihoods and high classiﬁcation accuracy, suggesting that to better capture what is unique for a label, some performance of the generative model needs to be sacriﬁced [8].
We challenge these ideas and add a new datapoint to the ongoing debate around generative classiﬁers demonstrating that near state-of-the-art likelihoods can be combined with classiﬁcation accuracy at the level of discriminative models [45].
More speciﬁcally, score-based generative models have been proposed in recent years as a promising approach towards modeling complex high-dimensional data distributions [41, 16, 42].
They are built on the idea of gradually diffusing a complex data distribution with a parameterized stochastic differential equation into a tractable noise distribution.
Previous work has suggested the effectiveness of these diffusion-based models for out-of-domain (OOD) tasks such as adversarially robust classiﬁcation [51] and as a general prior for natural images [48, 19].
In this work, we propose score-based generative classiﬁers (SBGC) and show that while these models overcome previous limitations — such as trading off likelihood for accuracy — they still do not solve the problem of worst-case distribution shifts on natural images.
More precisely, using our SBGC model, we report state-of-the-art classiﬁcation accuracy for generative models on CIFAR-10 [22], along with highly competitive likelihood values.
Nonetheless, we ﬁnd that score-based generative classiﬁers show only slight improvements, if at all, in OOD benchmarks and are vulnerable to adversarial examples.
Methods 2.1 Score-Based Generative Models
This work builds on recent advances in score-based generative models of natural images [41, 16, 42, 43, 20].
The general idea behind this approach is to deﬁne a stochastic differential equation (SDE) describing the temporal transformation of the complex data distribution ( p ) into a tractable noise distribution ( p T , isotropic Gaussian).
This dynamical process can then be reversed by learning a time-dependent score function f θ p x p t q , t q modeled by a neural network with parameters θ .
Intuitively, this score function acts as an inﬁnitesimal denoiser [19].
To generate new samples from the data distribution, one starts with random noise x p T q and follows the dynamics of the reverse SDE to produce a sample x p 0 q from the data distribution [see 42, for more details].
Moreover, by removing the stochasticity, the SDE becomes a (neural) ordinary differential equation [42, 5]. Using a continuous time analog of the change of variables formula, one can compute the likelihood ( p 0 ) of an input image x p 0 q under the model [42]:
[Equation]
We use the training objective suggested in Song et al. [43] which maximizes the likelihood of the training data under the model. This is equivalent to minimizing the Kullback-Leibler divergence (KL) between the model distribution and the data distribution, i.e., min θ KL p p [-
-]0 | p q [29].
2.2 Score-Based Generative Classiﬁers
The most straightforward application of score-based generative models as classiﬁers would be to train one model per class as in Schott et al.
However, this produces very unstable training [Table] and considerable overﬁtting because the dataset size is, for each model, effectively reduced by one order of magnitude.
Instead, we add the image label y P t 1 , ..., 10 u as a conditioning variable to the score function f θ p x p t q , t, y q ; note that this also reduces the required amount of memory for multiple models. In terms of the network architecture
[Table]
we follow concurrent work of Nichol and Dhariwal [34] and integrate this additional input by ﬁrst converting it into a one-hot vector and then adding it to the model’s representation in the same way as the time variable [see 42].
For classiﬁcation of a given input x p 0 q , we can then condition the score function on each label y {+log p 
+}
p x p 0 q| y q “ log p T ˜ x p 0 q ` ż
[Equation]
0 f [Equation] d t ¸ ` ż T 0 ∇
[Equation]
f
[Equation]
p
[Equation]
{+(2) and predict the label that yields the highest conditional likelihood max y log p 
+}
p x p 0 q| y q .
To get the unconditional likelihood of samples we marginalize over the classes, i.e. p p x q “ ř y i p p x | y “ y i q .
Results 3.1 Classiﬁcation Accuracy and Data Likelihood
First of all, we note that previous methods [8, 1, 27] have discussed a trade-off between classiﬁcation accuracy and likelihoods (see Figure 1).
This is further supported by observations on MNIST, where increasing the latent dimensionality of a variational autoencoder increases the likelihood at the cost of decreasing classiﬁcation accuracy and robustness [6].
Older work has focused on hybrid models that combine the data efﬁciency of (shallow) generative models with the asymptotic performance of discriminative models. However, this comes with a cost in model likelihood [10, 36].
In contrast to these previous methods, we ﬁnd that our model achieves both state-of-the-art accuracy and likelihoods for generative classiﬁers on CIFAR-10 (Table 1).
We also test the deeper architecture proposed in Song et al.
[43]: While this does not change the classiﬁcation accuracy, it slightly increases the NLL to 3 .
08 bits/dim.
We leave further exploration of this direction to future research.
3.2 Out-of-Distribution Robustness: Common Image Corruptions
Most previous approaches towards improving out-of-distribution performance on common image corruptions [14] have relied on some form of data augmentation in the form of carefully hand-crafted transformations [15] or adversarial training [4].
Here, we propose an orthogonal approach that is based on different modeling assumptions.
Speciﬁcally, we are interested in seeing whether the inductive bias implicit in score-based generative classiﬁers improves their classiﬁcation accuracy when generalizing to the image domains of CIFAR-10-C [14].
We ﬁnd that our model performs better than some previous models that are solely trained on the original data (Table 2).
Since score-based models are trained like denoisers, they might work speciﬁcally well on noisy data.
Thus, just comparing the mean accuracy over all corruption types, could give them an unfair advantage over the baselines.
Hence, we also compare the mean accuracy for all corruptions excluding [Table] noises 2 (Table 2).
Here, we see that our SBGC model does not perform better than the baselines.
We conclude that the improvements of the SBGC model in terms of out-of-distribution robustness are limited to noise corruptions.
When integrating data augmentation methods [Table] discriminative models achieve higher performance (Table 2, top).
However [Table] we note that the approach of augmenting datasets is independent of our suggested changes, with the caveat that changing the dataset needs to be done carefully for generative models [18].
Preliminary experiments indicate that we can increase our performance on CIFAR-10-C further by leveraging more data augmentations [15] (accuracy „ 83% ).
However [Table] there is still a large gap towards discriminative models with data augmentation.
3.3 Out-of-Distribution Robustness: Adversarial Perturbations
Despite their small magnitude [Table] adversarial perturbations can cause a drastic change in the behavior of a neural network [44] and can be seen as a worst-case distribution shift.
To assess the robustness of our SBGC model against such perturbations, we ﬁrst generate them using a projected gradient descent (PGD) [28] on the negative cross-entropy of the model’s prediction using Foolbox [37].
For this, we estimate the gradients of the model likelihoods with respect to the inputs d p 0 ,y p x q{ d x using the adjoint sensitivity method [5].
This requires second-order optimization because of the Jacobian term in equation 2.
We implement this efﬁciently with Hessian-vector-products in JAX [3] and verify the implementation by comparing the calculated gradient with numerical estimations based on ﬁnite differences.
Since computing gradients is still computationally expensive (a single forward and backward pass take approximately 90 minutes on an NVIDIA Tesla V100 GPU), we follow common practice in this domain and perform the adversarial evaluation on a reduced sample size of N “ 216 .
While our approach partially improves robustness against common image corruptions, it strikingly fails in the presence of adversarial perturbations (Tab. 3).
Within both standard (cid:96) 8 perturbation norms ( 8 { 255 ) as well as standard (cid:96) 2 perturbation norms ( 0 . 5 ), our model miss-classiﬁes every adversarially perturbed input image. [Table] [Table] [Figure]
i
[Figure]
d [Figure] 3.4 Likelihood Estimation on Interpolated Samples In a previous study, Fetaya et al.
[8] highlighted a problem with generative classiﬁers.
They found that the likelihood of an interpolated image on CIFAR-10 is actually higher than the likelihoods of both the start x A and endpoint x B of the interpolation.
More precisely, they demonstrated that the function
[Equation]
is concave, which is counter-intuitive because a generative model should assign low likelihood to inputs outside of its training distribution such as nonsensical interpolations in pixel space.
To test this for our SBGC model, we measure the function f p t q for 24 values for t P r 0 , 1 s and average this over 500 image pairs.
As depicted in Figure 2, we ﬁnd that the interpolation of the likelihood is generally convex.
Thus, in contrast to the analysis of previous models by [8], our SBGC model assigns lower likelihood values to interpolated images that are outside the (training) data distribution.
Conclusion
We have shown that the latest advances in score-based generative modeling of natural images translate into generative classiﬁers which have highly competitive classiﬁcation accuracies as well as likelihoods.
However, in the search for further beneﬁts, we found that these models show only minor or no improvements on common image corruptions.
Similarly, but in contrast to previous results [51], they spectacularly fail on worst-case distribution shifts.
This highlights the necessity to perform extensive and, if at all possible, gradient-based adversarial attacks.
The lack of adversarial robustness aligns with previous results [21, 25, 1, 27].
Interestingly, this vulnerability remains despite resolving the problem of high likelihood on interpolated images [8], and so we can exclude this as a possible explanation for the lacking robustness.
Thus, we are left wondering why generative classiﬁcation of natural images does not show the same robustness to worst-case distribution shifts as observed on MNIST [40].
A possible explanation might be that the positive results on MNIST were due to the fact that Euclidean distances (on which the likelihood of variational autoencoders with a Gaussian likelihood is based) are a useful metric on
MNIST but not on real images.
This also led to the provable robustness on MNIST [40].
However, on natural images Euclidean distances are badly aligned with human perception [46, 50, 23] and we have no guarantee about the model behavior even in small (cid:96) p epsilon balls around the training data.
In a nutshell, while we achieve almost the same accuracy on clean data as a baseline discriminative classiﬁer and show high data likelihoods, we ﬁnd no advantage over discriminative classiﬁers regarding worst-case distribution shifts and common corruptions.
A Appendix A.1 Training and Implementation Details
We build on the architecture (baseline model with 4 residual blocks) proposed by Song et al.
[43] and extend it to condition the score function also on the class label.
All in all, our model has 67 .
19 M parameters.
Furthermore, we use the same SDE (variance preserving) and maximum likelihood training procedure (trained for 950 , 000 steps with a batch size of 128 and importance sampling) as
Song et al.
The training takes approximately ﬁve days on two NVIDIA Tesla V100 GPUs.
For the likelihood inference, i.e., for solving the SDE, we use the Runge-Kutta 4(5) solver of SciPy ( scipy.integrate.solve_ivp ) [49] with absolute and relative precision equal to 10 ´ 5 .
We perform uniform dequantization (i.e., adding uniform noise with a magnitude of 1 { 256 ) and integrate the SDE up to a minimal (cid:15) “ 10 ´ 5 for numerical stability [42].
Furthermore, the ﬁnal likelihood is computed by taking the average over n “ 30 random draws for the epsilon trace estimator [42].
For the OOD tasks (CIFAR-10-C and adversarial perturbations), we lower this to n “ 10 to reduce the compute time, thus giving a more conservative lower estimate of the performance on these datasets (see also section A.2.1).
For the WideResNet-28-12 baseline we use an existing implementation 3 in PyTorch.
To make this baseline more comparable with our SBGC, we also use dequantization noise for augmenting the training data of the WideResNet.
A.2 Additional Analyses A.2.1 Convergence Analysis
Here, we study the effect of varying the number of repetitions of the trace estimator (see previous section).
In Figure.
3 on the left we see that the classiﬁcation accuracy improves monotonically with the number of repetitions.
A number of n “ 10 gives nearly asymptotic performance and is, thus, a good trade-off for our OOD tasks which are computationally intensive.
For asymptotic performance, we see that the accuracy rises above 95% .
On the right side of Figure.
3, we study the convergence of the trace estimator which is part of the likelihood (equation 2.
We can see that the standard error of the mean (errorbars) converges very slowly.
We therefore report the likelihood at n “ 30 repetitions.
Note also that the likelihood of the correct class converges (averaged across all test images) to a higher value as that of the wrong class with the highest likelihood.
This is expected for a model with good classiﬁcation accuracy.
Thus, provided labels, our model attains slightly higher likelihoods than in the unconditioned case.
Cl. 1 2 3
L o g - L i k e li
h oo d
Gaussian Noise 0.
Cl. 1 2 3
Shot Noise 0.
Cl. 1 2 3
Impulse Noise 0.
Cl. 1 2 3 43.
Defocus Blur 0.
Cl. 1 2 3 44.
3.0 Frosted Glass Blur 0.
A cc u r a c y
Cl. 1 2 3 43.
L o g - L i k e li
h oo d
Motion Blur 0.
Cl. 1 2 3 43.
Zoom Blur 0.
Cl. 1 2 3 44.
Snow 0.
Cl. 1 2 3 44.
Frost 0.
Cl. 1 2 3 43.
Fog 0.
A cc u r a c y
Cl.
1 2 3 4 5 Corruption Severity
L o g - L i k e li
h oo d
Brightness 0.
Cl.
1 2 3 4 5 Corruption Severity
Contrast 0.
Cl. 1 2 3 [-
 [Figure]-]4 5 Corruption Severity 3.
Elastic 0.
Cl.
1 2 3 4 5 Corruption Severity 4.
Pixelate 0.
Cl.
1 2 3 4 5 Corruption Severity 3.
2.5 JPEG Compression 0.
A cc u r a c y
Here, we further resolve the model performance on CIFAR-10-C for all different corruption types and strengths in Figure 4. Interestingly, the results highlight that likelihood and accuracy need not always be aligned. For the different corruptions based on blurring or some sort of low-pass ﬁltering (e.g., fog or JPEG compression), we observe that while the accuracy decreases (as signal is lost) the likelihood under the model actually increases.
This observation is in line with previous studies [31].
We hypothesize that this highlights the fact that natural images follow a 1 { f power spectrum [9] and, thus, low-pass ﬁltering will increase the likelihood of any image.
This also explains previous observations of non-natural images obtaining high likelihoods under generative models: Starting from a non-natural image we can always blur the image to obtain high likelihoods under our model (see, e.g., our Figure 4 zoom blur and Table 3 in [8]).
Similarly, we can start from a natural image (e.g., from the training set) and add white noise.
The high-frequency perturbations will quickly deteriorate the likelihoods (see Figure 4 Gaussian noise).
Relatedly, Nalisnick et al.
[30] found that a measure of dataset complexity can be leveraged in conjunction with likelihood estimates to perform OOD detection.