CARD: Classification and Regression Diffusion Models
Abstract
Learning the distribution of a continuous or categorical response variable y given its covariates x is a fundamental problem in statistics and machine learning.
Deep neural network-based supervised learning algorithms have made great progress in predicting the mean of y given x , but they are often criticized for their ability to accurately capture the uncertainty of their predictions.
In this paper, we introduce classification and regression diffusion (CARD) models, which combine a denoising diffusion-based conditional generative model and a pre-trained conditional mean estimator, to accurately predict the distribution of y given x .
We demonstrate the outstanding ability of CARD in conditional distribution prediction with both toy examples and real-world datasets, the experimental results on which show that
CARD in general outperforms state-of-the-art methods, including Bayesian neural network-based ones that are designed for uncertainty estimation, especially when the conditional distribution of y given x is multi-modal.
In addition, we utilize the stochastic nature of the generative model outputs to obtain a finer granularity in model confidence assessment at the instance level for classification tasks.
Our implementation is publicly available at https://github.com/XzwHan/CARD.
Introduction
A fundamental problem in statistics and machine learning is to predict the response variable y given a set of covariates x .
Generally speaking, y is a continuous variable for regression analysis and a categorical variable for classification.
Denote f ( x ) ∈ R C as a deterministic function that transforms x into a C dimensional output.
Denote f c ( x ) as the c -th dimension of f ( x ) .
Existing methods typically assume an additive noise model: for regression analysis with y ∈ R C , one often assumes y =
f ( x ) + ϵ , ϵ ∼ N (0 , Σ ) , while for classification with y ∈ { 1 , . .
, C } , one often assumes y = arg max (cid:0) f 1 ( x ) + ϵ 1 , . . .
, f C ( x ) + ϵ C (cid:1) , where ϵ c iid ∼ EV 1 (0 , 1) , a standard type-1 extreme value distribution.
Thus we have the expected value of y given x as E
[ y | x ] = f ( x ) in regression and P ( y = c | x ) =
E [ y = c | x ] = softmax c (cid:0) f ( x ) (cid:1) =
exp( f c ( x )) (cid:80) Cc ′ =
1 exp( f c ′ ( x )) in classification.
These additive-noise models are primarily focusing on accurately estimating the conditional mean
E [ y | x ] , while paying less attention to whether the noise distribution can accurately capture the uncertainty of y given x .
For this reason, they may not work well if the distribution of y given x clearly deviates from the additive-noise assumption.
For example, if p ( y | x ) is multi-modal, which commonly happens when there are missing categorical covariates in x , then E [ y | x ] may not be close to any possible true values of y given that specific x .
More specifically, consider a person whose weight, height, blood pressure, and age are known but gender is unknown, then the testosterone or estrogen level of this person is likely to follow a bi-modal distribution and the chance ∗ Equal contribution. of developing breast cancer is also likely to follow a bi-modal distribution.
Therefore, these widely used additive-noise models, which use a deterministic function f ( x ) to characterize the conditional mean of y , are inherently restrictive in their ability for uncertainty estimation.
In this paper, our goal is to accurately recover the full distribution of y conditioning on x given a set of N training data points, denoted as D = { ( x i , y i ) } 1 ,N .
To realize this goal, we consider the diffusion-based (a.k.a. score-based) generative models (Sohl-Dickstein et al., 2015; Song and Ermon, 2019;
Ho et al., 2020; Song and Ermon, 2020; Song et al., 2021c) and inject covariate-dependence into both the forward and reverse diffusion chains.
Our method can model the conditional distribution of both continuous and categorical y variables, and the algorithms developed under this method will be collectively referred to as C lassification A nd R egression D iffusion (CARD) models.
Diffusion-based generative models have received significant recent attention due to not only their ability to generate high-dimensional data, such as high-resolution photo-realistic images, but also their training stability.
They can be understood from the perspective of score matching (Hyvärinen and Dayan, 2005; Vincent, 2011; Kadkhodaie and Simoncelli, 2021) and Langevin dynamics (Neal, 2011; Welling and Teh, 2011), as pioneered by Song and Ermon (2019).
They can also be understood from the perspective of diffusion probabilistic models (Sohl-Dickstein et al., 2015; Ho et al., 2020), which first define a forward diffusion to transform the data into noise and then a reverse diffusion to regenerate the data from noise.
These previous methods mainly focus on unconditional generative modeling.
While there exist guided-diffusion models (Song and Ermon, 2019; Song et al., 2021c; Dhariwal and Nichol, 2021;
Nichol et al., 2022; Ramesh et al., 2022) that target on generating high-resolution photo-realistic images that match the semantic meanings or content of the label, text, or corrupted-images, we focus on studying diffusion-based conditional generative modeling at a more fundamental level.
In particular, our goal is to thoroughly investigate whether CARD can help accurately recover p ( y | x , D ) , the predictive distribution of y given x after observing data D
= { ( x i , y i ) } i =1 ,N .
In other words, our focus is on regression analysis of continuous or categorical response variables given their corresponding covariates.
We summarize our main contributions as follows: 1) We show CARD, which injects covariatedependence and a pre-trained conditional mean estimator into both the forward and reverse diffusion chains to construct a denoising diffusion probabilistic model, provides an accurate estimation of p ( y | x , D ) .
We provide a new metric to better evaluate how well a regression model captures the full distribution p ( y | x , D ) .
3) Experiments on standard benchmarks for regression analysis show that CARD achieves state-of-the-art results, using both existing metrics and the new one. 4) For classification tasks, we push the assessment of model confidence in its predictions towards the level of individual instances, a finer granularity than the previous methods.
Methods and Algorithms for CARD
Given the ground-truth response variable y 0 and its covariates x , and assuming a sequence of intermediate predictions y 1: T made by the diffusion model, the goal of supervised learning is to learn a model such that the log-likelihood is maximized by optimizing the following ELBO:
[Equation]
where q ( y 1: T | y 0 , x ) is called the forward process or diffusion process in the concept of diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020).
Denote D KL ( q || p ) as the Kullback–Leibler (KL) divergence from distribution p to distribution q .
The above objective can be rewritten as
[Equation]
Here we follow the convention to assume L T does not depend on any parameter and it will be close to zero by carefully diffusing the observed response variable y 0 towards a pre-assumed distribution p ( y T | x ) .
The remaining terms will make the model p θ ( y t − 1 | y t , x ) approximate the corresponding tractable ground-truth denoising transition step q ( y t − 1 | y t , y 0 , x ) for all timesteps.
Different from the vanilla diffusion models, we assume the endpoint of our diffusion process to be
[Equation]
) where f ϕ ( x ) is the prior knowledge of the relation between x and y 0 , e.g. , a network pre-trained with D to approximate E [ y | x ] , or 0 if we assume the relation is unknown.
With a diffusion schedule { β t } t =1: T ∈ (0 , 1) T , we specify the forward process conditional distributions in a similar fashion as Pandey et al.
(2022), but
for all timesteps including t = 1 :
[Equation]
which admits a closed-form sampling distribution with an arbitrary timestep t :
[Equation]
where α t := 1 − β t and ¯ α t := (cid:81) t α t .
Note that the mean term in Eq.
(7) can be viewed as an interpolation between true data y 0 and the predicted conditional expectation f ϕ ( x ) , which gradually changes from the former to the latter throughout the forward process.
Such formulation corresponds to a tractable forward process posterior:
[Equation]
where
[Equation]
We provide the derivation in Appendix A.1.
The labels under the terms are used in Algorithm 2.
2.1 CARD for Regression
For regression problems, the goal of the reverse diffusion process is to gradually recover the distribution of the noise term, the aleatoric or local uncertainty inherent in the observations (Kendall and
Gal, 2017; Wang and Zhou, 2020), enabling us to generate samples that match the true conditional p ( y | x ) .
Following the reparameterization introduced by denoising diffusion probabilistic models (DDPM)
(Ho et al., 2020), we construct ϵ θ (cid:0) x , y t , f ϕ ( x ) , t (cid:1) , which is a function approximator parameterized by a deep neural network that predicts the forward diffusion noise ϵ sampled for y t .
The training and inference procedure can be carried out in a standard DDPM manner.
[-1: Pre-train f  ( x ) that predicts E ( y x ) with MSE 2: repeat 3: Draw y 
 ⇠ q ( y 0 | x ) 4:-] Draw t ∼ Uniform ( { 1 . . .
T } ) 5: Draw ϵ ∼ N ( 0 , I ) 6: Compute noise estimation loss
[Equation]
Take numerical optimization step on: ∇ [Equation] L
[Equation]
8: until Convergence
1: y T ∼ N ( f ϕ ( x ) , I ) 2: for t =
T to 1 do 3:
Draw z ∼ N ( 0 , I ) if t >4:
Calculate reparameterized[-ˆ y 
-]
[Equation]
Let y t − 1 = γ
ˆ y [-
-]0 + γ 1 y t + γ 2 f ϕ x ) + (cid:113) ˜
β t z if t > 1 , else set y t −
= ˆ y6: end for 7: return y2.2 CARD for Classification
We formulate the classification tasks in a similar fashion as in Section 2.1, where we:
Replace the continuous response variable with a one-hot encoded label vector for y 0 ; 2.
Replace the mean estimator with a pre-trained classifier that outputs softmax probabilities of the class labels for f ϕ ( x ) .
This construction no longer assumes y 0 to be drawn from a categorical distribution, but instead treats each one-hot label as a class prototype, i.e. , we assume a continuous data and state space, which enables us to keep the Gaussian diffusion model framework.
The sampling procedure would output reconstructed y 0 in the range of real numbers for each dimension, instead of a vector in the probability simplex.
Denoting C as the number of classes and 1 C as a C -dimensional vector of 1 s, we convert such output to a probability vector in a softmax form of a temperature-weighted Brier score (Brier, 1950), which computes the squared error between the prediction and 1 C .
Mathematically, the probability of predicting the k th class and the final point prediction ˆ y can be expressed as
[Equation]
where τ > 0 is the temperature parameter, and ( y 0 − 1 C ) 2 k indicates the k th dimension of the vector of element-wise square error between y 0 and 1 C , i.e. , ( y 0 − 1 C ) 2 k = ∥
y [-
-]0 k − 1 ∥ 2 .
Intuitively, this construction would assign the class whose raw output in the sampled y 0 is closest to the true class, encoded by the value of 1 in the one-hot label, with the highest probability.
Conditional on the same covariates x , the stochasticity of the generative model would give us a different class prototype reconstruction after each reverse process sampling, which enables us to construct predicted probability intervals for all class labels.
Such stochastic reconstruction is in a similar fashion as DALL-E 2 (Ramesh et al., 2022) that applies a diffusion prior to reconstruct the image embedding by conditioning on the text embedding during the reverse diffusion process, which is a key step in the diversity of generated images.
Related Work
Under the supervised learning settings, to model the conditional distribution p ( y | x ) besides just the conditional mean E [ y | x ] through deep neural networks, existing works have been focusing on quantifying predictive uncertainty, and several lines of work have been proposed.
Bayesian neural networks (BNNs) model such uncertainty by assuming distributions over network parameters, capturing the plausibility of the model given the data (Blundell et al., 2015; Hernández-Lobato and
Adams, 2015; Gal and Ghahramani, 2016; Kingma et al., 2015; Tomczak et al., 2021).
Kendall and
Gal (2017) also model the uncertainties in the model outputs besides model parameters, by including the additive noise term as part of the neural network output.
Meanwhile, ensemble-based methods (Lakshminarayanan et al., 2017; Liu et al., 2022) have been proposed to model predictive uncertainty by combining multiple neural networks with stochastic outputs.
Furthermore, the Neural Processes
Family (Garnelo et al., 2018b,a; Kim et al., 2019; Gordon et al., 2020) has introduced a series of models that capture predictive uncertainty in an out-of-distribution fashion, particularly designed for few-shot learning settings.
These above mentioned models have all assumed a parametric form in p ( y | x ) , namely Gaussian distribution, or a mixture of Gaussians, and optimize the network parameters based on a Gaussian negative log-likelihood objective function.
Deep generative models, on the other hand, have been known for modeling implicit distributions without parametric distributional assumptions, but very few works have been proposed to utilize such feature to tackle regression tasks.
GAN-based models are introduced by Zhou et al.
(2021) and Liu et al.
(2021) for conditional density estimation and predictive uncertainty quantification.
For classification tasks, on the other hand, generative classifiers (Revow et al., 1996; Fetaya et al., 2020; Ardizzone et al., 2020; Mackowiak et al., 2021) is a class of models that also perform classification with generative models; among them, Zimmermann et al.
(2021) propose score-based generative classifiers to tackle classification tasks with score-based generative models (Song et al., 2021b,c).
They model p ( x | y ) and predict the label with the largest conditional likelihood of x , while CARD models p ( y | x ) instead.
In recent years, the class of diffusion-based (or score-based) deep generative models has demonstrated its outstanding performance in modeling high-dimensional multi-modal distributions (Ho et al., 2020;
Song et al., 2021a; Kawar et al., 2022; Xiao et al., 2022; Dhariwal and Nichol, 2021; Song and Ermon, 2019, 2020), with most work focusing on Gaussian diffusion processes operating in continuous state spaces.
Hoogeboom et al.
(2021) introduce extensions of diffusion models for categorical data, and Austin et al.
(2021) have proposed diffusion models for discrete data as a generalization of the multinomial diffusion models, which could provide an alternative way of performing classification with diffusion-based models.
Experiments
For the hyperparameters of CARD in both regression and classification tasks, we set the number of timesteps as T = 1000 , a linear noise schedule with β 1 = 10 − 4 and β T = 0 .
02 , same as Ho et al.
We provide a more detailed walk-through of the experimental setup, including training and network architecture, in Appendix A.8.
4.1 Regression
Putting aside its statistical interpretation, the word regress indicates a direction opposite to progress , suggesting a less developed state.
Such semantics in fact translates well into the statistical domain, in the sense that traditional regression analysis methods often only focus on estimating E ( y | x ) , while leaving out all remaining details about p ( y | x ) .
In recent years, Bayesian neural networks (BNNs) have emerged as a class of models that aims at estimating the uncertainty (Hernández-Lobato and
Adams, 2015; Gal and Ghahramani, 2016; Lakshminarayanan et al., 2017; Tomczak et al., 2021), providing a more complete picture of p ( y | x ) .
The metric that they use to quantify uncertainty estimation, negative log-likelihood (NLL), is computed with a Gaussian density, implying their assumption such that the conditional distributions p ( y | x = x ) for all x are Gaussian.
However, this assumption is very difficult to verify for real-world datasets: the covariates can be arbitrarily high-dimensional, making the feature space increasingly sparse with respect to the number of collected observations.
To accommodate the need for uncertainty estimation without imposing such restriction for the parametric form of p ( y | x ) , we apply the following two metrics, both of which are designed to empirically evaluate the level of similarity between the learned and the true conditional distributions:
Prediction Interval Coverage Probability (PICP); 2.
Quantile Interval Coverage Error (QICE).
PICP has been described in Yao et al.
(2019), whereas QICE is a new metric proposed by us.
We describe both of them in what follows.
4.1.1 PICP and QICE
The PICP is computed as
[Equation]
where ˆ y low n and ˆ y high n represent the low and high percentiles, respectively, of our choice for the predicted y outputs given the same x input.
This metric measures the proportion of true observations that fall in the percentile range of the generated y samples given each x input.
Intuitively, when the learned distribution represents the true distribution well, this measurement should be close to the difference between the selected low and high percentiles.
In this paper, we choose the 2 .
5 th and 97 .
5 th percentile, thus an ideal PICP value for the learned model should be 95% .
Meanwhile, there is a caveat for this metric: for example, imagine a situation where the 2 .
5 th to 97 .
5 th percentile of the learned distribution happens to cover the data between the 1 st and 96 th percentiles from the true distribution.
Given enough samples, we shall still obtain a PICP value close to 95% , but clearly there is a mismatch between the learned distribution and the true one.
Based on such reasoning, we propose a new empirical metric QICE, which by design can be viewed as PICP with finer granularity, and without uncovered quantile ranges.
To compute QICE, we first generate enough y samples given each x , and divide them into M bins with roughly equal sizes.
We would obtain the corresponding quantile values at each boundary.
In this paper, we set M = 10 , and obtain the following 10 quantile intervals (QIs) of the generated y samples: below the 10 th percentile, between the 10 th and 20 th percentiles, . . .
, between the 80 th and 90 th percentiles, and above the 90 th percentile.
Optimally, when the learned conditional distribution is identical to the true one, given enough samples from both learned and true distribution we shall observe about 10% of true data falling into each of these 10 QIs.
We define QICE to be the mean absolute error between the proportion of true data contained by each
QI and the optimal proportion, which is 1 /M for all intervals:
[Equation]
where r m = 1 N
N (cid:88)
[Equation]
Intuitively, under optimal scenario with enough samples, we shall obtain a QICE value of 0 .
Note that each r m is indeed the PICP for the corresponding QI with boundaries at ˆ y low m n and ˆ y high m n .
Since the true y for each x is guaranteed to fall into one of these QIs, we are thus able to overcome the mismatch issue described in the above example for PICP: fewer true instances falling into one QI would result in more instances captured by another QI, thus increasing the absolute error for both QIs.
QICE is similar to NLL in the sense that it also utilizes the summary statistics of the samples from the learned distribution conditional on each new x to empirically evaluate how well the model fits the true data.
Meanwhile, it does not assume any parametric form on the conditional distribution, making it a much more generalizable metric to measure the level of distributional match between the learned and the underlying true conditional distributions, especially when the true conditional distribution is known to be multi-modal.
We will demonstrate this point through the regression toy examples.
4.1.2 Toy Examples
To demonstrate the effectiveness of CARD in regression tasks for not only learning the conditional mean E ( y | x ) , but also recreating the ground truth data generating mechanism, we first apply CARD on 8 toy examples, whose data generating functions are designed to possess different statistical characteristics: some have a uni-modal symmetric distribution for their error term (linear regression, quadratic regression, sinusoidal regression), others have heteroscedasticity (log-log linear regression, log-log cubic regression) or multi-modality (inverse sinusoidal regression, 8 Gaussians, full circle).
We show that the trained CARD models can generate samples that are visually indistinguishable from the true response variables of the new covariates, as well as quantitatively match the true distribution in terms of some summary statistics.
We present the scatter plots of both true and generated data for all 8 tasks in Figure 1.
For tasks with uni-modal conditional distribution, we fill the region between the 2 .
5 th and 97 .
5 th percentile of the generated y ’s.
We observe that within each task, the generated samples blend remarkably well with the true test instances, suggesting the capability of reconstructing the underlying data generation mechanism by CARD.
A more detailed description of the toy examples, including more quantitative analyses, is presented in Appendix A.13.
4.1.3 UCI Regression Tasks
We continue to investigate our model through experiments on real-world datasets.
We adopt the same set of 10 UCI regression benchmark datasets (Dua and Graff, 2017) as well as the experimental protocol proposed by Hernández-Lobato and Adams (2015) and followed by Gal and Ghahramani (2016) and Lakshminarayanan et al.
The dataset information is provided in Table 14.
We apply multiple train-test splits with 90% / 10% ratio in the same way as Hernández-Lobato and
Adams (2015) ( 20 folds for all datasets except 5 for Protein and 1 for Year), and report the metrics by their mean and standard deviation across all splits.
We compare our method to all aforementioned
BNN frameworks: PBP, MC Dropout, and Deep Ensembles, as well as another deep generative model that estimates a conditional distribution sampler, GCDS (Zhou et al., 2021).
We note that
GCDS is related to a concurrent work of Yang et al.
(2022), who share a comparable idea but use it in a different application.
Following the same paradigm of BNN model assessment, we evaluate the accuracy and predictive uncertainty estimation of CARD by reporting RMSE and NLL.
Furthermore, we also report QICE for all methods to evaluate distributional matching.
Since this new metric was not applied in previous methods, we re-ran the experiments for all BNNs and obtained comparable or slightly better results in terms of other commonly used metrics reported in their literature.
Further details about the experimental setup for these models can be found in Appendix A.9.
The experiment results with corresponding metrics are shown in Tables 1, 2, and 3, with the number of times that each model achieves the best corresponding metric reported in the last row.
We observe that CARD outperforms existing methods, often by a considerable margin (especially on larger datasets), in all metrics for most of the datasets, and is competitive with the best method for the remaining ones: we obtain state-of-the-art results in 9 out of 10 datasets in terms of RMSE, 8 out of 10 for NLL, and 5 out of 10 for QICE.
It is worth noting that although we do not explicitly optimize our model by MSE or by NLL, we still obtain better results than models trained with these objectives.
4.2 Classification
Similar to Lakshminarayanan et al.
(2017), our motivation for classification is not to achieve state-ofthe-art performance in terms of mean accuracy on the benchmark datasets, which is strongly related to network architecture design.
Our goal is two-fold:
We aim to solve classification problems via a generative model, emphasizing its capability to improve the performance of a base classifier with deterministic outputs in terms of accuracy; 2.
We intend to provide an alternative sense of uncertainty, by introducing the idea of model confidence at the instance level, i.e. , how sure the model is about each of its predictions, through the stochasticity of outputs from a generative model.
As another type of supervised learning problems, classification is different from regression mainly for the response variable being discrete class labels instead of continuous values.
The conventional operation is to cast the classifier output as a point estimate, with a value between 0 and 1 .
Such
[Table] [Table]
[Table]
design is intended for prediction interpretability: since humans already have a cognitive intuition for probabilities (Cosmides and Tooby, 1996) [Table] the output from a classification model is intended to convey a sense of likelihood for a particular class label.
In other words, the predicted probability should reflect its confidence, i [Table] e. , a level of certainty, in predicting such a label.
Guo et al.
(2017) provide the following example of a good classifier, whose output aligns with human intuition for probabilities: if the model outputs a probability prediction of 0 .
8 , we hope it indicates that the model is 80% sure that its prediction is correct; given 100 predictions of 0 . 8 , one shall expect roughly 80 of them are correct.
In that sense, a good classification algorithm not only can predict the correct label, but also can reflect the true correctness likelihood through its probability predictions, i.e. , providing calibrated confidence (Guo et al., 2017).
To evaluate the level of miscalibration by a model, metrics like
Expected Calibration Error (ECE) and Maximum Calibration Error (MCE) (Naeini et al., 2015) have been adopted in recent literature (Kristiadi et al., 2022; Rudner et al., 2021) for image classification tasks, and calibration methods like Platt scaling and isotonic regression have been developed to improve such alignment (Guo et al., 2017).
Note that these methods are all based on point estimate predictions by the classifier.
Furthermore, these alignment metrics can only be computed at a subgroup level in practice, instead of at the instance level.
In other words, one may not be able to make the claim with the existing classification framework such that given a particular test instance , how confident the classifier is in its prediction to be correct.
We discuss our analysis in ECE with more details in Appendix A.16, which may help justify our motivation in introducing an alternative way to measure model confidence at the level of individual test instances in Section 4.2.1.
4.2.1 Predict with Instance Level Model Confidence via Generative Models
We propose the following framework to assess model confidence for its predictions at the instance level : for each test instance, we first sample N class prototype reconstructions by CARD through the classification version of Algorithm 2, and then perform the following computations:
We directly calculate the prediction interval width (PIW) between the 2 .
5 th and 97 .
5 th percentiles of the N reconstructed values for all classes, i.e. , with C different classes in total, we would obtain C PIWs for each instance; 2.
We then convert the samples into probability space with Eq.
(10), and apply paired twosample t -test as an uncertainty estimation method proposed in Fan et al.
(2021): we obtain the most and second most predicted classes for each instance, and test whether the difference in their mean predicted probability is statistically significant.
This framework would require the classifier to not produce the exact same output each time, since the goal is to construct prediction intervals for each of the class labels.
Therefore, the class of generative models is a preferable modeling choice due to its ability to produce stochastic outputs, instead of just a point estimate by traditional classifiers.
In practice, we view each one-hot label as a class prototype in real continuous space (introduced in Section 2.2), and we use a generative model to reconstruct this prototype in a stochastic fashion.
The intuition is that if the classifier is sure about the class that a particular instance belongs to, it would precisely reconstruct the original prototype vector without much uncertainty; otherwise, different class prototype reconstructions of the same test instance tend to have more variations: under the context of denoising diffusion models, given different samples from the prior distribution at timestep T , the label reconstructions would appear rather different from each other.
4.2.2 Classification with Model Confidence on CIFAR-10 Dataset
We demonstrate our experimental results on the CIFAR-10 dataset.
We first contextualize the performance of CARD in conventional metrics including accuracy and NLL with other BNNs in ResNet-18 architecture in Table 4.
The metrics of other methods were reported in Tomczak et al.
(2021), a recent work in BNNs that proposes tighter ELBOs to improve variational inference performance and prior hyperparameter optimization.
Following the recipe in Section 2.2, we first pre-train a deterministic classifier with the same ResNet-18 architecture, and achieve a test accuracy of 90 .
39% , with which we proceed to train CARD.
We then obtain our instance prediction through majority vote, i.e. , the most predicted class label among its N samples for each image input, and achieve an improved test accuracy with a mean of 90 .
93% across 10 runs, showing its ability to improve test accuracy from the base classifier.
Our NLL result is competitive among the best ones, even though the model is not optimized with a cross-entropy objective function, as we assume the class labels to be in the real continuous space.
[Table]
We now present the results from one model run with the proposed framework in Section 4.2.1 for evaluating the instance-level prediction confidence.
After obtaining the PIW and paired two-sample t -test ( α = 0 . 05 ) result from each test instance, we first split the test instances into two groups by the correctness of majority-vote predictions, then we obtain only the PIW corresponding to the true class for each instance, and compute the mean PIW of the true class within each group.
In addition, we split the test instances by t -test rejection status, and compute the mean accuracy in each group.
We report the results from these two grouping procedures in Table 5, where the metrics are computed across all test instances and at the level of each true class label
[Table]
[Table]
We observe from Table 5 that under the scope of the entire test set, the mean PIW of the true class label among the correct predictions is narrower than that of the incorrect predictions by an order of magnitude, indicating that when CARD is making correct predictions, its class label reconstructions have much smaller variations.
We may interpret such results as that CARD can reveal what it does not know through the relativity in reconstruction variations.
Furthermore, when comparing the mean
PIWs across different classes, we observe that the class with a higher prediction accuracy tends to have a sharper contrast in true label PIW between correct and incorrect predictions; additionally, the PIW values of both correct and incorrect predictions tend to be larger in a less accurate class.
Meanwhile, it is worth noting that if we predict the class label by the one with the narrowest PIW for each instance, we can already obtain a test accuracy of 87 .
84% , suggesting a strong correlation between the prediction correctness and instance-level model confidence (in terms of label reconstruction variability).
Moreover, we observe that the accuracy of test instances rejected by the t -test is much higher than that of the not-rejected ones, both across the entire test set and within each class.
We point out that these metrics can reflect how sure CARD is about the correctness of its predictions, and can thus be used as an important indicator of whether the model prediction of each instance can be trusted or not.
Therefore, it has the potential to be further applied in the human-machine collaboration domain (Madras et al., 2018;
Raghu et al., 2019; Wilder et al., 2020; Gao et al., 2021), such that one can apply such uncertainty measurement to decide if we can directly accept the model prediction, or we need to allocate the instance to humans for further evaluation.
Conclusion
In this paper, we propose Classification And Regression Diffusion (CARD) models, a class of conditional generative models that approaches supervised learning problems from a conditional generation perspective.
Without training with objectives directly related to the evaluation metrics, we achieve state-of-the-art results on benchmark regression tasks.
Furthermore, CARD exhibits a strong ability to represent the conditional distribution with multiple density modes.
We also propose a new metric Quantile Interval Coverage Error (QICE), which can be viewed as a generalized version of negative log-likelihood in evaluating how well the model fits the data.
Lastly, we introduce a framework to evaluate prediction uncertainty at the instance level for classification tasks.
Acknowledgments
The authors acknowledge the support of NSF IIS 1812699 and 2212418, and the Texas Advanced
Computing Center (TACC) for providing HPC resources that have contributed to the research results reported within this paper.