UQGAN: A Unified Model for Uncertainty Quantification of Deep Classifiers trained via Conditional GANs
Abstract
We present an approach to quantifying both aleatoric and epistemic uncertainty for deep neural networks in image classification, based on generative adversarial networks (GANs).
While most works in the literature that use GANs to generate out-of-distribution (OoD) examples only focus on the evaluation of OoD detection, we present a GAN based approach to learn a classifier that produces proper uncertainties for OoD examples as well as for false positives (FPs).
Instead of shielding the entire in-distribution data with GAN generated OoD examples which is stateof-the-art, we shield each class separately with out-of-class examples generated by a conditional GAN and complement this with a one-vs-all image classifier.
In our experiments, in particular on CIFAR10, CIFAR100 and Tiny ImageNet, we improve over the OoD detection and FP detection performance of state-of-the-art
GAN-training based classifiers.
Furthermore, we also find that the generated GAN examples do not significantly affect the calibration error of our classifier and result in a significant gain in model accuracy.
Introduction
Deep learning has shown outstanding performance in image classification tasks [22, 43, 13].
However, due to the enormous capacity and the inability of rejecting examples, deep neural networks (DNNs) are not capable of expressing their uncertainties appropriately.
DNNs have demonstrated the tendency to overfit training data [46] and to be easily fooled into wrong class predictions with high confidence [9, 12].
Also, importantly, ReLU networks have been proven to show high confidence far away from training data [14].
This is contrary to the behavior that would be expected naturally by humans, which is to be uncertain when being confronted with new data examples that have not been observed during training.
Several approaches to predictive uncertainty quantification have been introduced in recent years, considering uncertainty in a Bayesian sense [2, 7, 18] as well as from a frequentist’s point of view [15, 5, 34].
A common evaluation protocol is to discriminate between true and false positives (FPs) by means of a given uncertainty quantification.
For an introduction to uncertainty in machine learning, we refer to [17], for a survey on uncertainty quantification methods for DNNs see [8].
By design of ordinary DNNs for image classification, their uncertainty is often studied on indistribution examples [1].
The task of out-of-distribution (OoD) detection (or novelty detection) is oftentimes considered separately from uncertainty quantification [29, 44, 31].
Thus, OoD detection in deep learning has spawned an own line of research and method development.
Among others, changes in architecture [5], loss function [41, 48] and the incorporation of data serving as OoD proxy [16] have been considered in the literature.
Generative adversarial networks (GANs) have been used to replace that proxy by artificially generated data examples.
In [27], examples of OoD data are created such that they shield the in-distribution regime from the OoD regime.
Note that this often requires pre-training of the generator.
E.g. in the aforementioned work, the authors constructed
OoD data in their 2D example to pretrain the generator.
While showing promising results, such
GAN-based approaches mostly predict a single score for OoD detection and do not yield a principled approach to uncertainty quantification distinguishing in-distribution uncertainty between classes and out-of-distribution uncertainty.
In this work, we propose to use GANs to, instead of shielding all classes at once, shield each class separately from the out-of-class (OoC) regime (also cf.
figure 1).
Instead of maximizing an uncertainty measure, like softmax entropy, we combine this with a one-vs-all classifier in the final
DNN layer.
This is learned jointly with a class-conditional generator for out-of-class data in an adversarial framework.
The resulting classifiers are used to model (class conditional) likelihoods.
Via Bayes rule we define posterior class probabilities in a principled way.
Our work thus makes the following novel contributions:
We introduce a GAN-based model yielding a classifier with complete uncertainty quantification.
Our model allows to distinguish uncertainty between classes (in large sample limit, if properly learned, approaching aleatoric uncertainty) from OoD uncertainty (approaching epistemic uncertainty).
By a conditional GAN trained with a Wasserstein-based loss function, we achieve class shielding in low dimensions without any pre-training of the generator. 4.
In higher dimensions, we use a class conditional autoencoder and train the GAN on the latent space.
This is coherent with the conditional GAN, allows us to use less complex generators and critics and reduces the influence of adversarial directions.
We improve over the OoD detection and FP detection performance of state-of-the-art GANtraining based classifiers.
We present in-depth numerical experiments with our method on MNIST, CIFAR10, CIFAR100 and
Tiny ImageNet, accompanied with various OoD datasets.
We outperform other approaches, also GANbased ones, in terms of OoD detection and FP detection performance on CIFAR10.
Also on MNIST,
CIFAR100 and Tiny ImageNet we achieve superior OoD detection performance.
Noteworthily, on the more challenging CIFAR10, CIFAR100 and Tiny ImageNet datasets, we achieve significantly stronger model accuracy compared to other approaches based on the same network architecture.
Related Work
In this section we give an overview of common uncertainty quantification methods as well as publications related to the different parts of our method.
In this context the task of uncertainty quantification is to assign scalar values to predictions, quantifying aleatoric (in-distribution) and epistemic (out-of-distribution) uncertainty.
(Baselines)
The works by [52, 15, 5] can be considered as early baseline methods for the task of OoD detection.
They are frequentist approaches relying on confidence scores gathered from model outputs.
The problem can oftentimes be attributed to the usage of the softmax activation function, which leads to overconfident predictions, in particular far away from training data, or to the decoupling of the confidence score from the original classification model during test time.
Our proposed method does not make use of the softmax activation function and can produce unified uncertainty estimates during test time without the requirement of auxiliary confidence scores.
(Conformal methods) Methods associated with the term conformal predictions [35, 3, 30] are predicting, for a given confidence value, a set of classes that is likely to contain the real class.
As such, they can give a more intuitive explanation to which classes might be getting confused.
In essence, the uncertainty of a prediction can be derived from the size of the predicted set, assigning higher prediction uncertainty to larger sets.
They are however not able to quantify both, aleatoric and epistemic uncertainty, and assigning a scalar uncertainty value to predictions, as in our setting, might not be straight forward.
(Data perturbation and auxiliary data)
Many methods use perturbed training examples [28, 37, 29] or auxiliary outlier datasets [16, 20].
[28] use them for their confidence score based on class conditional Gaussian distributions, while [29] and [37] are utilizing them to increase the separability between in- and out-of-distribution examples.
[16] use a hand picked auxiliary outlier dataset during model training and [20] use it for selecting a suitable GAN discriminator during training which then serves for OoD detection.
The common problem with using perturbed examples is the sensitivity to hyperparameter selection which might render the resulting examples uninformative.
Additionally, auxiliary outlier datasets cannot always be considered readily available and pose the problem of covering only a small proportion of the real world.
In contrast, our method is able to produce OoC examples that are very close to the in-distribution but still distinguishable from it, thus we do not require explicit data perturbation or any auxiliary outlier datasets.
(Bayesian methods)
Bayesian approaches [7, 2, 24] provide a strong theoretical foundation for uncertainty quantification and OoD detection.
[24] propose deep ensembles that approximate a distribution over models by averaging predictions of multiple independently trained models.
[7] are utilizing Monte-Carlo (MC) sampling with dropout applied to each layer.
In [2] a variational learning algorithm for approximating the intractable posterior distribution over network weights has been proposed.
While the theoretical foundation is strong, these methods often require changing the architecture, restricting the model space and/or increased computational cost.
While making use of
Bayes-rule, we are staying in a frequentist setting and are not dependent on sampling or ensemble techniques.
This reduces computational cost and enables our model to produce high quality aleatoric and epistemic uncertainty estimates with a single forward pass.
Also, our proposed framework does not change the network architecture, except for the output layer activation function, and thus makes it compatible with previously published techniques.
(One-vs-All methods) One-vs-All methods in the context of OoD detection have been recently studied by [6, 34, 38].
In the work by [6] an ensemble of binary neural networks is trained to perform one-vs-all classification on the in-distribution data which are then weighted by a standard softmax classifier.
[34] use a DNN with a single sigmoid binary output for every class and explore the possibility of training the one-vs-all network with a distance based loss function instead of the binary cross entropy.
Domain adaptation is considered in [38], where they utilize a one-vs-all classifier for a first OoD detection step before classifying into known classes with a second model.
Their training objective is also accompanied by a hard negative mining on the in-distribution data.
All these methods use the maximum predicted probability as a score for OoD detection and/or classification and do not aggregate the other probabilities into a single score like the method proposed by us.
They also do not distinguish into different kinds of uncertainties as in our work.
Lastly their training objectives are only based on in-distribution data.
Generated OoC data as used in the present work is not considered.
(Generative methods) More recently generative model based methods [39, 27, 45, 47, 51] have shown strong performance on the task of out-of-distribution detection by supplying classification models with synthesized out-of-distribution examples.
[39] utilize the latent space of a GAN by gradient based reconstruction of an input example.
In the work by [27], a GAN architecture with an additional classification model is built.
The classification model is trained to output a uniform distribution over classes on GAN examples close to the in-distribution.
This approach is further improved by [45] who show improvements on the task of out-of-distribution detection.
The generalization to distant regions in the sample space and the quality of generated boundary examples is however questionable [51].
A similar approach using a normalizing flow and randomly sampled latent vectors is proposed by [10].
The high level idea of the architectures proposed in the previously mentioned works is similar to the one proposed by us.
However, other works are not able to approximate the boundary of data distributions with multiple modes as shown by [51].
Due to the fact that our GAN is class conditional and trained on a low dimensional latent space, we are able to follow multiple distribution modes resulting from different classes.
We improve the in-distribution shielding by using a low-dimensional regularizer and have an additional advantage in terms of computational cost as our cGAN model architecture can be chosen with considerably smaller size due to it being trained in the latent space.
Furthermore, these methods do not yield separate uncertainty scores for FP and OoD examples.
Generating OoD data based on lower dimensional latent representations has been explored in [50, 40].
[50] utilize a variational Autoencoder (vAE) to produce examples that are inside the encoded manifold (type I) as well as outside of it (type II).
[40] (GEN) also use a vAE and train a GAN in the resulting latent space, assigning generated examples to the OoD domain in order to estimate a
Dirichlet distribution on the class predictions.
Utilizing a vAE has the advantage that one can make assumptions on the distribution of the latent space but also results in slightly blurry reconstructions.
While the work of [40] is the most similar to our method, we improve on several shortcomings.
First of all we employ class conditional models to improve diversity and class shielding.
Additionally, we are able to distinguish aleatoric and epistemic uncertainty while the method by [40] is not.
It assigns the same type of uncertainty to OoD and FP examples.
Method 3.1 One-vs-All Classification
We start by formulating our classification model as an ensemble of one-vs-all classifiers.
Let
C ( o | x, y ) model the probability that for a given class y ∈ Y = { 1 , . . . , n } , an example with features x ∈ X =
R d is OoC. Analogously,
C
( i | x, y )
= 1 − C
( o | x, y ) for a given class y models the probability of x being in-class.
Let S ⊆
X
× Y be our training dataset with ˆ p ( y ) ,
y ∈ Y , the estimated relative class frequencies.
To model C ( i | x, y ) , we use a DNN with n ≥ 2 output neurons equipped with sigmoid activations.
For each class output y , the data corresponding to class y serves as in-class data and all other data as OoC data.
Hence, a basic variant of our training objective is given by a weighted empirical binary cross entropy
[Equation]
Therein, ˆ p ( y ) ˆ p ( y ′ ) is weighting the OoC loss to counter potential class imbalance.
Similarly, 1 n − 1 weighs the OoC loss compared to the in-class loss.
Applying the transformation
[Equation]
after training, we obtain an appropriate classifier, formalized as follows:
Lemma 1 (Class posterior) .
Under typical assumptions of statistical learning theory and training C on equation (1) it holds that for | S | → ∞
[Equation]
A enc ( x,y ) z y e ˜ z y
G (˜ z,y )
z y ˜
z
y
D (˜ z,y ) ( z,y )
A dec ˜ x x
C
L C
L GD
C ( i | x,y ) C
( o | x,y ′ ) C ( o | ˜ x,y )
C ( o | ˜ x , y )
See appendix A for the detailed assumptions and the proof, explaining our choice of the classification model.
We distinguish between aleatoric and epistemic uncertainty.
While aleatoric uncertainty is considered to be inherent to all observations and irreducible (for a fixed X ), epistemic uncertainty is induced by the model choice and the amount of data and can be reduced by additional data and appropriate model selection.
Using ˆ p ( y | x ) , we estimate the probability of an example x being in-distribution by defining
[Equation]
which yields a quantification of epistemic uncertainty via ˜ C
( o | x ) = 1 − ˜ C
( i | x ) .
For aleatoric uncertainty estimation, we consider the Shannon entropy of the predicted class probabilities
[Equation]
This is a sensible definition since we showed that ˆ p ( y | x )
→ p ( y | x ) for | S | → ∞ .
If the real p ( y | x ) is close to a uniform distribution over the classes (which maximizes the Shannon entropy) we have a high uncertainty and vice versa.
For epistemic uncertainty ˜ C ( i | x ) can be considered as a proxy for p ( i | x ) , which results in the epistemic uncertainty by 1 − ˜ C
( i | x ) .
In addition, we generate for each class OoC examples with a conditional GAN.
A joint training of classifier and GAN is introduced in the upcoming section 3.2.
3.2 GAN Architecture
Similar to [27], we combine our classification model with a GAN and train both alternatingly.
The assumptions that we made for lemma 1 are standard in statistical learning theory, however it is important to notice that we exclude the GAN-generated data from the statement.
By incorporating
GAN-generated data into the training of our classification model introduced in section 3.1, we are violating this assumption.
However, in section 4 we demonstrate empirically that this does not harm our classification model performance-wise.
The Wasserstein GAN with gradient penalty proposed by [11] serves a basis for our conditional GAN (cGAN).
Additionally, we condition the generator as well as the critic on the class labels to generate class specific OoC examples.
Inspired by [40, 50] where the latent space of a variational autoencoder (vAE) is utilized for GAN training, we proceed analogously, however using a vanilla conditional autoencoder (cAE) as we observed reconstructions with higher visual quality for this choice.
Training the cGAN on a latent space, we do not generate adversarial noise examples and can use less complex generators and critics.
Prior to the cGAN training, we train the cAE on in-distribution data and then freeze the weights during the cGAN and classifier training.
The optimization objective of the cAE is given as pixel-wise binary cross-entropy
[Equation]
with ˆ x i =
A dec ( z, y ) the decoded latent variable, z =
A enc ( x, y ) being the encoded example, x i the i -th pixel of example x and N x the number of pixels belonging to x .
Therein, the pixel values are assumed to be in the interval [0 , 1] , while 0 · log(0) = 0 .
The cGAN is trained using the objective function
[Equation]
with D the conditional critic, z
=
A enc ( x, y ) , ˜ z
= G ( e, y )
the latent embedding produced by the conditional generator, e ∼ U (0 , 1) noise from a uniform distribution, y a class label and L gp the gradient penalty from [11].
Integrating the classification objective into the cGAN objective, we alternate between
[Equation]
and
[Equation]
C
[Equation]
with λ real ∈ [0 , 1] being an interpolation factor between real and generated OoC examples and L R being an additional regularization loss for the generated latent codes with hyperparameter λ R ≥ 0 , which we introduce in section 3.3.
The latent embeddings produced by the cGAN are decoded with the pretrained cAE, thus ˜ x
= A dec (˜ z, y ) =
A dec ( G ( e, y ) , y ) .
That is, the cGAN is trained on the latent space while the classification model is trained on the original feature space.
Our entire
GAN-architecture is visualized in figure 2.
3.3 Low-Dimensional Regularizer
In low dimensional latent spaces we found it to be advantageous to apply an additional regularizer to the generated latent embeddings ˜ z to improve class shielding.
Let ( z, y ) be the latent embedding of an example x with its corresponding class label y and Z ( z, y ) =
{ ˜ z − z | ˜ z
= G ( e, y ) , e ∼ U (0 , 1) } = n ¯ z 1 , . . .
, ¯ z N zy
o all generated latent codes with the same class label and normalized to origin z .
We encourage the generator to produce latent codes that more uniformly shield the class y by maximizing the average angular distance between all ¯ z ∈ Z ( z, y ) , which corresponds to minimizing
[Equation]
with ∗ being the dot-product.
The logarithm introduces an exponential scaling for very small angular distances, encouraging a more evenly spread distribution of the generated latent codes.
This loss is then averaged over all class labels and training data examples
[Equation]
[Equation]
with N y =
|{ ( x, y ′ ) |
y ′ =
y }| the number of examples with class label y .
In appendix B a more detailed explanation of this regularization loss is provided.
During our experiments we studied different regularizer losses such as Manhattan/Euclidean distance, infinity norm and standard cosine similarity.
In experiments, we found that for our purpose equation (11) performed best.
We argue that this can be attributed to the independence of the latent space
[Table]
[Table]
[Table]
[Figure]
examples for value range, which can have a large impact on the p -norm distance metrics, and to the exponential scaling for very small angular distances.
To this end, the basic variant of our method does not account for model uncertainty.
Thus, we also include results with MC-Dropout, which also demonstrates the compatibility of our model with existing methods.
Another limitation of our approach is that we are not explicitly accounting for adversarial attacks because during training in the cAE latent space we are removing any adversarial directions.
Experiments
We compare our method with a number of related approaches (cf.
table 1).
While the first two methods are simple baselines, the subsequent three ones are Bayesian ones and the final two methods are
GAN-based (cf.
section 2).
Following the publications [40, 50, 37, 15], we consider four experimental setups using the MNIST ( 28 × 28 , 10 classes) [26], CIFAR10 ( 32 × 32 , 10 classes) [21], CIFAR ( 32 × 32 , 100 classes) [21] and Tiny ImageNet ( 64 × 64 , 200 classes) [25] datasets as in-distribution, respectively.
Similar to [40, 50] and others, we split the datasets class-wise into two non-overlapping sets, i.e., MNIST 0-4 / 5-9, CIFAR10 0-4 / 5-9, CIFAR100 0-49 / 50-99 and Tiny ImageNet 0-99 / 100-199.
While the first half serves as in-distribution data, the second half constitutes OoD cases close to the training data and therefore difficult to detect.
For the MNIST 0-4 dataset, we consider the
MNIST 5-9, EMNIST-Letters [4], Fashion-MNIST [53], Omniglot [23], SVHN [33] and CIFAR datasets as OoD examples.
For the CIFAR10 0-4 datasets, we use CIFAR10 5-9, LSUN [55], SVHN,
Fashion-MNIST and MNIST as OoD examples.
The same OoD datasets are used for CIFAR except CIFAR10 5-9 being replaced by CIFAR100 50-99.
For the Tiny ImageNet 0-99 dataset we use Tiny ImageNet 100-199, SVHN, Fashion-MNIST and MNIST as OoD examples.
These selections yield compositions of training and OoD examples with strongly varying difficulty for state-of-the-art OoD detection.
Besides that, we examine our method’s behavior on a 2D toy example with two overlapping Gaussians (having trivial covariance structure), see figure 1.
Additionally, we split the official training sets into 80% / 20% training / validation sets, where the latter are used for hyperparameter tuning and model selection.
Like related works, we utilize the LeNet-5 architecture on MNIST and a small ResNet on CIFAR10, CIFAR100 and Tiny ImageNet as classification models.
To ensure fair conditions, we re-implemented all aforementioned methods while following the authors recommendations for hyperparameters and their reference implementations.
For methods involving more complex architectures, e.g. a GAN or a VAE as in [27, 40], we used the proposed architectures for those components, while for the sake of comparability sticking to our choice of classifier models.
All implementations are using PyTorch [36] and can be found in the associated repository 1 .
For each method, we selected the network checkpoint with maximal validation accuracy during training.
For a more detailed overview of the hyperparameters used in our experiments, we refer to appendix D or the implementation.
For evaluation we use the following well established metrics:
• Classification accuracy on the in-distribution datasets.
[Table] [Table]
• Area under the Receiver Operating Characteristic Curve (AUROC).
We apply the AUROC to the binary classification tasks in-/out-of-distribution (via the score from equation (4)) and TP/FP (Success/Failure) (via the score from equation (5)).
• Expected Calibration Error (ECE) [32] applied to the estimated class probabilities ˆ p ( y | x ) for in-distribution examples x , computed on 15 bins.
• Area under the Precision Recall Curve (AUPR) w.r
[Table]
t. the binary in-/out-of-distribution decision (via the score from equation (4)).
We further distinguish between AUPR-In and AUPR-Out.
For AUPR-in the in-distribution class is the positive one, while for AUPR-Out the out-of-distribution class is the positive one.
• FPR @ 95% TPR computes the False Positive Rate (FPR) at the decision threshold on the OoD score from equation (4) ensuring a True Positive Rate (TPR) of 95%.
Before discussing our results on MNIST, CIFAR10, CIFAR100 and Tiny ImageNet
[Table]
we briefly discuss our findings on the 2D example.
As can be seen in figure 1, the generated OoC examples are nicely shielding the respective in-distribution classes.
OoC examples of one class can be in-distribution examples of other classes.
This is an intended feature and to this end, the loss term for the synthesized
OoC examples in equation (9) is class conditional.
This feature is supposed to make our one-vs-all classifier predict a high OoC probability in the OoC regime.
One can also observe that the estimated epistemic and aleatoric uncertainties are complementary, resulting in a high aleatoric uncertainty in the overlapping region of the Gaussians while also having a low epistemic uncertainty there.
This is one of the main advantages that sets our approach apart from related methods.
Results for a slightly more challenging 2D toy example on the two moons dataset and on a grid of Gaussians are presented in appendix C. We now demonstrate that this result generalizes to higher dimensional problems.
For MNIST and CIFAR10, figure 3 shows the OoC examples produced at the end of the generator training.
Due to using a conditional GAN and an AE, we are able to generate OoC examples (instead of only out-of-distribution as in related works) during test time.
It can be seen that the resulting examples resemble a lot of semantic similarities with the original class while still being distinguishable from them.
[Table] [Table]
All presented results are computed on the respective (official) test sets of the datasets.
We also conducted an extensive parameter study on the validation sets, which is summarized in appendix E.
The conclusion of this parameter study is that the performance of our framework is in general stable w.r.t.
the choice of the hyperparameters.
Increasing λ reg positively impacts the model performance up to a certain maximum.
The best performance is obtained by choosing latent dimensions such that the cAE is able to compute reconstructions of good visual quality.
Across all datasets, choosing λ real ∈ [0 . 5 , 0 . 6] achieves the best detection scores, also indicating a positive influence of the generated OoC examples on the model’s classification accuracy.
Tables 2 to 5 present the results of our experiments.
The scores in the columns of the section
In-Distribution are solely computed on the respective in-distribution dataset.
The scores in the columns of the section Out-of-Distribution are displaying the OoD detection performance when presenting examples from the respective in-distribution dataset as well as from the entirety of all assigned OoD datasets.
Note that we did not apply any balancing in the OoD datasets but included the respective test sets as is (see appendix D for the sizes of the test sets).
As an upper bound on the OoD detection performance we also show results for two oracle models, supplied with the real
OoD training datasets they are evaluated on.
One of them is trained with the standard softmax and binary-cross-entropy to maximize entropy on OoD examples and the other one with our proposed loss function, cf.
equation (9).
We first discuss the in-distribution performance of our method.
W.r.t.
MNIST, the results given in the left section of table 2 show that we are on par with state-of-the-art GAN-based approaches while still having a similar ECE, only being surpassed by Deep-Ensembles and the other baselines by a fairly small margin.
However, considering the respective CIFAR10 results in the left section of table 3, we clearly outperform state-of-the-art GAN-based methods as well as all other baseline methods by a large margin.
Noteworthily, we achieve an accuracy of 90 .
26% which is 5 to 8 percent points (pp.)
above the other classifiers and an AUROC S/F of 89 .
19% which is 4 to 6 pp.
higher than for the other methods.
This corresponds to a relative improvement of 33% in accuracy and 26% in AUROC
S/F compared to the second best method.
A similar improvement in accuracy can be observed for
CIFAR100 and Tiny ImageNet (tables 4 and 5) where the AUROC S/F is slightly lower but still on par with other approaches.
Considering ECE on CIFAR10/100 and Tiny ImageNet, our MC-Dropout variant is outperforming nearly all related methods, except for CIFAR100 and Tiny ImageNet where it is ranked second.
This signals that, although we incorporate generated OoC examples impurifying the distribution of training data presented to the classifier, empirically there is no evidence that this harms the learned classifier, neither w.r.t.
calibration, nor w.r.t.
separation.
Considering the OoD results from the right-hand sections of tables 2 to 5, the superiority of our method compared to the other ones is now consistent over all four in-distribution datasets.
On the MNIST dataset we are outperforming previously published works, especially considering the
AUPR-In and FPR @ 95% TPR metrics, with a 25% and 27% relative improvement over the second best method, respectively.
This is consistent with the results for CIFAR10 and CIFAR100 as indistribution datasets where we achieve for both AUROC and AUPR-In a relative improvement of 19% ± 26% and 3% ± 15% , respectively, over the second best method.
On the higher dimensional Tiny
ImageNet dataset, which has 20 × more classes than MNIST/CIFAR10 and 4 × more input pixels, the results are even stronger with an 36% ± 67% relative improvement over the second best competitor.
Comparing our results with the ones of the oracles, two observations become apparent.
Firstly, in some OoD experiments the GAN-generated OoC examples achieve results fairly close to or even surpassing the ones of the oracles while also in some of them there is still room for improvement left (in particular w.r.t. FPR@95%TPR).
Secondly, GAN-generated OoC examples can help improve generalization (in terms of classification accuracy) while real OoD data might be too far away from the in-distribution data.
An OoD-dataset-wise breakdown of the results in tables 2 to 5 is provided in appendix G. For MNIST, this breakdown reveals that our method performs particularly well in the difficult task of separating MNIST 0-4 and MNIST 5-9.
On the other MNIST-related tasks we achieve mid-tier results, being slightly behind the other GAN-based methods.
The same holds for
CIFAR100 0-49 and CIFAR100 50-99.
However, with regards to CIFAR10 and Tiny ImageNet we are consistently outperforming the other methods by large margins.
In appendix F we present results of an experiment where we perform FP and OoD detection jointly on the computed uncertainty scores.
The main observations there are that our method outperforms other GAN-based methods and that our method including dropout achieves the overall best detection performance.
Conclusion
In this work, we introduced a GAN-based model yielding a one-vs-all classifier with complete uncertainty quantification.
Our model distinguishes uncertainty between classes (in large sample limit approaching aleatoric uncertainty) from OoD uncertainty (approaching epistemic uncertainty).
We have demonstrated in numerical experiments that our model sets a new state-of-the-art w.r.t.
OoD as well as FP detection.
The generated OoC examples do not harm the training success in terms of calibration, but even improve it in terms of accuracy.
We have seen that incorporating MC dropout to account for model uncertainty can further improve the results.
Broader Impact
We are contributing generally applicable methodology which can be used in any real world application.
The general direction of making the deployment of models more feasible due to improved quality of uncertainties might ultimately result in a reduction of the number of jobs in the respective sectors.
However, as of now, it is also important to note that this method cannot give any guarantees on the estimated uncertainties and thus these should be used with caution and never solely relied upon.
Acknowledgment
M.R. acknowledges useful and interesting discussions with Hanno Gottschalk.
Funding
No third-party funding or support has been received for this work.
[-It is also very apparent, that for λ reg = 
-]
[-(aleatoric uncertainty) and b = H ( 1 K (cid:80) K 
-]
[-ECE        128 λ reg 82 .85 .
-]