D ETECTING M ISCLASSIFICATION E RRORS IN N EURAL N ETWORKS WITH A G AUSSIAN P ROCESS M ODEL
Anonymous authors
Paper under double-blind review
A BSTRACT
As neural network classiﬁers are deployed in real-world applications, it is crucial that their predictions are not just accurate, but trustworthy as well.
One practical solution is to assign conﬁdence scores to each prediction, then ﬁlter out lowconﬁdence predictions.
However, existing conﬁdence metrics are not yet sufﬁciently reliable for this role.
This paper presents a new framework that produces more reliable conﬁdence scores for detecting misclassiﬁcation errors.
This framework, RED, calibrates the classiﬁer’s inherent conﬁdence indicators and estimates uncertainty of the calibrated conﬁdence scores using Gaussian Processes.
Empirical comparisons with other conﬁdence estimation methods on 125 UCI datasets demonstrate that this approach is effective.
An experiment on a vision task with a large deep learning architecture further conﬁrms that the method can scale up, and a case study involving out-of-distribution and adversarial samples shows potential of the proposed method to improve robustness of neural network classiﬁers more broadly in the future.
I NTRODUCTION
Classiﬁers based on Neural Networks (NNs) are widely deployed in many real-world applications (LeCun et al., 2015; Anjos et al., 2015; Alghoul et al., 2018;
Shahid et al., 2019).
Although good prediction accuracies are achieved, lack of safety guarantees becomes a severe issue when NNs are applied to safety-critical domains, e g., healthcare (Selis¸teanu et al., 2018; Gupta et al., 2007; Shahid et al., 2019), ﬁnance (Dixon et al., 2017), self-driving (Janai et al., 2017; Hecker et al., 2018) etc.
One way to estimate trustworthiness of a classiﬁer prediction is to use its inherent conﬁdence-related score, e.g., the maximum class probability (Hendrycks & Gimpel, 2017), entropy of the softmax outputs (Williams & Renals, 1997), or difference between the highest and second highest activation outputs (Monteith & Martinez, 2010).
However, these scores are unreliable and may even be misleading: high-conﬁdence but erroneous predictions are frequently observed (Provost et al., 1998;
Guo et al., 2017; Nguyen et al., 2015; Goodfellow et al., 2014; Amodei et al., 2016).
In a practical setting, it is beneﬁcial to have a detector that can raise a red ﬂag whenever the predictions are suspicious.
A human observer can then evaluate such predictions, making the classiﬁcation system safer.
In order to construct such a detector, quantitative metrics for measuring predictive reliability under different circumstances are ﬁrst developed, and a warning threshold then be set based on users’ preferred precision-recall tradeoff.
Existing such methods can be categorized into three types based on their focus: error detection, which aims to detect the natural misclassiﬁcations made by the classiﬁer (Hendrycks & Gimpel, 2017; Jiang et al., 2018; Corbi`ere et al., 2019); out-of-distribution (OOD) detection, which reports samples that are from different distributions compared to training data (Liang et al., 2018; Lee et al., 2018a; Devries & Taylor, 2018); and adversarial sample detection, which ﬁlters out samples from adversarial attacks (Lee et al., 2018b; Wang et al., 2019; Aigrain &
Detyniecki, 2019).
Among these categories, error detection, also called misclassiﬁcation detection (Jiang et al., 2018) or failure prediction (Corbi`ere et al., 2019), is the most challenging (Aigrain & Detyniecki, 2019) and underexplored.
For instance, Hendrycks & Gimpel (2017) deﬁned a baseline based on maximum class probability after softmax layer.
Although the baseline performs reasonably well in most testing cases, reduced efﬁcacy in some scenaria indicates room for improvement (Hendrycks & Gimpel,
Jiang et al.
(2018) proposed Trust Score, which measures the similarity between the original classiﬁer and a modiﬁed nearest-neighbor classiﬁer.
The main limitation of this method is scalability: the Trust Score may provide no or negative improvement over the baseline for high-dimensional data.
ConﬁdNet (Corbi`ere et al., 2019) builds a separate NN model to learn the true class probablity, i.e. softmax probability for the ground-truth class.
However, ConﬁdNet itself is a standard
NN, so its conﬁdence scores may be unreliable or misleading: A random input may generate a random conﬁdence score, and ConﬁdNet do not provide any information regarding uncertainty of these conﬁdence scores.
Moreover, none of these methods can differentiate natural classiﬁer errors from risks caused by OOD or adversarial samples; if a detector could do that, it would be easier for practitioners to ﬁx the problem, e.g., by retraining the original classiﬁer or applying better preprocessing techniques to ﬁlter out OOD or adversarial data.
To meet these challenges, a new framework is developed in this paper for error detection in NN classiﬁers.
The main idea is to utilize RIO (Residual Input/Output) (Qiu et al., 2020), a regression model based on Gaussian Processes, on top of the original NN classiﬁer.
Although the original RIO is limited to regression problems, the proposed approach manages to extend RIO for misclassiﬁcation detection by modifying its components.
The new framework not only produces a calibrated conﬁdence score based on the original maximum class probability, but also provides a quantitative uncertainty estimation of that score.
The reliability of error detection is therefore enhanced.
It is notable that the proposed method does not change the prediction accuracy of the original classiﬁcation model.
Instead, it is a supporting tool that can provide a quantitative metric for detecting misclassiﬁcation errors of the original classiﬁcation model.
The framework, referred to as RED (RIO for Error Detection), is compared empirically to existing approaches on 125 UCI datasets and on a large-scale deep learning architecture.
The results demonstrate that the approach is effective and robust.
A further case study with OOD and adversarial samples shows the potential of using RED to diagnose the sources of mistakes as well, thereby leading to a possible comprehensive approach for improving trustworthiness of neural network classiﬁers in the future.
R ELATED W ORK
In the past two decades, a large volume of work was devoted to calibrating the conﬁdence scores returned by classiﬁers.
Early works include Platt Scaling (Platt, 1999;
Niculescu-Mizil & Caruana, 2005), histogram binning (Zadrozny & Elkan, 2001), isotonic regression (Zadrozny & Elkan, 2002), with recent extensions like Temperature Scaling (Guo et al., 2017), Dirichlet calibration (Kull et al., 2019), and distance-based learning from errors (Xing et al., 2020).
These methods focus on reducing the difference between reported class probability and true accuracy, and generally the rankings of samples are preserved after calibration.
As a result, the separability between correct and incorrect predictions is not improved.
In contrast, RED aims at deriving a score that can differentiate incorrect predictions from correct ones better.
A related direction of work is the development of classiﬁers with rejection/abstention option.
These approaches either introduce new training pipelines/loss functions (Bartlett & Wegkamp, 2008; Yuan & Wegkamp, 2010; Cortes et al., 2016), or deﬁne mechanisms for learning rejection thresholds under certain risk levels (Dubuisson & Masson, 1993; Santos-Pereira & Pires, 2005; Chow, 2006;
Geifman & El-Yaniv, 2017).
In contrast to these methods, RED assumes an existing pretrained NN classiﬁer, and provides an additional metric for detecting potential errors made by this classiﬁer, without specifying a rejection threshold.
Designing metrics for detecting potential risks in NN classiﬁers has also become popular recently.
While most approaches focus on detecting OOD (Liang et al., 2018; Lee et al., 2018a; Devries & Taylor, 2018) or adversarial examples (Lee et al., 2018b; Wang et al., 2019; Aigrain & Detyniecki, 2019), work on detecting natural errors, i.e., regular misclassiﬁcations not caused by external sources, is more limited.
Ortega (1995) and Koppel & Engelson (1996) conducted early work in predicting whether a classiﬁer is going to make mistakes, and Seewald & F¨urnkranz (2001) built a meta-grading classiﬁer based on similar ideas.
However, these early works did not consider NN classiﬁers.
More recently, Hendrycks & Gimpel (2017) and Haldimann et al.
(2019) demonstrated that raw maximum class probability is an effective baseline in error detection, although its performance was reduced in some scenaria.
More elaborate techniques for error detection have also been developed recently.
Mandelbaum &
Weinshall (2017) proposed a conﬁdence score based on the data embedding derived from the penultimate layer of a NN.
However, their approach requires modifying the training procedure in order to achieve effective embeddings.
Jiang et al.
(2018) introduced Trust Score to measure the similarity between a base classiﬁer and a modiﬁed nearest-neighbor classiﬁer.
Trust Score outperforms the maximum class probability baseline in many cases, but negative improvement over baseline can be observed in high-dimensional problems, implying poor scalability of local distance computations.
ConﬁdNet (Corbi`ere et al., 2019) learns to predict the class probability of true class with another
NN, while Introspection-Net (Aigrain & Detyniecki, 2019) utilizes the logit activations of the original NN classiﬁer to predict its correctness.
Since both models themselves are standard NNs, the conﬁdence scores returned by them may be arbitrarily high without any uncertainty information.
Moreover, existing approaches for error detection cannot differentiate natural misclassiﬁcation error from OOD or adversarial samples, making it difﬁcult to diagnose the sources of risks.
In contrast,
RED explicitly reports its uncertainty about the estimated conﬁdence score, providing more reliable error detection.
The uncertainty information returned by RED may also be helpful in clarifying the cause of classiﬁer mistakes, as will be demonstrated in this paper.
M ETHODOLOGY
This section gives the general problem statement, introduces the basic idea of original RIO, on which
RED is built, and describes the technical details of RED.
3. P ROBLEM S TATEMENT
Consider a training dataset D =
( X , y )
= { ( x i , y i ) }
Ni =1 , and a pretrained NN classiﬁer that outputs a predicted label
ˆ y i and class probabilities for each class σ i =
[ˆ p i, 1 , ˆ p
i, 2 , . . .
, ˆ p i,K ] given x i , where
N is the total number of training points and K is the total number of classes.
The problem is to develop a metric that can serve as a quantitative indicator for detecting natural misclassiﬁcation errors made by the pretrained NN classiﬁer.
3. RIO
The original RIO (Qiu et al., 2020) was developed to quantify point-prediction uncertainty in regression models.
More speciﬁcally, RIO ﬁts a Gaussian Process (GP) to predict the residuals, i.e. the differences between ground-truth and original model predictions.
It utilizes an I/O kernel, i.e. a composite of an input kernel and an output kernel, thus taking into account both inputs and outputs of the original regression model.
As a result, it measures the covariances between data points in both the original feature space and the original model output space.
For each new data point, a trained RIO model takes the original input and output of the base regression model, and predicts a distribution of the residual, which can be added back to the original model prediction to obtain both a calibrated prediction and the corresponding predictive uncertainty.
In the original RIO work, SVGP (Hensman et al., 2013; 2015) was used as an approximate GP to improve the scalability of the approach.
Both empirical results and theoretical analysis showed that RIO is able to consistently improve the prediction accuracy of base model as well as provide reliable uncertainty estimation.
Moreover, RIO can be directly applied on top of any pre-trained models without retraining or modiﬁcation.
It therefore forms a promising foundation for improving reliability of error detection metrics as well.
3. RIO FOR E RROR D ETECTION (RED)
Although RIO performs robustly in a wide variety of regression problems, it cannot be directly applied to classiﬁcation models.
A new framework, namely RED, is proposed to utilize RIO for error detection in classiﬁcation domains.
Building on the fact that the original maximum class probability is a strong baseline for error detection (Hendrycks & Gimpel, 2017; Haldimann et al., 2019), the main idea of RED is to derive a more reliable conﬁdence score by stacking RIO on top of the original maximum class probability.
Since
RIO was designed for single-output regression problems, it contains an output kernel only for scalar
[Figure]
Algorithm
Require:
( X , y )
= { ( x i , y i ) }
Ni =1 : training data ˆy = { ˆ y i } Ni =1 : labels predicted by original NN classiﬁer on training data σ = { σ i
=
[ˆ p i, 1 , ˆ p
i, 2 , . . .
, ˆ p i,K ] } Ni =1 : softmax outputs of original NN classiﬁer on training data ˆc
= { ˆ c i = max( σ i ) } Ni =1 : maximum class probability returned by original NN classiﬁer on training data x ∗ : data to be predicted σ ∗ : softmax outputs of original NN classiﬁer on x ∗
ˆ c ∗ : maximum class probability returned by original NN classiﬁer on x ∗ Ensure: ˆ c (cid:48)∗ ∼ N (ˆ c ∗ + ¯ˆ r ∗ , var(ˆ r ∗ ))
: ˆ c ∗ + ¯ˆ r ∗ can be used as conﬁdence score for error detection, and var(ˆ r ∗ ) represents the uncertainty of returned conﬁdence score
1: obtain target conﬁdence score c
= { c i =
δ y
i , ˆ y i } Ni =1 , where δ y i , ˆ y
i is the Kronecker delta ( δ y
i , ˆ y i =
if y i
= ˆ y i ,
otherwise δ y
i , ˆ y i = 0 ) 2: calculate residuals r
= { r i
=
c i − ˆ c i } Ni = 3: for each optimizer step do 4: calculate covariance matrix K c (( X , σ ) , ( X , σ )) , where each entry is given by k c (( x i , σ i ) , ( x j , σ j ))
=
k in ( x i , x j ) + k out ( σ i , σ j ) , for i, j = 1 , 2 , . .
, N 5: optimize GP hyperparameters by maximizing log marginal likelihood log p ( r |X , σ )
= − 12 r (cid:62) ( K c (( X , σ ) , ( X , σ ))
+ σ 2 n I ) − 1 r − 12 log | K c (( X , σ ) , ( X , σ ))
+ σ 2 n I | − n 2 log 2 π
Deployment Phase: 6: calculate residual mean ¯ˆ r ∗ =
k (cid:62)∗ ( K c (( X , σ ) , ( X , σ ))
+ σ 2 n I ) − 1 r and residual variance var(ˆ r ∗ ) =
k c (( x ∗ , σ ∗ ) , ( x ∗ , σ ∗ ))
− k (cid:62)∗ ( K c (( X , σ ) , ( X , σ ))
+ σ 2 n I ) − 1 k ∗ , where k ∗ denotes the vector of kernel-based covariances (i.e., k c ( x ∗ , x i ) ) between x ∗ and all the training points 7: return distribution of calibrated conﬁdence score ˆ c (cid:48)∗ ∼ N (ˆ c ∗ + ¯ˆ r ∗ , var(ˆ r ∗ )) outputs. In RED, this original output kernel is extended to multiple outputs, i.e. to vector outputs such as those of the ﬁnal softmax layer of a NN classifer, representing estimated class probabilities for each class. This modiﬁcation allows RIO to access more information from the classiﬁer outputs. The targets for RIO training need to be redesigned as well.
The raw targets of a classiﬁcation problem are the ground-truth labels; they are in categorical space, while RIO works in continuous space.
To solve this issue, RED constructs a different problem: Instead of predicting the labels directly, it predicts whether the original prediction is correct or not.
A target conﬁdence score is assigned to each training data point accordingly.
The residual between this target conﬁdence score and the originally returned maximum class probability is calculated, and a RIO model is trained to predict these residuals.
Given a new data point, the trained RIO model combined with the original
NN classiﬁer thus provides a calibrated conﬁdence score for detecting misclassiﬁcation errors.
Figure 1 illustrates the RED training and deployment processes conceptually, and Algorithm 1 speciﬁes them in detail.
In the training phase, the ﬁrst step is to deﬁne a target conﬁdence score
c
i for each training sample ( x i , y i ) .
For simplicity, all training samples that are correctly predicted by the original NN classiﬁer receive 1 as the target conﬁdence score, and those that are incorrectly predicted receive 0.
The validation dataset during the original NN training is included in the training dataset for RED.
After the target conﬁdence scores are assigned, a regression problem is formulated for the RIO model:
Given the original raw features { x i } Ni =1 and the corresponding softmax outputs of the original NN classiﬁer { σ i =
[ˆ p i, 1 , ˆ p
i, 2 , . . .
, ˆ p i,K ] } Ni =1 , predict the residuals r = { r i
=
c i − ˆ c i } ni =1 between target conﬁdence scores c = { c i } Ni =1 and the original maximum class probabilities ˆc
= { ˆ c i = max( σ i ) } Ni =1 .
The RIO model relies on an I/O kernel consisting of two components: the input kernel k in ( x i , x j ) , which measures covariances in the raw feature space, and the modiﬁed multi-output kernel k out ( σ i , σ j ) , which calculates covariances in the softmax output space.
The hyperparameters of the I/O kernel are optimized to maximize the log marginal likelihood log p ( r |X , σ ) .
In the deployment phase, given a new data point x ∗ , the trained RIO model provides a Gaussian distribution for the estimated residual ˆ r ∗ ∼ N (¯ˆ r ∗ , var(ˆ r ∗ )) .
By adding the estimated residual back to the original maximum class probability ˆ c ∗ , a distribution of calibrated conﬁdence score is obtained as ˆ c (cid:48)∗ ∼ N (ˆ c ∗ + ¯ˆ r ∗ , var(ˆ r ∗ )) .
The mean ˆ c ∗ + ¯ˆ r ∗ can be directly used as a quantitative metric for error detection, and the variance var(ˆ r ∗ ) represents the corresponding uncertainty of the conﬁdence score.
E MPIRICAL E VALUATION
In this section, the error detection performance of RED is evaluated comprehensively on 125 UCI datasets, comparing it to other related methods.
Its scale-up properties are then evaluated in a largescale deep learning architecture, and its potential to improve robustness more broadly in a case study involving OOD and adversarial samples.
4. E XPERIMENTS ON UCI DATASETS
As a comprehensive evaluation of RED, an empirical comparison with seven related approaches on 125 UCI datasets (Dua & Graff, 2017) was performed.
The reference approaches were maximum class probability (MCP) baseline (Hendrycks & Gimpel, 2017), Trust Score (Jiang et al., 2018), ConﬁdNet (Corbi`ere et al., 2019), Introspection-Net (Aigrain & Detyniecki, 2019), entropy of the original softmax outputs (Steinhardt & Liang, 2016), DNGO (Snoek et al., 2015), and the original SVGP (Hensman et al., 2013; 2015).
The 125 UCI datasets include 121 datasets used by Klambauer et al.
(2017) and four more recent ones.
Full details about the datasets and parametric setups of all tested algorithms, and a downloadable link for source codes are provided in Appendix A.1.
Following the experimental setup of Hendrycks & Gimpel (2017); Corbi`ere et al.
; Aigrain &
Detyniecki (2019), the task for each algorithm is to provide a conﬁdence score for each testing point.
An error detector can then use a predeﬁned ﬁxed threshold on this score to decide which points are probably misclassiﬁed by the original NN classiﬁer.
For RED, the mean of calibrated conﬁdence score ˆ c ∗ + ¯ˆ r ∗ was used as the reported conﬁdence score.
Five threshold-independent performance metrics were used to compare the methods: AUPR-Error, which computes the area under the Precision-Recall (AUPR) Curve when treating incorrect predictions as positive class during the detection; AUPR-Success, which is similar to AUPR-Error but uses correct predictions as positive class; AUROC, which computes the area under receiver operating characteristic (ROC) curve for the error detection task; AP-Error, which computes the average precision (AP) under different thresholds treating incorrect predictions as positive class; and AP-Success, which is similar to AP-Error but uses correct predictions as positive class.
AUPR and AUROC are commonly used in prior work (Hendrycks & Gimpel, 2017; Corbi`ere et al., 2019; Aigrain & Detyniecki, 2019), but as discussed by Davis & Goadrich (2006) and Flach & Kull (2015), AUPR may [Table]
[Table]
provide overly-optimistic measurement of performance.
To compensate for this issue, AP-Error and
AP-Success are included as additional metrics.
Since the target for the conﬁdence metrics is to detect misclassiﬁcation errors, the following discussion will focus more on AP-Error and AUPR-Error.
Ten independent runs were conducted for each dataset.
During each run, the dataset was randomly split into training dataset and testing dataset
[Table]
and a standard NN classiﬁer trained and evaluated on them.
The same dataset split and trained NN classiﬁer was used to evaluate all methods.
Full details of the experimental setup are provided in Appendix A.1.
Table 1 shows the ranks of each algorithm averaged over all 125 UCI datasets.
The rank of each algorithm on each dataset is based on the average performance over 10 independent runs.
RED performs best on all metrics; the performance differences between RED and all other methods are statistically signiﬁcant under paired t -test and Wilcoxon test.
Trust Score has the highest standard deviation, suggesting that its performance varies signiﬁcantly across different datasets.
As a more detailed comparison, Table 2 shows how often RED performs statistically signiﬁcantly better, how often the performance is not signiﬁcantly different, and how often it performs signiﬁcantly worse than the other methods.
RED is most often signiﬁcantly better, and very rarely worse.
In a handful of datasets Trust Score is better, but most often it is not.
To further illustrate the performance of RED and four state-of-the-art approaches, Figure 2 shows the distribution of the relative rank of RED, MCP baseline, Trust Score, ConﬁdNet and IntrospectionNet as a function of the number of samples and the number of features in the dataset.
These plots are based on the AP-Error metric; other metrics provide similar results.
RED performs consistently well over different dataset sizes and feature dimensionalities.
Trust Score performs best in several datasets, but occasionally also worst in both small and large datasets, making it a rather unreliable choice.
ConﬁdNet generally exhibits worse performance on datasets with large dataset sizes and high feature dimensionalities, i.e. it does not scale well to larger problems.
To investigate the robustness of RED when applying to other types of base models.
A NN classiﬁer using MC-dropout technique (Gal & Ghahramani, 2016) and a Bayesian Neural Network (BNN) classiﬁer (Wen et al., 2018) are trained as the base classiﬁers, RED is then applied to each of them.
Same experiments as in Table 2 are performed on 125 UCI datasets for both MC-dropout and BNN classiﬁers.
More implementation details are provided in Appendix A.1. Table 3 shows the pairwise [Figure] [Table]
[Table]
+ / = /
[Table]
+ / = /
[Table]
+ / = /
[Table]
+ / = /
[Table]
comparisons between RED and the internal conﬁdence scores returned by the base models.
MCP and Entropy represent the maximum class probability and entropy of softmax outputs, respectively, after averaging over 100 test-time samplings.
According to the results, RED signiﬁcantly improves over both MC-dropout and BNN classiﬁer in most datasets.
To conﬁrm that the RED approach scales up to large deep learning architectures, a VGG16 model (Simonyan & Zisserman, 2015) was trained on the CIFAR-10 dataset using a state-of-the-art training pipeline (see Appendix A.2 for details).
In order to remove the inﬂuence of feature extraction in image preprocessing and to make the comparison fair, all approaches used the same logit outputs of the trained VGG16 model as their input features.
10 independent runs are performed.
During each run, a VGG16 model is trained, and all the methods are evaluated based on this VGG16 model.
Table 4 shows the results on the two main error detection performance metrics.
It is notable that the numbers in Table 4 are the absolute values of the corresponding metrics.
Trust Score performs much better than in previous literatures (Corbi`ere et al., 2019).
This difference may be due to the fact that logit outputs are used as input features here, whereas Corbi`ere et al.
(2019) utilized a higher dimensional feature space for Trust Score.
Based on the results, RED signiﬁcantly outperforms all the counterparts in both metrics.
This result demonstrates the advantages of RED in scaling up to larger architectures.
4. A C ASE S TUDY WITH OOD AND A DVERSARIAL S AMPLES
In all experiments so far, the mean of calibrated conﬁdence score ˆ c ∗ + ¯ˆ r ∗ was used as RED’s conﬁdence score.
Although good performance is observed in error detection by only using the mean, the variance of calibrated conﬁdence score var(ˆ r ∗ ) may be helpful if the scenario is more complex, e.g., the dataset includes some OOD data, or even adversarial data. [Table] [Figure] mean/variance of score for annealing
[Figure] hand, indicates that RED is uncertain about its conﬁdence score, which can be used to identify OOD and adversarial samples. In this manner, RED can serve as a foundation for improving robustness of classiﬁers more broadly in the future. A preliminary investigation of RED in such scenaria was performed by manually adding OOD and adversarial data into the test set of the UCI ”annealing” dataset, on which RED detected errors well.
The synthetic OOD and adversarial samples were created to be highly deceptive, aiming to evaluate the performance of RED under difﬁcult circumstances.
The OOD data were sampled from a Gaussian distribution with mean 0 and variance 1, and the number of added OOD data was the same as the number of samples in the original test set.
Note that all data points from original dataset are normalized to have mean 0 and variance 1 for each feature dimension during preprocessing, so the OOD data and in-distribution data have similar scales.
The adversarial data was created by adding negligible modiﬁcations to training samples that the original NN classiﬁer predicted incorrectly with highest conﬁdence.
This process mimics an adversary that can arbitrarily alter the output of the NN classiﬁer with minuscule changes to the input (Goodfellow et al., 2014).
Figure 3 shows the distribution of mean and variance of calibrated conﬁdence scores for testing samples, including correctly and incorrectly labeled actual samples, as well as the synthetic OOD and adversarial samples.
The mean is a good separator for correctly classiﬁed and incorrectly classiﬁed samples, which tend to cluster on the top and bottom half of the image, respectively.
On the other hand, variance is a promising indicator of OOD and adversarial samples.
RED’s conﬁdence scores of in-distribution samples have low variance because they covary with the training samples.
The variance thus represents RED’s conﬁdence in its conﬁdence score.
Samples with large variance indicate that RED is uncertain about its conﬁdence score, which can be used as a basis for detecting
OOD and adversarial samples.
Although the main usage of RED is on misclassiﬁcation detection, the preliminary results in this subsection show the potential of extending RED for detecting broader error types.
D ISCUSSION AND F UTURE W ORK
One interesting observation from the experiments is that RED almost never performs worse than the MCP baseline.
This result suggests that there is almost no risk in applying RED on top of an existing NN classiﬁer.
Since RED is based on a GP model, the estimated residual ¯ˆ r ∗ is close to zero if the predicted sample is far from the distribution of the original training samples, resulting in no change to the original MCP.
In other words, RED does not make random changes to original
MCP if it is very uncertain about the predicted sample, and this uncertainty is explicitly represented in the variance of the estimated conﬁdence score.
This property makes RED a particularly reliable technique for error detection.
Another interesting observation is that the variance is also helpful in detecting OOD and adversarial samples.
This result follows from the design of the RIO uncertainty model.
Since RIO in RED has an input kernel and an output kernel, lower estimated variance requires that the predicted sample is close to training samples in both the input feature space and the classiﬁer output space.
This requirement is difﬁcult for OOD and adversarial attacks to achieve, providing a basis for detecting them.
In real-world deployment, a threshold is required to be pre-deﬁned for triggering error warnings based on conﬁdence scores returned by RED.
A practical way is to use a validation dataset for investigating how the tradeoffs between precision and recall rate in terms of error detection change over different thresholds.
The users can then decide the threshold based on their preference on the precision-recall tradeoff.
The most compelling direction of future work is to extend this capability of RED further.
Instead of using a single dimensional conﬁdence score for error detection, it is possible to use mean and variance simultaneously, leading to a two dimensional detection space.
Further separation between
OOD and adversarial samples may be possible by adding one more dimension, such as the ratio between input kernel output and output kernel output.
Also, instead of using a hard target conﬁdence score (i.e. either 0 or 1), it may be possible to deﬁne a soft target conﬁdence that may be more informative.
Further, RED may be used to calibrate other existing conﬁdence metrics, such as the
Trust Score, which may lead to a further improvement in detection performance.
C ONCLUSION
This paper introduced a new framework, RED, for error detection in neural network classiﬁers that can produce a more reliable conﬁdence score than previous methods.
RED is able to not only provide a calibrated conﬁdence score, but also report the uncertainty of the estimated conﬁdence score.
Experimental results show that RED’s scores consistently outperform state-of-the-art methods in separating the misclassiﬁed samples from correctly classiﬁed samples.
Preliminary experiments also demonstrate that the approach scales up to large deep learning architectures, and can form a basis for detecting OOD and adversarial samples as well.
It is therefore a promising foundation for improving robustness of neural network classiﬁers.
A A PPENDIX A. E XPERIMENTAL S ETUPS FOR S ECTION 4.
General Information 10 indenpendent runs are performed for each dataset.
During each run, the dataset is randomly split into a training set ( 80% ) and a testing set ( 20% ), then a fully connected feedforward NN classiﬁer with 2 hidden layers, each with 64 hidden neurons, are trained on the training set.
The activation function is ReLU for all the hidden layers.
The maximum number of epochs for training is 1000.
20% of the training set is used as validation set, and the split is random at each independent run.
An early stop is triggered if the loss on validation set has not be improved for 10 epochs.
The optimizer is Adam with learning rate 0.001, β 1 = 0 .
9 , and β 2 = 0 .
The loss function is cross entropy loss.
During each independent run, the same random dataset split and trained base NN classiﬁer is used for evaluating all algorithms.
All source codes for reproducing the experimental results can be downloaded from (https://drive.google.com/drive/folders/1X5R6sEkjmucR7B4MvFrY6QnCvos6a7n?usp=sharing).
Dataset Description
In total, 125 UCI datasets are used in the experiments, among which 121 are from Klambauer et al.
(2017), and 4 are recent datasets released in Dua & Graff (2017).
All features in all datasets are normalized to have mean 0 and standard deviation 1.
Full details regarding the number of samples N, number of features M, and number of classes K for each dataset are shown in
Table 5.
Parametric Setup for Algorithms
• RED: SVGP (Hensman et al., 2013; 2015) is used as an approximator to original GP.
The number of inducing points are 50.
RBF kernel is used for both input and multi-output kernel.
Automatic Relevance Determination (ARD) feature is turned on.
The signal variances and length scales of all the kernels plus the noise variance are the trainable hyperparameters.
The optimizer is L-BFGS-B with default parameters as in Scipy.optimize documentation (https://docs.scipy .org/doc/scipy/reference/optimize.minimize-lbfgsb.html), and the maximum number of iterations is set as 1000.
The optimization process runs until the L-BFGS-B optimizer decides to stop.
To overcome the sensitivity of GP optimization to initialization of the hyperparameters (Ulapane et al., 2020), 20 random initialization of the hyperparameters are tried for each independent run.
For each random initialization, the signal variances are generated from a uniform distribution within interval [0 , 1] , and the lengthscales are generated from a uniform distribution within interval [0 , 10] .
For 10 initializations, the hyperparameters of input kernel are ﬁrst optimized while the multi-output kernel is temporarily turned off, then after the optimizer stops, the multi-output kernel is turned on, and both the two kernels are optimized simultaneously.
For the other 10 initializations, both kernels are optimized simultaneouly from the start.
The average performance of the 3 best optimized model in terms of corresponding metrics are used as the ﬁnal performance of RED on each independent run.
During our preliminary investigation, several statistic metrics on training set is effective in picking the true best-performing model out of these 20 trials, e.g., the gap between average estimated conﬁdence scores of correctly classiﬁed training samples and incorrectly classiﬁed training samples, the scale of optimized noise variance of SVGP model, the ratio between sum of signal variances and noise variance after optimization, etc.
Since improving initialization and optimization of GP hyperparameters is out of the scope of this work, we simply use average performance of the best 3 models (
top
% ) in comparison
• MCP baseline: the maximum class probability of softmax outputs of the base NN classiﬁer is used as the conﬁdence score of MCP baseline.
Setups of the base NN classiﬁer is provided above.
• Trust Score: k=10, α = 0 , without ﬁltering.
This is the same as the default setup in https://github.com/google/TrustScore.
ConﬁdNet: During training, the input to ConﬁdNet is the raw feature, and the target is the class probability of the ground-truth class returned by base NN classiﬁer.
The architecture of ConﬁdNet is a fully connected feed-forward NN regressor with 2 hidden layers, each with 64 hidden neurons.
The activation function is ReLU for all the hidden layers.
The maximum number of epochs for training is 1000.
An early stop is triggered if the loss on validation data has not be improved for 10 epochs.
The optimizer is RMSprop with learning rate 0.001, and the loss function is mean squared error (MSE).
• Introspection-Net: During training, the input to Introspection-Net is the logit outputs of base NN classiﬁer, and the target is 1 for correctly classiﬁed sample, and 0 for incorrectly classiﬁed sample.
The architecture of ConﬁdNet is a fully connected feed-forward NN regressor with 2 hidden layers, each with 64 hidden neurons.
The activation function is ReLU for all the hidden layers.
The maximum number of epochs for training is 1000.
An early stop is triggered if the loss on validation data has not be improved for 10 epochs.
The optimizer is RMSprop with learning rate 0.001, and the loss function is mean squared error (MSE).
• Entropy: the entropy of softmax outputs of the base NN classiﬁer is used as the conﬁdence score of Entropy. Setups of the base NN classiﬁer is provided above. • DNGO: a Bayesian linear regression layer same as in (Snoek et al., 2015) is added after the logits layer of the original NN classiﬁer to predict whether an original prediction is correct or not (1 for correct and 0 for incorrect). Default parametric setups as in https://github.com/automl/pybnn/blob/master/pybnn/dngo.py is used.
• SVGP: the original SVGP without output kernel is used to directly predict whether a prediction made by the base NN classiﬁer is correct or not (1 for correct and 0 for incorrect).
All other parametric setups are identical with those in RED.
• BNN MCP: the standard dense layers in the base NN classiﬁer described in RED setups above is replaced with Flipout layers (Wen et al., 2018). All the other setups are identical with those in RED setups. The maximum class probability averaging over 100 test-time samplings is used as the conﬁdence score for error detection. • BNN Entropy: same setup as BNN MCP, except that now the entropy of softmax outputs averaging over 100 test-time samplings is used as conﬁdence score for error detection.
• MC-Dropout MCP: a dropout layer with dropout rate 0.5 is added after each dense layer of the base NN classiﬁer described in RED setups. All the other setups are identical with those in RED setups.
The maximum class probability averaging over 100 test-time Monte-Carlo samplings is used as the conﬁdence score for error detection. • MC-Dropout Entropy: same setup as MC-Dropout MCP, except that now the entropy of softmax outputs averaging over 100 test-time Monte-Carlo samplings is used as conﬁdence score for error detection. A. E XPERIMENTAL S ETUPS FOR S ECTION 4.
Setup of VGG16 Training The standard VGG16 architecture (Simonyan & Zisserman, 2015) is used.
The training pipeline is using the same default setups as in https://github.com/geifmany/cifarvgg/blob/master/cifar10vgg.py.
For CIFAR-10 dataset, 40000 samples are used as training set, 10000 as validation set, and 10000 as testing set.
Parametric Setup for Algorithms All algorithms use the logit outputs of the trained VGG model as input features.
The maximum class probability of softmax outputs of the trained VGG model is used as the conﬁdence score of MCP baseline.
The parametric setups for RED, Trust
Score, Entropy, DNGO and SVGP are identical with the UCI experiments.
For ConﬁdNet and Introspection-Net, all parametric setups are the same as in UCI experiments, except for that the number of hidden neurons for all hidden layers is incresed to 128.
A. D ETAILED R ESULTS FOR S ECTION 4.
This subsection shows all the detailed results for experiments performed in Section 4.1.
The re- sults averaging over 10 independent runs in terms of AP-Error, AP-Success, AUPR-Error, AUPRSuccess, and AUROC are shown in Table 5, Table 6, Table 7, Table 8, and Table9, respectively.
The column ”N”, ”M”, and ”K” denotes the number of samples, number of features, and number of classes for corresponding datasets.
For datasets that the original NN classiﬁer achieves 100% accuracy, the entries are marked as ”NA”.
For dataset splits that the number of samples in one particular class is too small for Trust Score to calculate neighborhood distance, the entries are marked as ”NA”.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
087 climate
[Table]
[Table]
[Table] [Table]
[Table]
[Table]
[Table]
rocks   0.
[Table]
0.170 connect-
[Table]
0.395 contrac   0.
[Table]
approval   0.
[Table]
[Table] [Table]
[Table]
422 dermatology
[Table]
253 echocardiogram
[Table] [Table]
407 ecoli
[Table] [Table] [Table]
284 energy-
[Table]
[Table] [Table] [Table]
[Table]
187 energy-
[Table]
[Table]
[Table] [Table] 278 fertility [Table]
[Table] [Table] [Table]
[Table] [Table] Continued on next page. [Table]
0.202 led-display   0.
0.512 lenses   0.
0.790 letter   0.
0.081 libras   0.
0.301 low-res-spect   0.
0.305 lung-cancer   0.
0.841 lymphography   0.
0.386 magic   0.
0.350 mammographic   0.
0.454 messidor   0.
0.426 miniboone   0.
0.371 molec-biol-promoter   0.
0.505 molec-biol-splice   0.
0.208 monks0.
0.343 monks0.
[Table] [Table]
[Table] [Table]
MATERIAL
[Table] [Table]
[Table] [Table]
[Table]
[Table]
planning
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
721 ringnorm
[Table]
[Table] [Table]
116 seeds
[Table] [Table]
333 semeion
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
117 teaching
[Table] [Table]
thyroid
[Table]
[Table] [Table] [Table] [Table]
toe
[Table] [Table] [Table] [Table]
[Table]
titanic
[Table]
[Table]
[Table]
[Table]
[Table]
Continued on next page.
[Table]
Continued from previous page.
[Table]
breast-tissue   0.
0.767 car   0.
0.988 cardiotocography-10clases   0.
0.872 cardiotocography-3clases   0.
0.964 chess-krvk   0.
0.838 chess-
[Table]
0.994 climate   0. 0.976 congressional-voting   0. 0.684 conn-
[Table]
-sonar-mines-rocks   0.
0.901 conn-
[Table] [Table]
-deterding   1. 0.984 connect0. 0.958 contrac   0. 0.675 credit-approval   0.
0.936 cylinder-bands   0. 0.809 dermatology   0. 0.981 echocardiogram   0.
0.924 ecoli   0. 0.879 energy-y    0. 0.974 energy-y    0. 0.946 fertility   0.
0.929 ﬂags   0. 0.617 glass   0. 0.738 haberman-survival   0. 0.835 hayes-roth   0.
0.864 heart-cleveland   0. 0.822 heart-hungarian   0. 0.915 heart-switzerland   0. 0.452 heart-va   0.
0.504 hepatitis   0. 0.896 hill-valley   0. 0.741 horse-colic   0. 0.913 ilpd-indian-liver   0.
0.863 image-segmentation   0.
0.984 ionosphere   0.
0.973 iris   0.
0.917 led-display   0.
0.850 lenses   0.
0.883 letter   0.
0.947 libras   0.
0.807 low-res-spect   0.
0.892 lung-cancer   0.
Continued on next page.
[Table]
0.678 statlog-german-credit   0.
0.850 statlog-heart   0.
0.917 statlog-image   0.
0.980 statlog-landsat   0.
0.959 statlog-shuttle   1.
0.998 statlog-vehicle   0.
[Table]
-plates   0.
[Table]
-control   1.
0.988 teaching   0.
0.640 thyroid   0.
0.984 tic-tac-toe   1.
0.998 titanic   0.
0.873 trains   0.
0.750 twonorm   0.
0.991 vertebral-column-2clases
[Table]
[Table]
[Table]
0.369 ilpd-indian-liver   0.
0.428 image-segmentation   0.
0.200 ionosphere   0.
0.364 iris   0.
0.173 led-display   0.
0.512 lenses   0.
0.759 letter   0.
0.125 libras   0.
0.282 low-res-spect   0.
0.288 lung-cancer   0.
0.812 lymphography   0.
0.339 magic   0.
0.348 mammographic   0.
0.441 messidor   0.
0.417 miniboone   0.
0.370 molec-biol-promoter   0.
0.436 molec-biol-splice   0.
0.203 monks0.
0.305 monks0.
0.412 monks0.
0.403 mushroom
NA NA
NA NA
NA musk0.
0.319 musk0.
0.061 nursery   0.
0.129 oocytes merluccius nucleus 4d   0.
0.368 oocytes merluccius states 2f   0.
0.338 oocytes trisopterus nucleus 2f   0.
0.447 oocytes trisopterus states 5b   0.
0.266 optical   0.
0.037 ozone   0.
0.127 page-blocks   0.
Continued on next page.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
teaching
[Table] [Table] [Table]
[Table] [Table]
toe
[Table] [Table] [Table]
461 titanic
[Table] [Table]
trains
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
noise
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
Continued on next page.
[Table]
0.912 led-display   0.
0.849 lenses   0.
0.845 letter   0.
0.888 libras   0.
0.800 low-res-spect   0.
0.889 lung-cancer   0.
0.703 lymphography   0.
0.851 magic   0.
0.959 mammographic   0.
0.910 messidor   0.
0.829 miniboone   0. 0.986 molec-biol-promoter   0. 0.873 molec-biol-splice   0. 0.845 monks0.
0.989 monks0.
[Table]
[Table]
[Table] [Table] [Table] 945 pima
[Table]
[Table]
[Table] [Table] [Table] [Table]
-MATERIAL [Table] [Table] [Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
SPAN
[Table]
[Table]
[Table]
[Table]
bridges
[Table]
OR
[Table]
[Table] [Table] [Table]
[Table]
bridges
[Table]
TYPE
[Table]
planning
[Table]
[Table] [Table]
[Table] [Table]
-texture
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
-tumor
[Table] [Table]
623 ringnorm
[Table] [Table]
[Table]
977 seeds
[Table]
[Table] [Table]
[Table]
960 semeion
[Table]
Continued on next page.
[Table] [Table]
NA
[Table]
NA
NA NA
NA [Table] [Table]
[Table] [Table]
[Table]
NA
[Table] [Table] [Table]
[Table]
[Table] [Table] [Table]
[Table]
[Table] [Table]
[Table] 651 car [Table]
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
approval
[Table] [Table]
dermatology
[Table] [Table]
0.766 echocardiogram   0. 0.687 ecoli   0. 0.556 energy-y    0.
0.650 energy-y    0. 0.668 fertility   0. 0.485 ﬂags   0.
0.618 glass   0.
0.560 haberman-survival   0. [Table] -roth   0. [Table]
cleveland   0.
[Table] 0.769 heart-hungarian   0. 0.730 heart-switzerland   0. Continued on next page.
[Table] [Table]
700 lymphography [Table]
[Table]
[Table]
[Table]
[Table]
622 magic [Table]
[Table]
796 mammographic [Table]
[Table] 742 messidor
[Table] [Table] [Table] [Table]
882 molec- [Table] -promoter [Table]
[Table]
719 molec
[Table]
biol [Table] splice
[Table] monks- [Table] [Table]
monks-
[Table]
[Table]
monks [Table] [Table] [Table]
[Table]
872 mushroom
NA NA
NA NA
NA musk- [Table] [Table] [Table]
[Table]
[Table]
[Table] 724 musk- [Table] [Table]
[Table]
[Table] [Table] 627 nursery [Table]
[Table] [Table] [Table] 603 oocytes merluccius nucleus 4d
[Table]
[Table]
[Table] [Table]
[Table] 0.731 statlog
[Table]
image   0.
0.683 statlog-landsat   0.
0.770 statlog-shuttle   0.
0.589 statlog
[Table]
vehicle   0.
[Table]
[Table]
-control   0.
0.547 teaching   0.
0.574 thyroid   0.
0.646 tic-tac-toe   0.
0.894 titanic   0.
0.671 trains   0.
0.500 twonorm   0.
0.812 vertebral
[Table]
2clases   0.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Continued on next page.
[Table]
[Table]