A C ALL TO R EFLECT ON E VALUATION P RACTICES FOR F AILURE D ETECTION IN I MAGE C LASSIFICATION
A BSTRACT
Reliable application of machine learning-based decision systems in the wild is one of the major challenges currently investigated by the field.
A large portion of established approaches aims to detect erroneous predictions by means of assigning confidence scores.
This confidence may be obtained by either quantifying the model’s predictive uncertainty, learning explicit scoring functions, or assessing whether the input is in line with the training distribution.
Curiously, while these approaches all state to address the same eventual goal of detecting failures of a classifier upon real-world application, they currently constitute largely separated research fields with individual evaluation protocols, which either exclude a substantial part of relevant methods or ignore large parts of relevant failure sources.
In this work, we systematically reveal current pitfalls caused by these inconsistencies and derive requirements for a holistic and realistic evaluation of failure detection.
To demonstrate the relevance of this unified perspective, we present a large-scale empirical study for the first time enabling benchmarking confidence scoring functions w.r.t. all relevant methods and failure sources.
The revelation of a simple softmax response baseline as the overall best performing method underlines the drastic shortcomings of current evaluation in the abundance of publicized research on confidence scoring.
Code and trained models are at https://github.com/IML-DKFZ/fd-shifts .
I NTRODUCTION
“Neural network-based classifiers may silently fail when the test data distribution differs from the training data.
For critical tasks such as medical diagnosis or autonomous driving, it is thus essential to detect incorrect predictions based on an indication of whether the classifier is likely to fail”.
Such or similar mission statements prelude numerous publications in the fields of misclassification detection (MisD)
(Corbière et al., 2019;
Hendrycks and Gimpel, 2017; Malinin and Gales, 2018),
Out-of-Distribution detection (OoD-D)
(Fort et al., 2021; Winkens et al., 2020; Lee et al., 2018;
Hendrycks and Gimpel, 2017; DeVries and Taylor, 2018; Liang et al., 2018), selective classification (SC) (Liu et al., 2019; Geifman and El-Yaniv, 2019; 2017), and predictive uncertainty quantification (PUQ) (Ovadia et al., 2019; Kendall and Gal, 2017), hinting at the fact that all these approaches aim towards the same eventual goal: Enabling safe deployment of classification systems by means of failure detection , i.e. the detection or filtering of erroneous predictions based on ranking of associated confidence scores .
In this context, any function whose continuous output aims to separate a classifier’s failures from correct predictions can be interpreted as a confidence scoring function (CSF) and represents a valid approach to the stated goal.
This holistic perspective on failure detection reveals extensive shortcomings in current evaluation protocols, which constitute major bottlenecks in progress toward the goal of making classifiers suitable for application in real-world scenarios.
Our work is an appeal to corresponding communities to reflect on current practices and provides a technical deduction of a unified evaluation protocol, a list of empirical insights based on a large-scale study, as well as hands-on recommendations for researchers to catalyze progress in the field.
Detecting failures should be seen in the context of the overarching goal of preventing silent failures of a classifier , which includes two tasks: preventing failures in the first place as measured by the "robustness" of a classifier (Task 1), and detecting the non-prevented failures by means of CSFs (Task 2, focus of this work).
For failure prevention across distribution shifts, a consistent task formulation exists (featuring accuracy as the primary evaluation metric) and various benchmarks have been released covering a large variety of realistic shifts (e.g. image corruption shifts, sub-class shifts, or domain shifts). In contrast, progress in the subsequent task of detecting the non-prevented failures by means of CSFs is currently obstructed by three pitfalls: 1) A diverse and inconsistent set of evaluation protocols for CSFs exists (MisD, SC, PUQ, OoD-D) impeding comprehensive competition. 2) Only a fraction of the spectrum of realistic distribution shifts and thus potential failure sources is covered diminishing the practical relevance of evaluation.
The task formulation in OoD-D fundamentally deviates from the stated purpose of detecting classification failures.
Overall, the holistic perspective on failure detection reveals an obvious need for a unified and comprehensive evaluation protocol, in analogy to current robustness benchmarks, to make classifiers fit for safety-critical applications.
Abbreviations: CSF: Confidence
Scoring Function, OoD-D: Out-of-Distribution Detection, MisD: Misclassification Detection, PUQ
Predictive Uncertainty Quantification, SC: Selective Classification.
P ITFALLS OF CURRENT EVALUATION PRACTICES
Figure 1 gives an overview of the current state of failure detection research and its relationship to the preceding failure prevention task, which is measured by classifier robustness.
This perspective reveals three main pitfalls, from which we derive three requirements R1 - R3 for a comprehensive and realistic evaluation in failure detection:
Pitfall 1: Heterogeneous and inconsistent task definitions.
To achieve a meaningful evaluation, all relevant solutions toward the stated goal must be part of the competition.
In research on failure detection, currently, four separate fields exist each evaluating proposed methods with their individual metrics and baselines.
Incomplete competition is first and foremost an issue of historically evolved delimitations between research fields, which go so far that employed metrics are by design restricted to certain methods.
MisD:
Evaluation in MisD (see Section B.2.1 for a formal task definition) exclusively measures discrimination of a classifier’s success versus failure cases by means of ranking metrics such as AUROC f 1 (Hendrycks and Gimpel, 2017; Jiang et al., 2018; Corbière et al., 2019;
Bernhardt et al., 2022).
This protocol excludes a substantial part of relevant CSFs from comparison, because any CSF that affects the underlying classifier (e.g. by introducing dropout or alternative loss functions) alters the set of classifier failures, i.e. ground truth labels, and thus creates their individual test set (for a visualization of this pitfall see Figure 4).
As an example, a CSF that negatively affects the accuracy of a classifier might add easy-to-detect failures to its test set and benefit in the form of high AUROC f scores.
As depicted in Figure 1, we argue that the task of detecting failures is no self-purpose, but preventing and detecting failures are two sides of the same coin when striving to avoid silent classification failures.
Thus, CSFs should be evaluated as part of a symbiotic system with the associated classifier.
While additionally reporting classifier accuracy associated with each CSF renders these effects transparent, it requires nontrivial weighting of the two metrics when aiming to rank CSFs based on a single score.
PUQ: Research in PUQ often remains vague about the concrete application of extracted uncertainties stating the purpose of "meaningful confidence values" (Ovadia et al., 2019; Lakshminarayanan et al., 2017) (see Appendix B.2.3 for a formal task definition), which conflates the related but independent use-cases of failure detection and confidence calibration .
This (arguably vague) goal is reflected in the evaluation, where typically strictly proper scoring rules (Gneiting and Raftery, 2007) such as negative log-likelihood assess a combination of ranking and calibration of scores.
However, for failure detection use cases, arguably an explicit assessment of failure detection performance is desired (see Appendix C for a discussion on how calibration relates to failure detection).
Furthermore, these metrics are specifically tailored towards probabilistic predictive outputs such as softmax classifiers and exclude all other CSFs from comparison.
→ Requirement 1 (R1): Comprehensive evaluation requires a single standardized score that applies to arbitrary CSFs while taking into account their effects on the classifier.
Pitfall 2:
Ignoring the major part of relevant failure sources.
As stated in the introductory quote, research on failure detection typically expects classification failures to occur when inputs upon application differ from the training data distribution.
As shown in Figure 1, we distinguish "covariate shifts" (label-preserving shifts) versus "new-class shifts" (label-altering shifts).
For a detailed formulation of different failure sources, see Appendix A.
The fact that in the related task of preventing failures a myriad of nuanced covariate shifts have been released on various data sets and domains (Koh et al., 2021; Santurkar et al., 2021;
Hendrycks and Dietterich, 2019; Liang and Zou, 2022; Wiles et al., 2022) to catalyze real-world progress of classifier robustness begs the question: If simulating realistic classification failures is such a delicate and extensive effort, why are there no analogous benchmarking efforts in the research on detecting failures ?
In contrast, CSFs are currently almost exclusively evaluated on i.i.d.
test sets ( MisD, PUQ, SC ).
Exceptions (see hatched areas in Figure 1) are a PUQ study that features corruption shifts (Ovadia et al., 2019), or SC evaluated on sub-class shift (comparing a fixed CSF under varying classifiers)
(Tran et al., 2022) and applied to question answering under domain shift (Kamath et al., 2020).
Further, research in OoD-D (see
Section B.2.2 for a formal task definition) exclusively evaluates methods under one limited fraction of failure sources: new-class shifts (see images 7 and 8 in Figure 2 (right panel)).
A recent trend in this area is to focus on "near OoD" scenarios, i.e. shifts affecting semantic image features but leaving the context unchanged (Winkens et al., 2020; Fort et al., 2021;
Ren et al., 2021).
While the notion that nuanced shifts might bear more practical relevance compared to vast context switches seems reasonable, the term "near" is misleading, as it ignores the whole spectrum of even "nearer" and thus potentially more relevant covariate shifts, which OoD-D methods are not tested against.
We argue that for most applications, it is not realistic to exclusively assume classification failures from label-altering shifts and no failures caused by label-preserving shifts.
→ Requirement 2 (R2): Analogously to robustness benchmarks, progress in failure detection requires to evaluate on a nuanced and diverse set of failure sources.
Pitfall 3: Discrepancy between the stated purpose and evaluation.
The described limitations of
OoD-D evaluation are only symptoms of a deeper rooted problem: Methods are not tested to predict failures of a classifier, but instead to predict an external, i.e. classifier-agnostic "outlier" label.
In some cases, this formulation reflects the inherent nature of the given problem, such as in anomaly detection, where no underlying task is defined and the data sets are potentially unlabeled (Ruff et al., 2021).
However, the majority of work on OoD-detection comes with a defined classification task, including training labels and states detecting failures of the classifier as their main purpose (Fort et al., 2021; Winkens et al., 2020; Lee et al., 2018;
Hendrycks and Gimpel, 2017; DeVries and Taylor, 2018; Liang et al., 2018).
Yet, this line of work falls short of justifying why associated methods are subsequently not shown to detect the said failures but are instead tested on the surrogate task of detecting distribution shifts in the data.
Figure 2 shows that the outlier label constitutes a poor tool to define which cases we wish to filter because the question "what is an outlier?" is highly subjective for covariate shifts (see purple question marks).
The ambiguity of the label extends to the concept of "inliers" (what extent of data variation is still considered i.i.d.?), which the protocol rewards to retain irrespective of whether they cause the classifier to fail (see purple lightning).
→ Requirement 3 (R3): If there is a defined classifier whose incorrect predictions are to be detected, its respective failure information should be used to assess CSFs w.r.t the stated purpose instead of a surrogate task such as distribution shift detection .
U NIFIED TASK FORMULATION
Parsing the quoted purpose statement at the start of Section 1 results in the following task formulation:
Given a data set { ( x i , y cl ,i ) } Ni =1 of size N with ( x i , y cl ,i ) independent samples from X × Y and y cl the class label, and given a pair of functions ( m, g ) , where g : X → R is a CSF and m ( · , w ) : X → Y is the classifier including model parameters w , the classification output after failure detection is defined as: [Equation] filter [Equation]
otherwise
[Equation]
Filtering ("detection") is triggered when g ( x ) falls below a threshold τ .
In order to perform meaningful failure detection, a CSF g ( x ) is required to output high confidence scores for correct predictions and low confidence scores for incorrect predictions based on the binary failure label
[Equation]
where ˆ y m = argmax c ∈Y P m ( y cl =
c | x, w ) and I is the identity function (1 for true events and 0 for false events).
Despite accurately formalizing the stated purpose of numerous methods from MisD, OoD-D, SC, and
PUQ, and allowing for evaluation of arbitrary CSFs g ( x ) , this generic task formulation is currently only stated in SC research (see Appendix B for a detailed technical description of all protocols considered in this work).
To deduct an appropriate evaluation metric for the formulated task, we start with the ranking requirement on g ( x ) , which is assessed e.g. by AUROC f in MisD leading to Pitfalls described in Section 2.
Following R1 and modifying AUROC f to take classifier performance into account lets us naturally converge (see Appendix B.2.5 for the technical process) to a metric that has previously been proposed in SC as a byproduct, but is not widely employed for evaluation (Geifman et al., 2019): the Area under the-Risk-Coverage-Curve (AURC, see Equation 31).
We propose to use AURC as the primary metric for all methods with the stated purpose of failure detection, as it fulfills all three requirements R1 - R3 in a single score.
The inherently joint assessment of classifier accuracy and CSF ranking performance comes with a meaningful weighting between the two aspects, eliminating the need for manual (and potentially arbitrary) score aggregation.
AURC measures the risk or error rate ( 1 − Accuracy ) on the non-filtered cases averaged over all filtering thresholds (score range: [0,1], lower is better) and can be interpreted as directly assessing the rate of silent failures occurring in a classifier .
While this metric enables a general evaluation of CSFs, depending on the use case, a more specific assessment of certain coverage regions (i.e. the ratio of remaining cases after filtering) or even single risk-coverage working points might be appropriate.
In Appendix F we provide an open source implementation of AURC fixing several shortcomings of previous versions.
3. R EQUIRED MODIFICATIONS FOR CURRENT PROTOCOLS
The general modifications necessary to shift from current protocols to a comprehensive and realistic evaluation of failure detection, i.e. to fulfill requirements R1-R3 , are straightforward for considered fields ( SC , MisD , PUQ , OoD-D ): Researchers may simply consider reporting performance in terms of AURC and benchmark proposed methods against relevant baselines from all previously separated fields as well as on a realistic variety of failure sources (i.e. distribution shifts).
An additional aspect needs to be considered for SC , where the task is to solve failure prevention and failure detection at the same time (see Task 1 and Task 2 in Figure 1), i.e. the goal is to minimize absolute AURC scores.
This setting includes studies that compare different classifiers while fixing the CSF (Tran et al., 2022).
On the contrary, the evaluation of failure detection implies a focus on the performance of CSFs (Task 2 in Figure 1) while monitoring the classifier performance as a requirement ( R1 ) to ensure a fair comparison of arbitrary CSFs.
This shift of focus is reflected in the fact that the classifier architecture, as well as training procedure, are to be fixed (with some exceptions as described in Appendix E.4) across all compared CSFs.
This way, external variations in classifier configuration are removed as a nuisance factor from CSF evaluation and the direct effect of the CSF on the classifier training is isolated to enable a relative comparison of AURC scores.
For evaluation of new-class shifts (as currently performed in OoD-D ), a further modification is required:
The current OoD-D protocol rewards CSFs for not detecting inlier misclassifications (see
Figure 2).
On the other hand, penalizing CSFs for not detecting these cases (as handled by AURC) would dilute the desired evaluation focus on new-class shifts.
Thus, we propose to remove inlier misclassifications from evaluation when reporting a CSF’s performance under new-class shift.
Figure 5 visualizes the proposed modification.
Notably, the proposed protocol does still consider the CSF’s effect on classifier performance (i.e. does not break with R1 ), since higher classifier accuracy will remain to cause higher AURC scores (see Equations 29-31).
3. O WN CONTRIBUTIONS IN THE PRESENCE OF S ELECTIVE C LASSIFICATION
Given the fact that the task definition in Equation 1 as well as AURC, the metric advocated for in this paper, have been formulated before in SC literature (see Appendix B.2.4 for technical details on current evaluation in SC)), it is important to highlight that the relevance of our work is not limited to advancing research in SC, but, next to the shift of focus on CSFs described in Section 3.1, we articulate a call to other communities (MisD, OoD-D, PUQ) to reflect on current practices.
In other words, the relevance of our work derives from providing evidence for the necessity of the SC protocol in previously separated research fields and from extending their scope of evaluation (including the current scope of SC) w.r.t. compared methods and considered failure sources.
E MPIRICAL S TUDY
To demonstrate the relevance of a holistic perspective on failure detection, we performed a large-scale empirical study, which we refer to as FD-shifts 2 .
For the first time, state-of-the-art CSFs from MisD,
OoD-D, PUQ, and SC are benchmarked against each other.
And for the first time, analogously to recent robustness studies, CSFs are evaluated on a nuanced variety of distribution shifts to cover the entire spectrum of failure sources.
4. U TILIZED D ATA S ETS
Appendix E features details of all used data sets, and Appendix A describes the considered distribution shifts.
FD-Shift benchmarks CSFs on CAMELYON-17-Wilds (Koh et al., 2021), iWildCam-2020Wilds (Koh et al., 2021), and BREEDS-ENTITY-13 (Santurkar et al., 2021), which have originally been proposed to evaluate classifier robustness (Task 1 in Figure 1) under sub-class shift in various domains.
Further sub-class shifts are considered in the form of super-classes of CIFAR-100 (Krizhevsky,
2009), where one random class per super-class is held out during training.
For studying corruption shifts, we report results on the 15 corruption types and 5 intensity levels proposed by Hendrycks and Dietterich (2019) based on CIFAR-10 as well as CIFAR-100.
Regarding new-class shifts, we test on SVHN (Netzer et al., 2011), CIFAR-10/100 and TinyImagenet (Le and Yang, 2015) in a rotating fashion while considering the shift between CIFAR data sets as semantic and other shifts as non-semantic .
Finally, we create additional semantic new-class shift scenarios by testing on held-out training classes (randomly sampled 40% of all training classes) on SVHN and iWildCam-2020-Wilds.
C OMPARED M ETHODS
We compare the following CSFs:
The maximum softmax response (MSR) calculated from the classifier’s softmax output.
PUQ: Predictive entropy based on softmax output (PE) and three predictive uncertainty measures based on Monte Carlo Dropout (MCD) (Gal and Ghahramani, 2016): mean softmax (MCD-MSR), predictive entropy (MCD-PE), and expected entropy (MCD-EE) (for technical formulations see Appendix I).
For MCD we take 50 samples at test time.
MisD:
We include ConfidNet (Corbière et al., 2019), which is trained as an extension to the classifier and uses its regressed true class probability as a CSF.
SC:
We include DeepGamblers (DG), which uses loss attenuation based on portfolio theory to learn a confidence-like reservation score (DG-RES) (Liu et al., 2019).
As the loss attenuation of DG’s training paradigm might have a positive effect on the classifier itself, we additionally evaluate the softmax output (DG-MCD-MSR).
OoD-D: We include the work of DeVries and Taylor (2018) 3 .
Notably, ConfidNet, DG, and the work by Devries et al. are excellent examples of artificial separation of previous evaluations, as all three have not been compared before, despite their strong conceptual and technical similarities.
We evaluate Maximum Logit Scores (MLS) as proposed by (Vaze et al., 2022) for semantic new-class shifts who argue that the softmax operation cancels out feature magnitudes relevant for OoD-D (we also add MLS scores averaged over MCD samples to the benchmark: MCD-MLS).
Finally, we include the recently reported state-of-the-art approach: Mahalanobis Distance (MAHA) measured on representations of a Vision Transformer (ViT) that has been pretrained on ImageNet (Fort et al., 2021).
Classifiers:
Because this change in the classifier biases the comparison of CSFs, we additionally report the results for selected CSFs when trained in conjunction with a ViT classifier.
For implementation and training details, see Appendix
E. Since drawing conclusions from re-implemented baselines has to be taken with care, we report reproducibility results for all baselines including justifications for all hyperparameter deviations from the original configurations in Appendix J.
4. R ESULTS
The broad scope of this work reflects in the type of empirical observations we make: We view the holistic task protocol as an enabler for future research, thus we showcase the variety of research questions and topics that are now unlocked rather than providing an in-depth analysis on a single observation.
Appendix G.1 features a discussion on how this study empirically confirms R1-R3 stated in Section 2.
Table 1 shows the results of the FD-Shifts benchmark measured as AURC scores.
The reproducibility study in Appendix J confirms that none of the observed effects are caused by faulty re-implementations.
None of the evaluated methods from literature beats the simple Maximum Softmax Response baseline across a realistic range of failure sources.
For both classifiers (CNN and ViT) the softmax baselines (either MSR or MCD-MSR) show the best or close to the best performance on all i.i.d.
test sets and all shifts except new class shifts 4 .
This is surprising given the claims of literature in
MisD, SC, and OoD-D: All three tested methods based on the CNN-classifier (DG-Res, Devries, and
ConfidNet) fail to generalize beyond the scenarios they have been proposed on, i.e. to more complex data sets (like iWildCam or BREEDS) and covariate distribution shifts (corruptions and sub-class
A second version of this table featuring CSF rankings is provided as Table
[Table]
shifts).
Even on the test data they have been proposed on, all three struggle to outperform simple baselines
[Table]
These findings indicate a pressing need to evaluate newly proposed CSFs for failure detection in a wide variety of data sets and distribution shifts in order to draw general methodological conclusions.
Prevalent OoD-D methods are only relevant in a narrow range of distribution shifts.
The proposed evaluation protocol allows, for the first time, to study the relevance of the predominant OoD-D methods in a realistic range of distribution shifts.
While for non-semantic new class shifts ("far OoD"), prevalent methods from OoD-D (MLS, MCD-MLS, MAHA) show the best performance across both classifiers, their superiority vanishes already on semantic new class shifts (only ViT-based
MAHA on SVHN shows best performance).
On the broad range of more nuanced (and arguable more realistic) covariate shifts, however, OoD-D methods are widely outperformed by softmax baselines.
This finding points out an interesting future research direction of developing CSFs that are able to detect failures across the entire range of distribution shifts.
AURC is able to resolve previous obscurities between classifier robustness and CSF performance.
The results of ConfidNet provide a vivid example of the relevance of assessing classifier performance and confidence ranking in a single score when evaluating CSFs.
The original publication reports superior results on CIFAR-10 and CIFAR-100 compared to the MCD-MSR baseline as measured by the MisD metric AUROC f .
These results are confirmed in Table 9, but we observe a beneficial effect of MCD on classifier training that leads to improved accuracy (see Table 8).
This poses the question: Which of the two methods (ConfidNet or MCD-MSR) will eventually lead to fewer silent failures of the classifier?
One directly aids the classifier to produce fewer failures and the other seems better at detecting the existing ones (at least on its test set with potentially more easily preventable failures)?
AURC naturally answers this question by expressing the two effects in a single score that directly
[Table]
[Table]
relates to the overarching goal of preventing silent failures.
This reveals that the MCD-MSR baseline is in fact superior to ConfidNet on the i.i.d.
test sets of both CIFAR-10 and CIFAR-100.
ViT outperforms the CNN classifier on most data sets.
Figure 8 shows a comparative analysis between ViT and CNN classifier across several metrics.
As for AURC, ViT outperforms CNN on all datasets except iWildCam, indicating that the domain gap of ImageNet-pretrained representations might be too large for this task.
This is an interesting observation, given that CAMELYON featuring images from the biomedical domain could intuitively represent a larger domain gap.
Further looking at Accuracy and AUROC f performance, we see that performance gains clearly stem from improved classifier accuracy 5 , but the CSF ranking performance is on par for ViT and CNN (although the failure detection task might be harder for ViT given fewer detectable failures compared to CNN).
Different types of uncertainty are empirically not distinguishable.
Considering the associations made in the literature between uncertainty measures and specific types of uncertainty (see Appendix
I), we are interested in the extent to which such relations can be confirmed by empirical evidence from our experiments.
As an example, we would expect mutual information (MCD-MI) to perform well on new class shifts where model uncertainty should be high and expected entropy (MCD-EE) to perform well on i.i.d.
cases where inherent uncertainty in the data (seen during training) is considered the prevalent type of uncertainty.
Although, as expected, MCD-EE performs generally better than MCD-MI on i.i.d.
test sets, the reverse behavior can not be observed for distribution shifts.
Therefore, no clear distinction can be made between aleatoric and epistemic uncertainty based on the expected benefits of the associated uncertainty measures.
Furthermore, no general advantages of entropy-based uncertainty measures over the simple MCD-MSR baseline are observed.
CSFs beyond Maximum Softmax Response yield well-calibrated scores.
We advocate for a clear purpose statement in research related to confidence scoring, which for most scenarios implies a separation of the tasks of confidence calibration and confidence ranking (see Section 2).
Nevertheless, to demonstrate the relevance of our holistic perspective, we extend FD-Shifts to assess the calibration error, a measure previously exclusively applied to softmax outputs, of all considered CSFs.
Platt scaling is used to calibrate CSFs with a natural output range beyond [0 , 1] (Platt, 1999).
Calibration errors of CSFs are reported in Table 10, indicating that currently neglected CSFs beyond MSR provide competitive calibration (e.g. MCD-PE on CNN or MAHA on ViT) and thus constitute appropriate confidence scores to be interpreted directly by the user.
This observation points out a potential research direction, where, analogously to the quest for CSFs that outperform softmax baselines in confidence ranking, it might be possible to identify CSFs that yield better calibration compared to softmax outputs across a wide range of distribution shifts.
The Maximum Softmax Response baseline is disadvantaged by numerical errors in the standard setting.
Running inference of our empirical study yields terabytes of output data.
When tempted to save disk space by storing logits as 16-bit precision floats instead of 32-bit precision, we found the confidence ranking performance of MSR baselines to drop substantially (reduced AURC and
AUROC f scores).
This effect is caused by a numerical error, where high logit scores are rounded to 1 during the softmax operation, thereby losing the ranking information between rounded scores.
Surprisingly, when returning to 32-bit precision, we found the rate at which rounding errors occur to still be substantial, especially on the ViT classifier (which has higher accuracy and confidence scores compared to the CNN).
Table 2 shows error rates as well as affected metrics for different floating point precisions.
Crucially, confidence ranking on ViT classifiers is still affected by rounding errors even in the default 32-bit precision setting (effects for the CNN are marginal as seen in AURC scores), see for instance AUROC f drops of 9% on CIFAR-10 and 5 .
47% on BREEDS (i.e. ImageNet data).
This finding has far-reaching implications affecting any ViT-based MSR baseline used for confidence ranking tasks (including current OoD-D literature).
We recommend either casting logits to 64-bit precision (as performed for our study) or performing a temperature scaling prior to the softmax operation in order to minimize the rounding errors.
Further results.
Despite its relevance for application, the often final step of failure detection, i.e. the definition of a decision threshold on the confidence score, is often neglected in research.
In Appendix
D we present an approach that does not require the calibration of scores and analyze its reliability under distribution shift.
In addition, Appendix G features Accuracy and AUROC f results for all experiments.
For a qualitative study of failure cases, see Appendix H.
C ONCLUSION & T AKE - AWAYS
This work does not propose a novel method, metric, or data set.
Instead, following the calls for a more rigorous understanding of existing methods (Lipton and Steinhardt, 2018; Sculley et al., 2018) and evaluation pitfalls (Reinke et al., 2021), its relevance comes from providing compelling theoretical and empirical evidence that a review of current evaluation practices is necessary for all research aiming to detect classification failures.
Our results demonstrate vividly that the need for reflection in this field outweighs the need for novelty:
None of the prevalent methods proposed in the literature is able to outperform a softmax baseline across a range of realistic failure sources .
Therefore, our take-home messages are:
Research on confidence scoring (including MisD, OoD-D, PUQ, SC) should come with a clearly defined use case and employ a meaningful evaluation protocol that directly reflects this purpose.
If the stated purpose is to detect failures of a classifier, evaluation needs to take into account potential effects on the classifier performance.
We recommend AURC as the primary metric, as it combines the two aspects in a single score.
Analogously to failure prevention ("robustness"), evaluation of failure detection should include a realistic and nuanced set of distribution shifts covering potential failure sources.
Comprehensive evaluation in failure detection requires comparing all relevant solutions towards the same goal including methods from previously separated fields.
The inconsistency of our results across data sets indicates the need to evaluate failure detection on a variety of diverse data sets.
Logits should be cast to 64-bit precision or temperature-scaled prior to the softmax operation for any ranking-related tasks to avoid subpar softmax baselines.
Calibration of confidence scoring functions beyond softmax outputs should be considered as an independent task.
Our open-source framework features implementations of baselines, metrics, and data sets that allow researchers to perform meaningful benchmarking of confidence scoring functions. A CKNOWLEDGEMENTS This work was funded by Helmholtz Imaging (HI), a platform of the Helmholtz Incubator on Information and Data Science. We thank David Zimmerer and Fabian Isensee for insightful discussions and feedback.
A F ORMULATION OF F AILURE S OURCES
In general, we distinguish three sources of error that can cause image classification systems to output false predictions (see Figure 2 for exemplary visualizations).
Inlier Misclassifications : This type of failure source is defined by occurring on cases that are sampled i.i.d.
with respect to the training distribution and is commonly addressed by work in MisD and SC.
Possible reasons for occurrence are missing evidence in the image related to the ground truth category, poor model fitting, or data variations that are not considered as distribution shifts yet in a specific use case.
Covariate
Shift : Images subject to a covariate shift (Quionero-Candela et al., 2009) can still be assigned to one of the training categories.
Various examples are investigated by recent robustness benchmarks:
Image corruptions shift (Hendrycks and Dietterich, 2019), domain shift such as medical images from different scanners and clinical sites or satellite images from different seasons (Koh et al., 2021), or subpopulation shift 6 , where unseen semantic variations of the training categories occur such as unseen breeds of an animal category (Santurkar et al., 2021).
We summarize domain shift and subpopulation shift under the term sub-class shift .
To the best of our knowledge, these relevant sub-class shifts have not been studied in the context of confidence scoring before.
New-class Shift : Images subject to new a class shift can not be assigned to any of the training categories.
This type of failure is commonly addressed by work in OoD detection.
We follow Ahmed and Courville (2020) in further distinguishing semantic (only foreground object is subject to semantic variations, e.g. previously unseen classes, a.k.a "near OoD" (Winkens et al., 2020)) and non-semantic new-class shifts (context changes, e.g. images from a new data set and classification task, a.k.a "far OoD").
B T ASK F ORMULATIONS ADDRESSING F AILURE D ETECTION
This section provides a technical exposition of current evaluation protocols for all task formulations stating to address the goal of preventing failures of a related classifier by means of CSFs.
Figuregives an overview of relevant metrics in this context.
B. C LASSIFICATION
To set the context we describe the standard evaluation protocol of a classification task, which is denoted in the notation with y cl for the class label, distinguishing it from the failure detection label y f .
Given a data set { ( x i , y cl ,i ) } Ni =1 of size N with ( x i , y cl ,i ) independent samples from X × Y , and including the discrete class label y cl ∈ Y = { 0 , 1 , 2 , ..., n classes − 1 } .
On this data set a classifier m ( · , w ) :
X → Y maps from the input images x to the predicted labels ˆ y with model parameters w .
ˆ y m,i is the classification decision obtained as the maximum class probability from the classification model’s probability output vector
[Equation]
with c ∈ { 0 , ...n classes − 1 } for the respective class.
According to this notation, arg max c ∈ Y P m ( y = c
| x i , w ) .
The performance of such a classifier is in multi-class setups often evaluated via accuracy,
[Equation]
where I (Event) is defined as indicator function:
[Equation]
1 , if the event Event occurs
[Equation]
Another way is to evaluate classes separately by computing the binary label y c ∈ [0 , 1] as
[Equation]
Following this notation y c,i = I
( y cl ,i = c ) is the binary class label for sample i .
Subsequently, the confusion matrix can be computed counting the four possible evaluation outcomes per case and class c :
[Equation]
These cardinalities are defined depending on the cut-off θ on the provided predicted class probabilities (PCP).
Subsequently, various counting metrics can be applied, for instance Sensitivity (also called recall or true positive rate), False Positive Rate (FPR, or 1 - Specificity), or Precision per class c :
[Equation]
[Equation]
[Equation]
Next to evaluating these metrics on a certain cut-off θ on PCPs, often multi-threshold metrics are employed, which scan over cut-offs from all PCP values present in the data set to obtain ROC-curves (Sensitivity plotted over FPR) or Precision-Recall-Curves (PRC, Precision plotted over Sensitivity).
Model performance in the form of a single score is extracted via computing the respective area under the curve, i.e. the AUROC for ROC-curves.
Here the multi-threshold list { θ t } Tt =0 of length T are the cut-off values obtained as the unique values of the descending ranking of all CSF values.
The class-wise AUROC values can then be computed as follows:
[Equation]
The AUPRC for PRC-curves is defined similarly, and commonly approximated by a average precision (AP) score due to interpolation issues:
[Equation]
Both areas under the curves can be interpreted as ranking metrics , i.e. they require cases of the class c to be separated from cases of other classes based on a ranking of PCP values.
B. F AILURE D ETECTION
Concrete application of CSFs for the task of detecting failures in order to prevent incorrect predictions of a classifier are formulated in Equation 1 in Section 3.
Based on the failure label in Equation 2, which is 1 for classification failure and 0 for success, a confusion matrix can be determined for a cutoff value τ as follows:
[Equation]
Analogous to the evaluation of the classification performance
[Equation]
one can compute different counting metrics for the confidence ranking task:
[Equation]
B.2. M ISCLASSIFICATION D ETECTION
This evaluation protocol for CSFs directly sticks to the binary classification task between correct and incorrect predictions defined above
[Equation]
and uses ranking metrics analogous to the area under the curves defined for classification evaluation based on a multi-threshold list { τ t } Tt =0 of length T , which are the cut-off values obtained as the unique values of ascending ranking of confidence scores.
Most commonly used are the failure detection AUROC:
[Equation]
and the failure detection AUPRC or AP-score:
[Equation]
The data set in these tasks is often biased towards many correct predictions and few incorrect predictions since one does not usually think about preventing failures if the classification performance is very poor in the first place.
Consequently, this biases the failure detection AP-score resulting in higher scores.
The reverse form defining errors as positive samples, which is not biased in the aforementioned way, is also evaluated:
[Equation]
As described in Section 2, this protocol comes with the pitfall of not considering the classifier performance, thus potential effects of CSFs on the classifier performance caused by joined training go unnoticed (for a visualization of this pitfall see Figure 4).
This effect can be nailed down to the (1 − y f ,i ) factor of Equation 24: The precision score (second factor of the product) only gets a weight for correct predictions
y f = 0 , so by perfect separation, it is possible to traverse all correct predictions (all "weights") while the precision score is one, and any subsequent incorrect predictions are not considered by the metric.
This behaviour prevents meaningful comparison of arbitrary CSFs in practice.
For instance, consider a comparison between two very standard CSFS, the confidence scores derived from the maximum softmax response (MSR) of a classifier against scores derived from the softmax mean over Monte Carlo Dropout (MCD) sampling of the same classifier.
This comparison can already lead to considerable evaluation bias because the classification decision based on MCD means approach most likely produces a different set of failure cases.
What’s more, associated metrics are notoriously sensitive to such label biases: While it is well-known, that high scores in failure detection AUROC f can be achieved via a bias towards true negatives (such as background objects in object detection), in failure detection AP f ,err with failure as the positive label is used (Hendrycks and
Gimpel, 2017; Corbière et al., 2019), which can be equally hacked by biases towards easy-to-detect true positives.
As a consequence, this metric typically gives the highest scores at the beginning of model training, when large amounts of failures are produced by the classifier.
affect the e.g. based on or additional loss functions resulting in distinctive classifiers the of
Crucially, the results from this evaluation (“failure labels”) will be used as reference labels for the
CSF evaluation (see “task 2: Failure detection” in Figure 1).
For simplicity, we assume that all three
CSFs output identical confidence scores.
However, ranking metrics such as AUROC f (as applied in MisD) yield vastly different results across CSFs because the evaluation is based different sets of reference labels.
In this case CSF1 shows the best failure detection performance, but is also the one causing additional failures in the first place.
This shows that isolated evaluation of failure detection (as done in MisD) is flawed, because CSF effects on the classifier training are not accounted for.
To serve the overarching purpose of preventing silent failures of a classifier (“task1 + task2” in Figure 1), one would need to select a CSF based on both, rankings of Accuracy and AUROC f .
But what is a natural way to weigh the two rankings?
The metric proposed in Section 3 directly reflects the purpose of preventing silent failures and can be interpreted as providing a natural weighting between
Accuracy and AUROC f rankings.
B.2. O UT - OF -D ISTRIBUTION D ETECTION
In out of distribution detection, the label y out signals whether a sample is an outlier y out = 1 or an inlier y out = 0 .
Based on this label the AUROC value is primarily used as a performance metric:
AUROC out =
T (cid:88) t (FPR
[Equation]
This formulation and the thresholds τ t are equivalent to the AUROC used for MisD in Equationexcept for the different label y out that is employed (for implications and pitfalls of using this label see Figure 2).
Due to the vast overlap of task formulations FD-Shifts enables a seamless integration of OoD-detection methods for holistic comparison (see Section 3).
B.2. P REDICTIVE U NCERTAINTY E STIMATION
Common evaluation in PUQ (e.g. Ovadia et al.
; Lakshminarayanan et al.
(2017) employs proper scoring rules like Negative-Log-Likelihood
[Equation]
and Brier Score
[Equation]
Both directly assess the classifier output and are not applicable to other CSFs.
Each metric requires both ranking and calibration of confidence scores which is interpreted as assessing the "general meaningfulness" of scores (see also Figure 6).
However, in case there is a clear use case defined requiring either ranking or calibration of confidence scores, evaluation with proper scoring rules might dilute this focus and hinder the practical relevance of results.
B.2. S ELECTIVE C LASSIFICATION
The idea of equipping a classifier with the option to abstain from certain decisions based on a selection function has been around for a long time (Chow, 1957;
El-Yaniv and Wiener, 2010).
SC directly evaluates the process of detecting failures of a classifier via CSFs as described by Equation 1.
SC essentially tries to optimize the trade-off between achieving low risk and high coverage.
Thereby, risk is defined as the error rate of cases remaining after selection:
[Equation]
And coverage as the ratio of cases remaining after selection:
[Equation]
Typical evaluation is performed for single cut-offs of τ reporting and comparing resulting risk and coverage scores.
One can also calculate the Area under the Risk-Coverage-Curve (AURC):
[Equation]
which uses the thresholds τ t equivalently to the failure detection AUROC for MisD in Equation 23 as the unique values of the ascending ranking of confidence scores
However, AURC is currently not used as the primary metric or part of an established evaluation protocol in SC.
Instead, e-AURC has been proposed (Geifman et al., 2019):
[Equation]
where it is used that Risk( τ 0 )
= (1 − Prevalence f ) is equal to the Negative Rate.
Further,
Prevalence f is equal to the Accuracy of the base classifier m .
Therefore e-AURC effectively subtracts the classifier performance aspect from AURC for exclusive focus on evaluating the ranking power of CSFs.
This process essentially collapses AURC back to a pure ranking metric such as employed in MisD.
However, as laid out in Section 2, evaluating CSFs while being agnostic to the classifier comes with significant pitfalls preventing comprehensive method assessment.
B.2. U NIFICATION OF T ASK F ORMULATION - T ECHNICAL DETAILS
In this work, we propose to establish AURC as the primary metric for failure detection, as it fulfills requirements R1 - R3 defined in Section 2.
Comparing Equation 31 to Equation 24, a deviation is observed in technical details such as the conservative interpolation of the AP-score versus trapezoidal interpolation in AURC, or the fact that AURC is defined with "reverse precision" in the form of
Risk requiring minimization of the score.
However, the one crucial conceptual difference is the fact that "weight" is put on the risk score (second factor of the product) in Equation 31 for all steps of coverage (first factor).
This is in contrast to the (1 − y f ,i ) factor in Equation 24, which ignores steps of incorrect predictions in sensitivity (first factor in the equation), and essentially implies that incorrect predictions of the classifier are penalized irrespective of how perfect the CSF separates these.
We argue that this aspect of evaluation is an essential part of evaluating CSFs in the context of failure detection, because to truly evaluate the property of a CSF to "prevent silent failures of a classifier" we don’t only need to check whether CSFs are able to filter existing failures.
We also need to check whether they might have caused new failures (or prevented further failures) by altering the training compared to a neutral classifier trained without the CSF, thereby creating their own test set.
In Appendix F we provide a revised implementation of AURC fixing various shortcomings of prior open-source implementations.
Figure 5 visualizes how OoD-D protocols need to be adapted in order to follow the unified task formulation can be
"filter"), because the trained classifier has no possibility to correctly predict unseen classes.
However, not all "inlier" are correct classifier decisions.
The current OoD-D protocol rewards CSFs for not detecting inlier misclassifications (see Figure 2).
On the other hand, penalizing CSFs for not these cases would the desired evaluation focus on new-class shifts.
Thus, we propose to remove inlier misclassifications from evaluation when reporting a CSF’s performance under new-class shift.
While taking those cases out of the confidence ranking assessment, we argue that at the same time it is required to fulfil R1, i.e. assess the effect of CSFs on classifier performance.
Thus, we propose to report AURC even on new-class shifts instead of AUROC f , which follows R1 because more inlier failures taken out lead to a higher ratio of OoD cases (failure cases by definition) in the test set and thus a lower accuracy.
C R ELATION BETWEEN C ONFIDENCE R ANKING AND C ONFIDENCE C ALIBRATION
Calibration in the context of CSFs describes the requirement of predicted class scores to match the empirical accuracy of associated cases (e.g. cases with a confidence score of 0.8 are correct 80% of the time), which is useful for use cases like interpretable decision making (e.g. human assessment or a need for interpretable cutoffs like " risk > 0 . 8 "), or assessing the applicability of a trained classifier to entire data sets (Guo et al., 2017).
Figure 6 shows how confidence calibration and confidence ranking are two independent requirements, i.e. either of the tasks can be solved without solving the other.
A CSF can in theory have zero calibration error (perfect calibration) but provide no ranking of single cases and vice versa, raising the questions: Which requirement do we actually want a CSF to satisfy given a concrete use case?
What can we do with calibrated scores in practice if there is no ranking?
What are concrete use cases where both tasks are required i.e. a “general meaningfulness” is measured such as by proper scoring rules like negative log-likelihood or Brier Score?
We argue that for all use cases where any sort of selection, cut-off, or separation between cases with lower or higher confidence is performed that does not rely on interpretation of raw confidence values, only confidence ranking and no calibration is required:
Given a well-separating CSF (good scores according to ranking metrics), one can define a cut-off value on a validation set according to practical requirements such as “filter only 20% of cases” (coverage guarantee) or “filter such that a maximum of 5% errors remain” (risk guarantee) and select cases according to this cut-off in a subsequent application.
Importantly, calibrated scores are not required at any stage in this process.
Appendix D demonstrates a realistic example of how to do this under distribution shifts.
While Tomani et al.
(2021) studied calibration under a variety of distribution shifts, to the best of our knowledge recent work in the context of neural networks only considers a single CSF: the maximum softmax response (MSR) of the classifier.
Applying our holistic perspective, the question arises:
If other CSFs beat MSR in confidence ranking tasks, is it not likely that they could also provide better-calibrated scores?
To this end, we extend our empirical study and analyze the calibration performance of all studied CSFs under distribution shifts (see Appendix G.
D D EFINING A F INAL D ECISION T HRESHOLD ON C ONFIDENCE S CORES
The idea exists that despite a well-ranked confidence scoring function one still requires calibration of scores in order to set a meaningful decision threshold at the end.
In Figure 7 we explore selection under guaranteed risk (SGR) as an alternative decision making tool under distribution shift by exemplary studying ConfidNet under image corruptions on CIFAR-100.
Again, comparing the practical applicability to the calibration approach: When a specific application and a potential practitioner observe 0 .
11 ECE on the i.i.d.
set, what guarantees for reliable decision making can be derived from this score?
The results of the SGR study show how given a desired risk r ∗ the thresholds determined on the validation set provide reliable risk guarantees on the i.i.d.
test set.
Under image corruption shift, as expected, these guarantees start to break in the order of confidence (here in the statistical sense) parameters δ .
While providing risk guarantees under unknown distribution shifts is certainly an open problem, SGR might help practitioners to develop a general feeling for risk excesses under potential shifts in their application domain and to understand how choices of δ might be able to counteract certain levels of expected shift.
For the experiments on SGR, we use the algorithm proposed by Geifman and El-Yaniv (2017), which computes the threshold τ that maximizes coverage, while satisfying the following condition:
[Equation]
where δ is a defined confidence parameter, r ∗ is the desired risk, R the risk and S i are i.i.d samples from a training or validation set.
m is the classifier and g the confidence scoring function defined in
Section B
E T RAINING D ETAILS E. D ATA S ET S PLITS
Table 3 shows the number of images per utilized data set and associated splits.
We used official splits for all data sets, except that we followed Liu et al.
(2019) in splitting a small amount (in our case 1000 images) of the i.i.d.
test set for validation and tuning.
Because there was no i.i.d.
test set for BREEDS-Entity, we held out 10000 training cases for this purpose (1000 of which were again split for validation).
The number of images in new-class shifts are sums over the original i.i.d.
test splits of SVHN (26032), CIFAR-10 (10000), CIFAR-100 (10000) and TinyImagenet 7 (10000).
The model was trained on each case while holding out the test split of the data set.
When training on
CIFAR-100 super-classes, we discarded the super-class "vehicle 2" because of its obvious semantic overlap with the "vehicle 1" super-class in order to mitigate evaluation noise due to label ambiguities.
The sub-class shift test set of the super-classes training was composed by adding up training and testing cases of each held out sub-class per super-class.
For the semantic new-class shift on SVHN and iWildCam we sample 40% of classes randomly to be held back and train on the remaining 60% of classes (each of the 5 runs per experiment receives a new sample of held-out classes)
[Table] [Table]
E. C LASSIFIER A RCHITECTURES
For the CNN training, we used a small convolutional network on SVHN (following Corbière et al.
, VGG-13 (Simonyan and Zisserman, 2015) on CIFAR-10/100 (following DeVries and Taylor (2018)), and a ResNet-50 (He et al., 2015) for the three robustness benchmarks.
For ViT training we use the ViT-B/16 architecture and scale up all images to 384x384 pixels following Dosovitskiy et al.
E. H YPERPARAMETERS
On SVHN and CIFAR-10/100 we sticked as close as possible to configurations of original publications (Corbière et al., 2019; DeVries and Taylor, 2018;
Liu et al., 2019) while at the same time converging to identical classification model configurations.
On robustness benchmarks, we stuck to the proposed configurations of respective baseline experiments (Santurkar et al., 2021;
Koh et al., 2021).
Thus, data augmentation was only used on CIFAR-10/100 (slight rotation, horizontal flip, and cutout following DeVries and Taylor (2018)) and on BREEDS-Entity-13 (horizontal flip and color jitter following Santurkar et al.
We used a cosine decay schedule without restarts following
Loshchilov and Hutter (2017) for all methods and data sets, as we found this schedule to robustly generate high-quality results (see Appendix J for ablations).
All classifiers are trained with SGD and momentum 0.9.
ConfidNet is trained with Adam, learning rate 10 − 4 and finetuned with learning rate 10 − 6 following the original configuration.
Table 4 shows training parameters that have been modified on each data set.
Table 7 shows training parameters used for finetuning the ViT.
[Table]
E. M ODEL S ELECTION
Using the validation set we selected whether or not to use dropout during training for all deterministic confidence scoring functions as well as the best hyperparameter o per DeepGambler-based confidence scoring function.
For the latter [Table] we repeated all experiments using o = [2 . 2 , 3 , 6 , 10] .
As we found higher o to be beneficial for more complex data sets [Table] we additionally ran o =
[12 , 15 , 20] on CIFAR
and o = 15 on iWildCam and BREEDS-Entity-13 (additional runs on the last two data sets had to be reduced due to computing resource constraints).
Table 5 shows selected o per method and data set, Table 6 shows whether or not dropout has been used for training per method and data set.
Table 7 shows whether or not dropout has been used and what learning rate was selected when finetuning the ViT. To select the learning rate we do a single run of training for all learning rates in [3 ∗ 10 − 2 , 10 − 2 , 3 ∗ 10 − 3 , 10 − 3 , 3 ∗ 10 − 4 , 10 − 4 ] and chose the lowest AURC on the validation set.
If dropout was used to
[Table] [Table]
[Table]
F R EVISED AURC IMPLEMENTATION
Our implementation of AURC is based on two implementations we found by Geifman et al. 8 and
Corbiere et al.
The two existing implementations have several shortcomings, for instance, in
Geifman et al.
steps in the RC curve are not defined as new unique values in the sorted list of confidence scores, but instead each data sample, including the ones with equal confidence scores, is considered as leading to an individual classification decision with an associated risk-coverage pair.
This effectively leads to a random interpolation of the RC-curve between unique confidence scores adding considerable noise to the result especially in dense singular points such as confidence scores of 0 or 1.
A shortcoming in the implementation by Corbiere et al.
is that there is no well-defined endpoint of the curve meaning the risk just drops to zero after the last RC-curve step (i.e. the coverage value corresponding to thresholding at the lowest confidence score), which effectively favours methods with higher "lowest coverages".
This can make a substantial difference in practice because methods often assign equal confidence scores to more than one case especially at 0 and 1.
Thus, analogously to scikit-learn’s PRC-curve implementation, we add a final point at zero coverage and the risk remaining the risk of the last RC-curve step.
G A DDITIONAL R ESULTS
This section contains additional results for Accuracy (Table 8), AUROC f
(Table 9, as well as
Expected Calibration Error (Table 10).
Table 11 shows rankings of confidence scoring functions based on AURC scores (i.e. based on Table 1).
G. E MPIRICAL C ONFIRMATION OF THE I MPORTANCE OF R EQUIREMENTS R1-R
R1:
Comprehensive evaluation requires a single standardized score that applies to arbitrary
CSFs while accounting for their effects on the classifier.
The arguments that lead to R1 are stated in Section 2 and visualized in Figure 4.
In Table 12 we provide evidence for the importance of this argument: Not following R1 and instead evaluating CSFs with pure rankings metrics like AUROC f (as common in MisD) leads to substantially differing rankings of CSFs.
This means that the effect of CSFs on the classifier is a crucial factor to consider in practice when ranking CSFs.
The finding “AURC is able to resolve previous obscurities between classifier robustness and CSF performance” in
Section 4.3 describes a concrete example of how neglecting this factor has led to misleading results in the literature.
Equivalently, we argue that conflating evaluation of failure detection with calibration, as long as the purpose for this dual-purpose assessment is not clearly stated, is a shortcoming of current practices.
Empirically showcasing the effects of this conflation on the ranking of CSFs analogously to Tableis not possible, because common metrics in PUQ (proper scoring rules) exclusively operate on the predicted class scores and are not compatible with arbitrary CSFs.
Instead, this restriction in itself acts as an exclusion criterion for these metrics for the comprehensive evaluation of CSFs.
The experiment shown in Table 12 underlines that fulfilling R1 is essential for meaningful comparison of arbitrary CSFs.
Thus, the general findings of our study described in Section 4.3 are all made possible by the proposed protocol following R1 and can be seen as further empirical confirmation of the importance of this requirement.
R2:
Analogously to robustness benchmarks, progress in failure detection requires to evaluate on a nuanced and diverse set of failure sources.
As described in Section 4.3, our study reveals that “none of the evaluated methods from literature beats the simple Maximum Softmax Response baseline across a realistic range of failure sources” and “prevalent OoD-D methods are only relevant in a narrow range of distribution shifts”.
These findings are a direct result of fulfilling R2 in our evaluation protocol and demonstrate how current protocols (without R2) lead to the proposition of
1 def AURC confidence): coverages =
[] risks =
[] n = len
) idx_sorted = np.argsort confidence) cov = n error_sum sum sorted]) coverages.append
/ n), risks.append(error_sum / n) weights =
[] tmp_weight =12for i in range(0, len(idx sorted) - 1): 14cov = coverror_sum = error_sum - residuals[idx_sorted[i]] selective_risk = error_sum /(n - i) tmp_weight +=
19if i =
0 or \ confidence[idx sorted[i]] !
= confidence[idx sorted[i
1]]: 22coverages.append
) risks.append(selective_risk) weights.append(tmp_weight / n) tmp_weight
27if
0: 29coverages.append(0) risks.append(risks[
1]) weights.append tmp
) 33aurc sum
[i] risks[i+1]) * 0.5 \ * i ]) 36curve risks) return curve, aurc Listing
Code for AURC calculation.
[Table]
CSFs with a lack of generalization ability that often directly contradicts their purpose statement: to enable the safe application of classifiers by detecting silent failures.
R3:
If there is a defined classifier whose incorrect predictions are to be detected, its respective failure information should be used to assess CSFs w.r.t the stated purpose instead of a surrogate task such as distribution shift detection.
The fact that evaluation w.r.t failure labels, as opposed to
OoD-labels, enables testing methods on a broad and realistic range of distribution shifts is the most important argument for R3.
This importance is empirically confirmed in the finding “prevalent OoD-D
methods are only relevant in a narrow range of distribution shifts” (see Section 4.3), which reveals a clear contradiction of many of these methods’ purpose statements and their actual functionality.
Besides the general importance of following R3, we would like to empirically highlight the importance of dismissing inlier misclassifications from CSF evaluation on new-class shifts (the proposed protocol for new-class shifts is visualized in Figure 5).
We argue that when assessing confidence ranking w.r.t failure detection on new class shifts, penalizing the detection of failures from the i.i.d.
set is an undesired behavior of the current OoD-D protocol and subsequently propose to dismiss these cases from ranking (they still affect the classifier performance considered by AURC thus fulfilling
R1).
Table 13 demonstrates the importance of this modification revealing considerable effects on the resulting CSF rankings (as measured by AUROC f ).
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table] [Table] [Table]
[Table]
281 1 MCD-PE
CNN 2    5 4 2 5 7 6 9 9 4 105 2 MCD-EE
CNN 4    5 5 4 6 8 7 7 594 3 MCD-MI
CNN 2    9 7 6 4 6 10 6 10 12 12 8 129 6 ConfidNet
CNN 12    1 6 8 8 10 10 2 4 4 74 11 11 12 DG-MCD-MSR CNN 5    11 1 1 9 11 3 4 11 11 1118 8 DG-Res
CNN 11 10   10 11 12 101 4 5 83 323 5 Devriesetal.
CNN 10 11   12 12 11 12 12 12 12 1 1 37 12
MSR
ViT 2    2 5 1 3 9 5 3 889 9 MLS
ViT 3    1 8 7 1 7 8 2 774 3 PE
ViT 1    2 5 1 2 8 5 3 888 8 MCD-MSR
ViT 6    5 1 5 8 5 1 7 436 6 MCD-MLS
ViT 8    9 9 8 4 2 7 1 232 1 MCD-PE
ViT 4    5 2 3 7 4 1 6 435 5 MCD-EE
ViT 6    5 2 6 6 3 1 5 323 4 MCD-MI
ViT 4    8 4 3 9 6 4 9 637 7 MAHA
ViT 9    4 7 9 5 1 9 7
[Table]
svhn
[Table]
metric α β α β α β α β α β α β α β α β α β α β α β α β α β α β α β α β α β α β α β α β α β α β α β MSR
CNN 6475692377331091077854101011 11 MLS
CNN 9911121292109118611211121103224 4 PE
CNN 7567592810991138964554779 9 MCD-MSR
CNN 11322452221892371010211667 7 MCD-MLS
CNN 88411117895572511514298111 1 MCD-PE
CNN 221324542335762499410552 2 MCD-EE
CNN 43142655445687137559443 3 MCD-MI
CNN 2685789766546106101212812996 6 ConfidNet
CNN 121258731688101024744734111112 12 DG-MCD-MSR CNN 5791121111119113411111111888 8 DG-Res
CNN 11101210101101112101114581233112335 5 Devriesetal.
CNN
MSR
ViT 22751525126395233858999 9 MLS
ViT 33887818732178332787343 2 PE
ViT 11651525114285133858788 8 MCD-MSR
ViT 66413151578851887433666 6 MCD-MLS
ViT 88578799893427521273221 1 MCD-PE
ViT 44123152366741776413555 5 MCD-EE
ViT 65123152654631665312434 4 MCD-MI
ViT 46123484389964999643777 7 MAHA
ViT
[Table]
674983552 2 MCD-EE
CNN 29883662443 4 MCD-MI
CNN 1210361012127996 6 ConfidNet
CNN 469255710111112 12 DG-MCD-MSR CNN 157111111114888 9 DG-Res
CNN 8111412238325 5 Devriesetal.
CNN
MSR
ViT 37395499889 9 MLS
ViT 21173247332 2 PE
ViT 15284388668 6 MCD-MSR
ViT 78858865776 6 MCD-MLS
ViT 83412122221 2 MCD-PE
ViT 46747644555 5 MCD-EE
ViT 54536533444 4 MCD-MI
ViT 69969976997 8 MAHA
ViT
H Q UALITATIVE A NALYSIS OF F AILURE C ASES
We categorize two types of errors for confidence scores: Overconfidence, i.e. assigning high confidence to wrong predictions, and underconfidence, i.e. assigning low confidence to correct predictions.
Figure 9a) shows the three most overconfident and underconfident test cases of the
MSR confidence score on CIFAR-100 per distribution shift (since test cases subject to new-class shift are incorrect by default, there are no underconfident predictions).
Looking at overconfident predictions, on the i.i.d.
test set we see some expected confusions between semantically close categories.
Images under corruption shift are subjectively hard to classify even for the human eye.
The sub-class shift reveals an ambiguity in the labels rather than a failure of the confidence score containing objects of two valid categories, fish and people, indicating that superclasses in CIFARhave not been specifically designed for sub-class shift by e.g. excluding such cases (Figure 9b looks at failure cases of BREEDS-Entitiy-13 which has been curated for meaningful sub-class shift).
On the new-class shifts, we observe the expected behaviour of predicting the training categories most semantically similar to the unknown categories.
Looking at underconfident cases reveals some interesting shortcut learning (Geirhos et al., 2020):
The fact that the two racoon images received low confidence despite showing their faces indicates that the model relies on spurious features perhaps related to the background.
Similarly, the fox image on the corruption tests may have received low confidence due to the artificial frame.
Further, the colored pixel noise in the hamster image as a source of confusion hints at the mode relying on local texture patterns rather than global semantic features.
While the previous qualitative study only showcases confidence based on MSR, the proposed unified protocol facilitates to study arbitrary confidence scoring functions in detail and across different distribution shifts.
Thus, in Figure 1b) we study failure cases of the Predictive Entropy obtained from
Monte Carlo Dropout (MCD-PE) on the BREEDS-Entity-13 data set.
Entropy scores are reversed and and quartiles, and include all CSFs and studies shown in Tablescaled to [0 , 1] for interpretability.
Note, how an "external" confidence score like entropy can be zero even on the predicted class (such decoupling would be inherently impossible with MSR).
Looking at overconfident cases again reveals label ambiguities on the i.i.d.
test set rather than actual failures of confidence scoring.
Examples are a bird in front of a monitor ("monitor" is a training sub-class to "equipment"), a woman wearing a scarf on top of a poncho ("poncho" is a training sub-class to "garment"), or a vehicle behind a fence ("fence" is a sub-class to man-made structure).
Further, this study reveals interesting shortcut learning such as focusing on knitting patterns ("garment" vs. "accessory"), or a dog being predicted as unlikely to appear in front of the water.
A really hard categorization task is posed by the image showing hot air balloons, as they are confused with volleyballs (training sub-class to "equipment").
I T ECHNICAL FORMULATION OF CLASSIFIER - OUTPUT BASED CONFIDENCE SCORING FUNCTIONS
The majority of confidence scoring functions explored in this study is based on one of the well-established methods for quantifying the uncertainty of a classifier’s prediction.
While strictly proper scoring rules such as NLL or Brier Score only assess performance related to failure detection for one of those measures, MSR, the protocol proposed in this study enables comparison across arbitrary measures.
While PE is often associated with capturing a prediction’s total uncertainty ((Smith and Gal, 2018; Depeweg et al., 2018; Malinin and Gales, 2018)), MCD-EE is said to capture the uncertainty inherent in the data (or aleatoric uncertainty ).
Consequentially, subtracting the latter from the former, i.e. the mutual information (MCD-MI) between the parameters θ and the categorical label y , is often associated with only capturing the model uncertainty (a form of epistemic uncertainty ).
The Maximum Logit Score (MLS), proposed by Vaze et al. (2022), uses the magnitude of the logits which they argue carries information relevant for OoD-D, where greater logits should correspont to more certain predictions.
Maximum Softmax Response (MSR):
[Equation]
where c represents the different classes.
Predictive Entropy (PE):
[Equation]
Maximum Logit Score (MLS):
[Equation]
[Equation]
where f is the function producing the logits that are used to create the probability vector
P ( y | x ∗ , D ) = softmax ( f ( x ∗ , D )) .
Maximum Softmax Response obtained from Monte Carlo Dropout (MCD-MSR):
[Equation]
37) where θ are the model parameters.
Predictive Entropy obtained from Monte Carlo Dropout (MCD-PE):
[Equation]
Expected Entropy obtained from Monte Carlo Dropout (MCD-EE):
[Equation]
Mutual Information obtained from Monte Carlo Dropout (MCD-MI):
[Equation]
Maximum Logit Score obtained from Monte Carlo Dropout (MCD-MLS):
[Equation]
where f is the function producing the logits that are used to create the probability vector P ( y | x ∗ , θ ) =
softmax ( f ( x ∗ , θ )) .
J R EPRODUCIBILITY AND D EVIATIONS FROM O RIGINAL B ASELINE C ONFIGURATIONS
As described in Section E.3, a meaningful evaluation of confidence scoring methods for failure detection requires deviating from original configurations of baselines in order to assimilate the underlying classifier across compared methods.
Since re-implementing and re-configuring baseline methods is prone to inducing biases in comparative studies and respective conclusions (Lipton and
Steinhardt, 2018), in this section, we want to explain all deviations in detail and provide evidence that baselines perform on a par with or superior to the original versions under our configuration.
D EEP G AMBLERS
In Table 14, we compare our configuration against the original configuration using the results reported on CIFAR-10.
We deviate from the original configuration in using additional data augmentation in the form of a cutout, decaying the learning rate via cosine scheduler, training a VGG-13 model instead of
VGG-16, and not using dropout for training (as selected by our model selection).
The hyperparameter o has been selected as 2.2 on this data set, which is identical to the original configuration.
The comparison shows, that our results are substantially better than the original configuration, mainly due to a better classifier accuracy.
[Table]
J. D EVRIES ET AL .
The only substantial deviation from the original protocol is the replacement of the proposed MultiStep learning rate scheduler with our cosine scheduler (the latter is more aggressive with overall higher learning rates).
As Figure 10 shows, this deviation has a negative effect on the OoD-detection performance only when training on CIFAR-10 and testing on TinyImagenet.
There, our configuration only achieves 0 .
965 AUROC instead of 97 .
AUROC from the original paper.
In all other settings, the cosine scheduler results in superior performance.
As part of the assimilation of the training configuration across compared methods, we increased the number of training epochs for Devries et al.
from 200 to 250.
Since performance did not benefit from this increase in training time, we believe there is no fairness issue with DeepGamblers and ConfidNet requiring additional training stages leading to overall more training exposure (see Table 4).
We did not include additional confidence measures such as MSR, PE, or any MCD variations based on the Devries et al.
learning objective, as we found no interesting synergies to occur (this is in contrast to DeepGamblers and ConfidNet).
J. C ONFID N ET
The original authors propose a three-stage epoch selection, first selecting the best classifier according to validation accuracy, second selecting the best ConfidNet training epoch according to fail-averageprecision (with failure defined as the positive label), and third selecting the best ConfidNet finetuning epoch according to fail-average-precision (Corbière et al., 2019).
We found results to be very volatile across single epochs, especially for fail-average precision.
Since reducing the validation set tocases in the process of assimilating training protocols (the original configuration used 5000 cases from the training data) amplified volatility further, we ran an ablation study comparing the proposed model selection to simply taking the last epoch at each stage.
Figure 11 shows the results, indicating that (at least for our smaller validation set), choosing the latest epoch performs superior to epoch selection.
We further deviate from the original configuration by using VGG-13 instead of VGGand applying cutout in addition for data augmentation.
Comparing our results to the ones in the original publication, we vastly improve classifier accuracy and thus AURC scores on SVHN and
CIFAR-10/100, although it is unclear how much of this increase is to be attributed to our configuration and how much to the different training and validation splits.
J. M AXIMUM L OGIT S CORE
Since Vaze et al.
(2022) do not include the ViT architecture in their evaluation, we only compare CNN results.
While Vaze et al.
(2022) use the VGG-32 classifier, we use different classifiers depending on the data set as discussed in Appendix E.2.
On each data set we use the same classifier for MLS evaluation as for the other methods.
The authors employ a variety of improvements to their baseline strategy of which we only use some: we do use cosine annealing to schedule learning rates, but do not employ label smoothing or ensembling to ensure fair comparison with other CSFs.
Thus, we expect our results to lie between their baseline and their improved baseline.
Table 15 compares our configuration to theirs as measured by AUROC f on the SVHN dataset in the original open-set setting, and confirms our expected performance relations.
[Table]
J. M AHALANOBIS S CORE
Fort et al.
(2021) compare different vision transformer architectures, due to computational constraints we only consider the ViT-B/16 architectue.
Since the original publication does not report how results have been obtained, i.e. whether multiple runs have been performed and how final results were selected from these runs, we report both our best run as well as an average over five runs, selected by best accuracy.
Table 16 compares our implementation to theirs as measured by AUROC f trained on
CIFAR-10/CIFAR-100 using CIFAR-100/CIFAR-10 as the out-of-distribution dataset respectively.
We see that our average is slightly below the original results, but results are very volatile with high standard deviations over runs.
Our best run results are en par with the original results.
Equivalently to original results, MAHA score are consistently higher compared to MSR.
[Table]