B EYOND C ONFIDENCE : R ELIABLE M ODELS S HOULD A LSO Q UANTIFY A TYPICALITY
A BSTRACT
While most machine learning models can provide confidence in their predictions, confidence is insufficient to understand and use the model’s uncertainty reliably.
For instance, the model may have a low confidence prediction for a sample that is far from the training distribution or is inherently ambiguous.
In this work, we investigate the relationship between how atypical (or rare) a sample is and the reliability of a model’s confidence for this sample.
First, we show that atypicality can predict miscalibration.
In particular, we empirically show that predictions for atypical examples are more miscalibrated and overconfident, and support our findings with theoretical insights.
Using these insights, we show how being atypicality-aware improves uncertainty quantification.
Finally, we give a framework to improve decision-making and show that the atypicality framework improves selectively reporting uncertainty sets.
Given these insights, we propose that models should be equipped not only with confidence but also with an atypicality estimator for reliable uncertainty quantification .
Our results demonstrate that simple post-hoc atypicality estimators can provide significant value.
I NTRODUCTION
Typicality is an item’s resemblance to other category members (Rosch & Mervis, 1975).
For example, while a dove and a sparrow are highly typical birds, a penguin is an atypical bird.
A large body of cognitive science literature (e.g., Rips (1989); Rips et al.
; Mervis & Pani (1980)) suggests that typicality plays a crucial role in the human understanding of categories.
For instance, humans have been shown to learn, remember, and refer to typical items a lot faster (Murphy, 2004).
Similarly, the representativeness heuristic is the tendency of humans to use the typicality of an event as a basis for decision-making (Tversky & Kahneman, 1974).
This cognitive bias is effective in allowing people to make swift decisions, but it can also result in poor judgments of uncertainty.
For instance, the likelihood of typical events can be overestimated (Tversky & Kahneman, 1974), or uncertainty judgments can be very poor for atypical events (Tversky & Kahneman, 1992).
While it is hard to quantify the uncertainty of human judgments, machine learning models report confidence in their predictions.
However, confidence alone can be insufficient to understand the reliability of a prediction.
For instance, a low-confidence prediction could have an ambiguity that is easily communicated, or it could be a result of the sample being far from the training distribution.
Similarly, a high-confidence prediction could be reliable or miscalibrated.
Our main proposal is that models should quantify not only the confidence but also the atypicality to understand how reliable predictions are, or to understand the coverage of the training distribution.
However, a large volume of machine learning practice works through downloading pretrained models that can report only confidence, without access to a notion of atypicality.
To support our position, we use a simple formalization of atypicality estimation.
With the following empirical and theoretical studies, we show how simple atypicality estimators improve a model’s reliability.
[Figure] Understanding Miscalibration: Calibration is a key metric for assessing the quality of probabilistic models (Gneiting & Raftery, 2007), measuring how well the predicted probabilities of a model match the true likelihoods of outcomes.
Neural networks (Guo et al., 2017) or even logistic regression (Bai et al., 2021) can be miscalibrated out-of-the-box.
Here, we argue that using atypicality can give insights into when to trust a model’s confidence.
Through extensive experiments, we demonstrate that atypicality is highly correlated with miscalibration.
We show that predictions for atypical points are more miscalibrated and overconfident.
We support our empirical study with theoretical insights.
Improving Calibration: Recalibration methods offer some mitigation to the problem of miscalibration (Platt et al., 1999; Guo et al., 2017).
These methods involve adjusting the output of a potentially miscalibrated probabilistic model to improve its calibration.
We show that models need different adjustments to the confidence according to the atypicality of individual points, and hence atypicality is a key factor in recalibration.
Here, we propose a method called Atypicality-Aware Recalibration .
Our algorithm takes into account the atypicality of a sample in the recalibration process and is simple to implement.
We empirically show that complementing recalibration methods with atypicality improves uncertainty quantification, and support our findings with theoretical insights.
Improving Decision-making: Uncertainty quantification can improve transparency and increase user trust (Bhatt et al., 2021).
One way to achieve this is through selective classification (Geifman & El-Yaniv, 2017), where the model can abstain from making a prediction.
Another approach leverages conformal prediction (Vovk et al., 2005) which provides the user with uncertainty sets, i.e., a set of classes that contain the true label with high probability.
However, these actions are often evaluated independently.
We argue that atypicality offers a unifying perspective, and propose
Selective Conformal Prediction.
We show that by taking atypicality into account, we can better decide when to report uncertainty sets and when to abstain.
Summary of Contributions: Overall, atypicality offers a valuable framework for understanding and improving uncertainty quantification.
First, we demonstrate that predictions for atypical examples are more miscalibrated and overconfident.
Using our insights, we present how being atypicality aware improves and complements existing recalibration methods.
Finally, we use atypicality to selectively report uncertainty sets.
Our findings are supported by both experiments and theory.
Thus, we propose that models should also quantify atypicality, and we show simple- and cheap-to-implement atypicality estimators can provide significant value .
I NTERPRETING U NCERTAINTY WITH A TYPICALITY
Motivation: In many applications of machine learning, we have access to a model’s confidence.
It aims to quantify the likelihood that a prediction will be accurate.
In classification, the model output is a probability distribution over the classes.
In most practical scenarios, model confidence is the primary tool used to evaluate the uncertainty of a prediction.
However, the uncertainty obtained from confidence can contain multiple sources of uncertainty (Mukhoti et al., 2021), and distinguishing the source can lead to actionable outcomes such as abstention (Geifman & El-Yaniv, 2017), or data collection (Kirsch et al., 2019).
Here, we first argue that model confidence can have different semantics in different cases.
We present examples from the Imagenet-R
(Hendrycks et al., 2021) dataset for illustration.
In Appendix B we discuss how confidence is not enough to distinguish between high-confidence and representative and high-confidence yet far from the support examples; or having low confidence due to being far from distribution and low confidence due to ambiguity .
In summary, relying solely on model confidence does not provide a complete understanding of the uncertainty in the predictions, and to better interpret the uncertainty we need a better vocabulary.
Even when the model is well-calibrated, confidence does not help us distinguish between an ambiguous sample from a sample that is far from the distribution.
Atypicality Provides a Natural Decomposition
We propose that atypicality provides a natural way to understand uncertainty when combined with confidence.
In general, we consider a sample to be typical if it is representative of the previously observed examples.
For instance, an image of a dog that closely resembles other dogs that were seen during training would be considered typical.
However, if the image is unlike any other observed during training, it would be considered atypical.
In the following sections, we will formally define and explain the concept of typicality.
We discuss the relation to earlier works in Appendix C.1.
The Reliable Quadrant contains typical, high-confidence examples.
Since the sample is typical and the model has high confidence in its prediction, these examples are likely to have higherquality predictions and uncertainty estimates.
The Extrapolation Quadrant contains atypical, high-confidence examples.
These samples are farther from the support of the training data, but the model still has high confidence in its predictions.
We note that in this case, the model may be poorly calibrated and the uncertainty estimates may not be reliable.
The Untrustworthy Quadrant comprises atypical, low-confidence examples.
These examples are highly atypical, and the model has low confidence in its predictions.
This quadrant can include corrupted examples, such as images with heavy noise or blur, or samples from classes that the model has not been trained on.
The Ambiguous
Quadrant comprises typical, low-confidence examples.
These examples are typical in the sense that they may belong to multiple classes, but due to the inherent ambiguity in the sample, the model has low confidence in its predictions.
This quadrant includes samples with inherent ambiguity, such as an image that could be of either a ferret or a polecat, or multi-label examples that contain multiple objects that are from valid classes in an object recognition task.
Formalizing Atypicality We use atypicality with respect to a training distribution.
Informally, a sample is atypical if it is far from the training distribution of a model, e.g., if there are no or a limited number of similar examples to a sample, then it can be called atypical.
More formally, let X ∈ R
d be the random variable denoting features and Y ∈ Y = { 1 , . . .
, C } denote the label where we focus on
C -class classification settings.
We use a ( x ) to denote the atypicality of a sample x .
Definition 2.1 (Atypicality) .
For a sample x , we define the atypicality of a sample as
[Equation]
For a dog image x , if P ( X = x | Y = dog ) is low, then we call x an atypical dog image.
If a ( x ) is high, then we call x an atypical sample.
In words, if a sample is not typical for any class, then it is atypical with respect to the training distribution.
Similarly, we can also use a notion of distance 2 or marginals P ( X = x ) to quantify atypicality.
Appendix E discusses estimation of atypicality .
U NDERSTANDING C ALIBRATION WITH A TYPICALITY
Here, we demonstrate how our framework can be applied to predict the miscalibration of samples.
Let us denote a probabilistic predictor by ˆ P ,
where ˆ P ( Y = y | X = x ) represents the predicted probability of an input x belonging to class y .
Let us define ˆ y = arg max y ∈Y
ˆ P ( Y = y
| X = x ) as the predicted class for input x .
We describe the datasets and models we use in Appendix F.
3. M ODEL C ALIBRATION AND A TYPICALITY
First, we use atypicality to understand calibration.
Calibration is used to evaluate the quality of a probabilistic model (Gneiting & Raftery, 2007).
Informally, a model is considered perfectly calibrated if all events that are predicted to occur P % of the time occur P % of the time for any P ∈ [0 , 100] .
There are different notions of miscalibration.
We employ the following definition:
For the sake of simplicity, consider a binary classification problem where Y ∈ { 0 , 1 } , and the predictor ˆ P : X → [0 , 1] .
Calibration Error (CE) is defined as: CE [ˆ P ] =
E [ | P ( Y | ˆ P ( X ) = p ) − p | ] .
However, it is computationally infeasible to calculate the above expectation with the conditional probability P ( Y | ˆ P ( X ) = p ) .
In practice Expected Calibration Error (ECE) is used to estimate CE, see Appendix G.1.
Here, we aim to examine the relationship between model calibration and atypicality.
Given any K > 1 , we consider the quantiles of a ( X ) , a 1 , a 2 , . . .
, a K +1 such that P ( a ( X ) ∈ ( a k , a k +1 ])
= 1 /K for k ∈ [ K ] .
Specifically, we investigate the atypicality-conditional calibration error ECE [ˆ P | a ( X ) ∈
( a k , a k +1 ]] , i.e., the expected calibration error of a sample that falls within the atypicality quantile k .
within the same confidence range, atypical groups have larger miscalibration rates and are more overconfident.
[Figure]
Atypical Examples are Poorly Calibrated: In Figure 6, we show the distribution of miscalibration where each bin within the grid contains the intersection of the corresponding confidence and atypical- ity quantiles.
We observe that within the same confidence range, predictions for atypical points have lower accuracies and are more miscalibrated and overconfident.
In Figure 3a, we present the miscalibration (ECE) analyses.
In these experiments, we split samples into 5 quantiles according to atypicality and compute the expected calibration error for samples within each group.
The results show a monotonically increasing relationship between atypicality and
ECE.
Specifically, we see that predictions for atypical examples are more miscalibrated compared to typical samples.
In Appendix Figure 7, we present the reliability diagrams decomposed by atypicality.
Predictions are more overconfident for atypical points; and our Theorem 3.1 supports this finding .
Overall, these results demonstrate that atypicality can predict calibration.
This finding has several practical implications.
First, having an atypicality estimator can help us identify samples where probabilistic predictors are less reliable.
Furthermore, it suggests that predictions for minority groups (groups with lower probability mass) are likely to be more miscalibrated, and this violates the sufficiency criterion in the fairness literature (Barocas et al., 2017).
3. C ONFORMAL P REDICTION : A TYPICAL S AMPLES H AVE W ORSE U NCERTAINTY S ETS
Conformal Prediction (Angelopoulos & Bates, 2021) is a framework that assigns a calibrated uncertainty set to each instance.
Given an instance X ∈ X and the true label Y ∈ Y , the goal is to find a function C : X → 2 Y that returns a subset of the label space such that Y ∈ C ( X ) with high probability.
The framework guarantees marginal coverage , i.e. P ( Y ∈ C ( X )) ≥ 1 − α , for a choice of α .
See Appendix H.2 for further details.
Conditional Coverage quantifies the coverage for a group, i.e., P ( Y ∈ C ( X ) | X ∈ G ) .
While conformal prediction provides marginal coverage guarantees, it may not guarantee the same for specific groups, lacking conditional coverage guarantees.
Lu et al.
(2022) showed that conformal prediction for skin lesion classifiers does not satisfy coverage when conditioned on groups of skin color.
Here, we investigate how conformal prediction performs with respect to atypicality, and focus on analyzing atypicality-conditional coverage : P ( Y ∈ C ( X ) | a ( X ) ∈ ( a k − 1 , a k ]) ≥ 1 − α .
Results:
In Figure 3b, we present the results.
For each experiment, we fit a threshold using the calibration set and produced prediction sets for each of the samples in the test set.
Next, similar to previous experiments, we split samples into atypicality quantiles and compute the coverage for that group.
The results show a monotonically decreasing relationship between atypicality and coverage.
Specifically, we observe that while typical examples satisfy the conditional coverage criterion, atypical examples do not perform as well.
See Tables 3,4 for the results in tabular format.
3. T HEORETICAL A NALYSIS : C HARACTERIZING C ALIBRATION E RROR WITH A TYPICALITY
Here, we aim to characterize the calibration error with atypicality.
We build on the results from Bai et al.
and extend the analyses for atypicality-conditional calibration.
We analyze Logistic
Regression with the Gaussian data model and describe the setting more formally in Appendix K.1.
Theorem 3.1.
Consider the data generative model and the learning setting in Appendix K.1.
For any K > 1 , suppose we consider the quantiles of a ( X ) , a 1 , a 2 , ..., a K , a K +1 such that P ( a ( X ) ∈ ( a k , a k +1 ])
= 1 /K for k ∈ [ K ] .
We assume ∥ β ∗ ∥ ≤ c 0 , and d/n = κ , for some sufficiently small c 0 , κ > 0 .
Then, for sufficiently large n , for k = 2 , . .
, K , we have
E [ u − P ( Y = 1 | ˆ P 1 ( X ) = u ) | a ( X ) ∈ [ a k − 1 , a k ]] > E [ u − P
( Y =
| ˆ P 1 ( X ) = u ) | a ( X ) ∈ ( a k , a k +1 ]]
That is, the resulting classification model is over-confident, and the level of over-confidence becomes larger when the data is more atypical (with smaller a ( X ) ).
In addition, the gap becomes larger for smaller sample sizes n .
Proof of the theorem can be found in Appendix K.2.
I MPROVING R ECALIBRATION WITH A TYPICALITY
In the following section, we delve into how atypicality can complement and improve post-hoc recalibration methods.
Specifically, we will examine Temperature Scaling (TS) (Guo et al., 2017).
TS adjusts the probability estimates of a model to improve its calibration.
By utilizing our understanding of atypicality, we will show how atypicality awareness helps improve TS and conformal prediction.
We empirically and theoretically show that taking atypicality into account improves recalibration. Atypicality Percentile 1. T e m p e
r a t u r e τ k
ImageNet 0. Atypicality Percentile 1. CIFAR Parametric Recalibration: Different Groups need Different Temperatures.
Temperature scaling, a single parameter variant of Platt Scaling (Platt et al., 1999), is a simple recalibration algorithm that calibrates the model using a single temperature parameter.
Namely, the predictor is of the form
[Equation]
2) where f : X → R |Y| , τ ∈ R .
f is the model that takes an input and outputs scores/logits, and τ is the recalibration parameter.
In practice, τ is optimized using a calibration set to minimize a proper scoring rule such as the cross-entropy loss.
We refer the reader to Guo et al.
(2017) for more details on TS.
Next, we look at the relationship between ECE and atypicality after recalibration.
Results: We separately perform Temperature Scaling on points grouped according to the atypicality quantiles.
Namely, we fit a separate temperature parameter to each of the atypicality quantiles, let us denote it by τ a k for the quantile covering a ( X ) ∈ ( a k − 1 , a k ] .
In Figure 4, we observe an increasing relationship between the values a k and τ a k .
We observe that different atypicality groups need different adjustments to become calibrated, and more atypical groups need larger temperatures.
This suggests that being atypicality aware can potentially improve calibration.
Furthermore, this can mean that while a single temperature value improves average calibration, it may hurt other groups.
Atypicality-Aware Recalibration.
Our findings point us to a simple proposal: Recalibration algorithms should be parameterized not only by the model confidence but also by the atypicality.
Generally, we suggest that a recalibration method should take the following form:
[Equation]
where the function is monotonic in both of its arguments, which are confidence and atypicality.
Concretely, we define a simple implementation with a quadratic function τ ( a ( X ))
=
c
2 a ( X )
+ c 1 a ( X ) + c 0 , where ˆ P ATS ( X ) =
Softmax ( f ( X ) /τ
( a ( X ))) .
We optimize the parameters c 0 , c 1 , cover a calibration set to perform post-hoc recalibration.
See Appendix H.1 for implementation details.
In Figure 3c, with the top 2 plots we show the ECE analyses, and with the bottom 2 plots we show the results of performing conformal prediction after Atypicality-Aware Recalibration.
Overall, we observe that we improve the calibration and coverage rates obtained with existing methods.
For instance, we observe that coverage rates are more uniform across typical and atypical points, without explicitly performing conformal prediction across these groups.
Similarly, we obtain lower calibration errors across all groups.
In Table 1, we present the results in the tabular format.
Our results show that atypicality awareness can improve the existing recalibration methods, and in this sense, it complements existing post-hoc approaches.
Theory:
In Appendix Theorem K.2, we analyze atypicality-aware recalibration and show that it achieves lower calibration errors compared to TS.
I MPROVING D ECISION -M AKING WITH A TYPICALITY
Reporting uncertainty can improve transparency and increase user trust (Bhatt et al., 2021; Chua et al., 2022).
Babbar et al.
(2022) reports that the use of uncertainty sets improves user trust and Human+AI performance.
In the Selective Classification setting (Geifman & El-Yaniv, 2017), the model can choose when to report a prediction or when to defer to a user.
We argue that the atypicality framework gives us a way to unify these actions.
Our argument follows from the difference between the
Untrustworthy and Ambiguous quadrants.
The Ambiguous quadrant has higher-quality uncertainty sets, while the Untrustworthy quadrant has lower coverage, as we showed that atypical examples have poor uncertainty sets.
We give a simple classification setup for Selective Conformal Prediction :
[Equation]
If we can reliably predict O ( X, Y ) , then we can pick the action based on this predicted outcome.
To do so, we can train a simple classifier ˆ O
( X, ˆ P ) to predict this outcome.
To demonstrate how atypicality provides useful information over using only confidence, we train two such classifiers: 1-Using only the confidence:
ˆ O
( X, ˆ P )
=
g (ˆ P ( X )) ; and 2-Using atypicality and the confidence:
ˆ O
( X, ˆ P )
=
g (ˆ P ( X ) , a ( X )) .
If the latter performs better than the former, we understand that using atypicality can improve the decision-making process.
The exact form of g can be flexible.
In the experiments we use XGBoost to make the decisions; we refer to Appendix J for details.
To evaluate our approach, we employ the Risk-Coverage curves used in the Selective Classification literature (Geifman & El-Yaniv, 2017).
For a fixed coverage rate, the goal is to achieve lower risk by knowing when to abstain.
Here, we define the notion of risk to be the fraction of samples where the uncertainty set does not contain the true label, namely 1 [ Y / ∈ C ( X )] , and the abstention is needed.
Results: In Figure 10, we present the Risk-Coverage curves.
Using atypicality in the decision function provides strictly better Risk-Coverage curves, providing an easy way to improve the decisionmaking process.
Namely, we can better prioritize when it is better to reject the sample, and when the uncertainty set is more likely to contain the true label.
C ONCLUSION
In conclusion, our research demonstrates that atypicality offers a valuable framework to better understand model reliability and generate actionable insights.
The atypicality framework can help us better predict uncertainty, better recalibrate models, and make better decisions.
We propose that pretrained models should be released not only with confidence but also with an atypicality estimator.
While there are many other relevant notions in the literature, our main goal is to show that atypicality can provide a unifying perspective to discuss uncertainty, understand individual data points, and improve fairness.
While we analyzed only classification problems, our analyses can be naturally extended to regression or generation settings.
Furthermore, we would like to extend the theoretical analysis to more general settings, as our empirical results demonstrate that the observed phenomena hold more broadly.
In light of our findings, we further encourage the community to mitigate the adverse effects that could be caused by using models on the atypical data.
A C ODE
You can find the code to generate the analyses in this paper at the anonymized repository https://anonymous.4open.science/r/ beyond-confidence-atypicality-icml23-B2BC/ .
B A TYPICALITY IS N EEDED TO U NDERSTAND U NCERTAINTY
High-confidence and representative:
A sample and prediction are considered reliable if the model makes a high-confidence prediction that is well-calibrated.
For instance, the first image on the top right quadrant of Figure 1 is highly representative of the golden retriever class, and the model makes a high-confidence prediction for this sample.
High-confidence yet far from the support: Even though the model has high confidence in a prediction, this confidence may not be reliable.
In case the sample does not have support in the training distribution yet the model has a confident prediction (extrapolation), the prediction may not be reliable.
For instance, the second image in the top left quadrant of Figure 1 is a toy hog, and the model may not have seen similar objects during training.
Low confidence and far from distribution: A model can have low confidence in far-from-thedistribution samples.
For example, the third image in Figure 1 bottom left quadrant is an origami, which can be out-of-distribution.
We expect low-quality predictions in such cases.
Low confidence due to ambiguity: A low confidence prediction may be the result of ambiguity in a sample.
For instance, the second image in the bottom left quadrant of Figure 1 is both a hog and can be a comic book.
This is an inherent ambiguity, which is tied to the concept of aleatoric uncertainty (Der Kiureghian & Ditlevsen, 2009).
C R ELATED W ORKS C. R ELATION TO EARLIER WORKS
The notion of atypicality has been shared in earlier works.
Our distinct goal is to show that the atypicality perspective is necessary and complementary to understanding and improving various aspects of model uncertainty.
We propose that if all models are equipped with simple-to-use atypicality estimators, we can improve the reliability of the predictions.
Methodologically, Mukhoti et al.
(2021) take a very similar approach to quantify density to distinguish epistemic and aleatoric uncertainty.
However, the source of uncertainty for an atypical example could be a combination of both epistemic and aleatoric uncertainty, as noted by Mukhoti et al.
and achieving this decomposition is not our primary goal.
An image of an unusual-colored dog would be atypical with high epistemic uncertainty, yet a corrupted or highly noisy sample would be atypical with high aleatoric uncertainty.
Liu et al.
(2020) propose a related notion of ‘distance awareness’.
The underlying principles are highly relevant, however, the findings and the methodologies proposed by both works are different.
The notion of ‘out-of-distribution’ (Hendrycks & Gimpel, 2016) is closely related to atypicality, however, we do not aim to make the binary decision between ‘in-distribution’ and ‘out-of-distribution’.
For instance, we emphasize that samples could be in-distribution and atypical, e.g. rare subgroups (Sagawa et al., 2019), and our goal is to perform reliably in the entire spectrum.
D A DDITIONAL R ELATED W ORK
Uncertainty and Atypicality: Closest in spirit and methodology to our work is Mukhoti et al.
(2021); Postels et al.
(2020) where they use density estimation in the feature space to quantify and disentangle the epistemic uncertainty and aleatoric uncertainty.
Using this decomposition, they show significant improvements in active learning and OOD detection.
Similarly, Seedat et al.
(2022) use epistemic-aleatoric decomposition to characterize tabular data points.
Lee et al.
(2018) use
Mahalanobis distance to detect OOD examples, utilizing class-conditional gaussian likelihoods.
In concurrent work, Gonen et al.
(2022) reports that perplexity, a measure of atypicality, is correlated with the performance of language models in zero-shot classification.
Hacohen et al.
(2022) discuss active learning, showing that model performance depends on atypicality in different regimes of active learning.
Liu et al.
(2020) propose the relevant notion of distance-awareness in uncertainty estimation, showing that accounting for distance leads to better uncertainty quantification.
Namely, they propose architecture and training modifications to improve uncertainty quantification whereas here, we analyze the uncertainty of existing models with respect to our framework, and propose very simple, post-hoc approaches to mitigate the found issues.
Liu et al. (2019a) relate the excess risk of a binary classifier to calibration, showing that the calibration gap for discrete groups is bounded by the excess risk for the group.
Here, we extend these results and provide the characterization with respect to atypicality.
Multicalibration (H´ebert-Johnson et al., 2018; Jung et al., 2021) is a relevant notion where the calibration criterion holds for a collection of discrete groups.
The notion of ‘out-of-distribution’ (Hendrycks & Gimpel, 2016) is highly relevant to atypicality, yet our goal is not to make the binary distinction between ‘in-distribution’ and ‘out-of-distribution’.
We aim to perform reliably in the entire spectrum from atypical (e.g., rare groups) to typical (e.g., ambiguous samples).
Related findings include poor calibration under distribution shifts (Ovadia et al., uncertainty in
Gaussian Processes (Rasmussen, 2004), forgetting time for rare examples (Maini et al., 2022), the poor performance of groups with lower sample sizes (Chen et al., 2018), using energy-based models improving calibration (Grathwohl et al., 2020) showing the relationship between some notion of atypicality versus uncertainty-related quantities.
Our new findings include showing that predictions for atypical samples are more miscalibrated and overconfident, and atypicality awareness improves recalibration.
More broadly, while there are many other relevant notions in the literature, our distinct goal is to show that post-hoc atypicality estimation is a simple yet useful framework to understand and improve uncertainty quantification, and complements existing methods.
Thus, we take the position that released models should have atypicality estimators.
Recalibration: There is a rich literature on recalibration methods: Temperature scaling (Guo et al., 2017), Platt Scaling (Platt et al., 1999), conformal calibration (Shafer & Vovk, 2008; Angelopoulos et al., 2020) among many.
Lu et al.
; Romano et al.
; Barda et al.
; Bastani et al.
(2022) make a relevant observation in classification and regression respectively, showing that the coverage of conformal prediction is not equal across all groups.
They propose group conformal calibration, which requires group labels whereas our proposal is unsupervised and does not depend on any attribute information.
Concurrent work (Joy et al., 2022) explores sample-dependent TS, where they modify the training pipeline and train a separate network.
However, our parameterization of temperature is based only on atypicality, and our approach is post-hoc and cheap.
Uncertainty and Decision-Making:
Uncertainty of a model has been used in various ways.
Selective classification (Geifman & El-Yaniv, 2017; Fisch et al., 2022) proposes abstaining as an option in downstream decisions.
Conformal prediction (Shafer & Vovk, 2008; Angelopoulos et al., 2020) proposes to provide uncertainty sets instead of a single class prediction.
However, it is still an open research direction to assess the utility of these approaches.
Bhatt et al.
(2021) gives a broad overview of the topic.
Babbar et al.
(2022) studies the utility of prediction sets, showing that when used carefully they can improve user trust and Human-AI collaboration performance.
Overall, our approach demonstrates that post-hoc atypicality estimators help unify these settings and achieve improvements.
E A TYPICALITY E STIMATION
Quantifying atypicality requires access to the class-conditional / marginal distributions.
However, in practice, we do not have access to these, and hence, we need to compute the estimates.
This estimation can be challenging if the dimensionality is large, or the data is unstructured as in language or vision modalities, requiring assumptions about the distributions.
Prior work such as Mukhoti et al.
; Lee et al.
(2018) have shown that Gaussian Mixture Models in the embedding space of neural networks can be used to model these distributions.
In our experiments we use class-conditional
Gaussian distributions with shared covariance matrices, i.e. ˆ P ( X = x | Y = c ) ∼ N (ˆ µ c , ˆΣ) , to estimate atypicality.
We perform these in the penultimate layer of the neural networks used to make predictions.
The parameters are estimated using maximum-likelihood estimation with samples from the training dataset.
We explore other methods to compute atypicality, such as k -Nearest Neighbors distance as an atypicality metric.
We give implementation details of these methods in Appendix E. and report results with different atypicality metrics.
Further, we refer to the probability of the top-class, i.e. max y ˆ P ( Y = y |
X = x ) as the Confidence .
E. D ENSITY E STIMATION
To estimate atypicality, we use two ways to estimate the likelihood of a point under the training distribution.
Fitting individual Gaussians to Class Conditionals Here, we follow a similar approach to Mukhoti et al.
Namely, we model the clas-conditionals with a gaussian, where the covariance matrix is tied across classes:
[Equation]
We fit the parameters µ y and Σ with maximum likelihood estimation.
The reason to tie the covariance matrix is due to the number of samples required to fit the density.
Namely, for a d -dimensional problem, the total number of parameters to fit individual matrices become O ( yd 2 ) , which results in low-quality estimates.
Then, the atypicality becomes
[Equation]
E. C OMPUTING DISTANCE WITH K -N EAREST N EIGHBORS
k-Nearest Neighbors: Similarly, we can use the nearest neighbor distance.
Concretely, we use the nearest neighbor distance, a ( x ) =
d min ( x, D train ) =
min x ′ ∈D train | x
′ − x | , as the atypicality metric.
Alternatively, we can use different notions such as the average of k-nearest neighbors, or the distance to the kth neighbor.
Below, we report the results by using the average distance to 10-nearest neighbors.
E. F ITTING THE ESTIMATORS
For all of the atypicality estimators, we fit the estimators using samples from the training sets and make inference for the calibration and test sets.
For instance, we use the training split of ImageNet to fit the corresponding density estimator and compute the atypicality for the samples from the validation/test split of ImageNet.
All of our results using atypicality are reported for the test splits of the below datasets.
E. A TYPICALITY AND C ONFIDENCE
Are atypicality and confidence equally informative?
Beyond the data perspective given in Figure 1, here we provide quantitative results to demonstrate the difference.
In Figure 6, we provide a grid plot where the x-axis indicates the typicality quantile of a point, and the y-axis indicates the confidence of a point.
coloring the accuracy within confidence
F E XPERIMENTAL D ETAILS
Throughout this study, we focus on classification problems across a range of datasets and architectures.
We utilize publicly available datasets and models, which are detailed in Appendix F. Specifically, we evaluate image classifiers ResNet (He et al., 2016) and ViT (Dosovitskiy et al., 2020), text classifiers RoBERTa
(Liu et al., 2019b) and DistilBERT (Sanh et al., 2019), and XGBoost (Chen & Guestrin, 2016) for tabular classification.
We employ CIFAR10, CIFAR100 (Krizhevsky, 2009), and ImageNet (Deng et al., 2009) for object recognition, MNLI (Williams et al., 2018) for natural language inference, Banking77 (Casanueva et al., 2020) for banking intent classification, and the
Electricity dataset (Vanschoren et al., 2013), which is a tabular classification dataset for predicting electricity price shifts.
F. D ATASETS
Below is a full list of datasets:
ImageNet (Deng et al., 2009) from Torchvision (Marcel & Rodriguez, 2010) is an object recognition dataset with 1000 classes.
CIFAR10, CIFAR100 (Krizhevsky, 2009) from Torchvision (Marcel & Rodriguez, 2010) are object recognition datasets with 10/100 classes.
MNLI (Williams et al., 2018) from Huggingface Datasets (Lhoest et al., 2021) is a natural language inference dataset with 3 classes, indicating entailment, neutral, and contradiction outcomes.
Banking77 (Casanueva et al., 2020) from Huggingface Datasets (Lhoest et al., 2021) is a banking intent classification dataset with 77 classes.
Electricity from OpenML (Vanschoren et al., 2013) obtained through Scikit-Learn (Pedregosa et al., 2011)
For the electricity dataset, we use the average distance to 10-nearest-neighbors as the atypicality metric.
All of our experiments were run on a single Nvidia GeForce RTX 2080Ti GPU.
F. M ODELS
Most of the models are public models, e.g., obtained from the Transformers Library (Wolf et al., 2020) or Torchvision (Marcel & Rodriguez, 2010).
Below we give the full model details and how one can access them:
1. ViT ( HuggingFace Ahmed
/Vit-Cifar100 ), pretrained on Imagenet-21k (Ridnik et al., 2021) then finetuned on CIFAR100.
One can use the id given here on HuggingFace to download the model.
2. RoBERTa
( HuggingFace roberta-large-mnli ) trained on the MNLI dataset.
One can use the id given here on HuggingFace to download the model.
ResNet18 from ( Torchvision (Marcel & Rodriguez, 2010)) trained on ImageNet.
ResNet20 trained on CIFAR10, which can be downloaded from ( PyTorchCV 4 ) 5.
DistilBERT trained on Banking77(HuggingFace optimum/distilbert-base-uncased-finetuned-banking77 ) which can be downloaded from HuggingFace 6.
XGBoost : We train our own XGBoost model on the tabular dataset where we use the XGBoost library, https://github.com/dmlc/xgboost .
For all BERT (Devlin et al., 2019) style models we use the [CLS] token embeddings in the final layer, and for all vision models, we use the penultimate layer embeddings to fit the density estimators and perform the analyses.
G C ALIBRATION
We run all our experiments with 10 different random seeds, where the seeds are { 0 , 1 , 2 , . . .
Randomness is over fitting the atypicality estimators, and calibration-test splits (we use the same splits with the recalibration experiments for the sake of consistency).
G. E XPECTED C ALIBRATION E RROR
To compute ECE, we generate B = { B 1 , B 2 , ..., B M } , M equally-spaced bins where samples are sorted and grouped according to their confidence, to compute
[Equation]
where acc ( B m ) = [Equation] | B m | [Equation]
| B m |
i =1 1 [ˆ y i
=
y i ] is the accuracy for the bin m , and conf ( B m ) =
1 | B m | (cid:80) | B m |
[Equation]
=1 ˆ P ( Y =
ˆ y i |
X =
x i ) gives the average confidence within the bin.
| B m | is the size of the bin m , N is the total number of samples, and 1 [ · ] is the indicator function.
Throughout our experiments, we let the number of bins | B | = 10 by default.
G. R ELIABILITY D IAGRAMS
Here, we present the reliability diagrams decomposed by atypicality for more fine-grained analysis.
For instance, one can observe that for ImageNet and ResNet18, the direction of miscalibration is towards overconfidence.
Particularly, predictions for more atypical points are more overconfident compared to the typical points, see Figure 7.
Furthermore, if we look at the miscalibration rates after recalibration, we observe that miscalibration rates get more flat with Atypicality-Aware recalibration, improving calibration across the board, see
Figure 8.
[Figure]
u
[Figure]
b e r o f p o i n t s i n t h
e d a t a
[Figure]
[Figure]
N u m b e r o f p o i n t s i n t
h e d a t a
[Figure]
r o f
[Figure]
o i n t s i n t h
[Figure]
r o f p o i n t s i n t h
[Figure]
t
[Figure]
more overconfident for atypical points, b)
[Table]
-Aware Recalibration improves ECE
for
all groups
G. ECE AND A TYPICALITY R ESULTS WITH D IFFERENT A TYPICALITY M ETRICS
We further experiment with different atypicality metrics, such as the average distance to the 10-nearest neighbors (Figure 9).
We broadly observe that while there are slight differences in the quantitative results between different atypicality metrics, the qualitative phenomena remain intact.
[Figure]
H R ECALIBRATION
Through all our recalibration results, we first split the test set into two equally sized calibration and test splits.
Then, we fit the recalibration method using the calibration split and compute the performance on the test split.
We run all our experiments with 10 different random seeds.
H. T EMPERATURE S CALING
To perform temperature scaling (Guo et al., 2017), we use the calibration set to fit the temperature parameter.
To perform the optimization, we use the LBFGS (Liu & Nocedal, 1989) algorithm from
PyTorch with strong Wolfe line search, following Guo et al.
Namely, we optimize the parameter τ with
[Equation]
(8) and then use it during inference to rescale the logits produced by f .
H
C ONFORMAL P REDICTION
We follow the presentation in Angelopoulos & Bates (2021);
Angelopoulos et al.
Let π ( X ) be the permutation of Y = { 1 , . .
, C } that sorts ˆ P ( Y = c | X ) , i.e. the predicted probabilities for each class c .
We define a score function
[Equation]
This means greedily including classes until the set contains the true label, and using the cumulative sum of the probabilities as the score function.
We compute all of the scores for the calibration set, S calib =
{ s ( x 1 , y 1 ) , ..., s ( x N , y N ) } , we the ⌈ ( N +1)(1 − α )
⌉ N th quantile of the scores, ˆ q .
Then, the uncertainty set is defined as
[Equation]
We can further add randomization to the procedure where we have the uncertainty set function to be C ( x, u ) :
X × [0 , 1] for randomization purposes to satisfy exact coverage.
We refer to Vovk et al.
Angelopoulos et al.
Angelopoulos & Bates (2021) for a more thorough presentation.
H. A TYPICALITY -A WARE R ECALIBRATION
To perform the Atypicality-Aware Recalibration, we parameterize the temperature with
[Equation]
) which leads to ˆ P ATS
[Equation]
Similar to vanilla temperature scaling, we use LBFGS with strong wolfe search to optimize the three parameters, with the same splits as temperature scaling.
I T ABLES FOR R ESULTS
Here, we present the table version of the results in Figure 3.
Tables 1 and 2 contain the ECE analysis, and Tables 3 and 2 contain the coverage analyses for the two atypicality metrics.
[Table]
CIFAR
[Table]
005 ± 0 .0 .
004 ± 0 .0 .
004 ± 0 .0.
011 ± 0 .0 . 010 ± 0 .0 . 012 ± 0 .0. 014 ± 0 .0 .
016 ± 0 .0 .
019 ± 0 .1. 020 ± 0 .0 . 022 ± 0 .0 . 026 ± 0 .
ViT on CIFAR 0.
004 ± 0 .0 . 009 ± 0 .0 . 004 ± 0 .0. 004 ± 0 .0 .
009 ± 0 .0 .
011 ± 0 .0. 011 ± 0 .0 . 010 ± 0 .0 . 026 ± 0 .0.
042 ± 0 .0 .
045 ± 0 .0 .
079 ± 0 .1.
085 ± 0 .0 .
100 ± 0 .0 .
196 ± 0 .
RoBERTa on MNLI 0.
006 ± 0 .0 .
013 ± 0 .0 .
005 ± 0 .0.
008 ± 0 .0 .
013 ± 0 .0 .
016 ± 0 .0.
018 ± 0 .0 .
015 ± 0 .0 .
034 ± 0 .0.
0 . 029 ± 0 .0 .
031 ± 0 .0 .
061 ± 0 .1.
022 ± 0 .0 .
028 ± 0 .0 .
065 ± 0 .
DistilBERT on Banking 0.
004 ± 0 .0 .
003 ± 0 .0 .
007 ± 0 .0.
005 ± 0 .
[Table]
[Table]
[Table]
[Table]
RoBERTa on MNLI
[Table]
DistilBERT on Banking
[Table]
[Table]
[Table]
For the decision-making setup, we train 2 sets of classifiers:
Using only the confidence: ˆ O
( X, ˆ P ) = g (ˆ P ( X )) 2.
Using atypicality and the confidence: ˆ O ( X, ˆ P )
= g (ˆ P ( X ) , a ( X ))
We XGBoost to predict the outcome.
We run hyperparameter search with 5-fold cross-validation using the calibration set, over the following parameters: min child weight ∈ { 1 , 5 , 10 } , gamma ∈ { 0 . 5 , 1 , 1 . 5 , 2 , 5 } , subsample ∈ { 0 . 6 , 0 . 8 , 1 . 0 } , colsample bytree ∈ { 0 . 6 , 0 . 8 , 1 . 0 } , max depth ∈ { 2 , 3 , 4 , 5 } , n estimators ∈ { 10 , 20 , 40 } .
We refer the reader to the XGBoost documentation 5 page for the detailed semantics of each hyperparameter.
[Table]
[Table]
[Table] RoBERTa on MNLI
[Table]
DistilBERT on Banking
[Table]
[Table]
K T HEORY K. S ETTING
Data Generative Model: We consider the well-specified logistic model for binary classification with
Gaussian data, where Y ∈ {− 1 , 1 } and the probability of Y = 1 given X is defined by the sigmoid function:
[Equation]
Where I d denotes the d -dimensional identity matrix, β ∗ is the ground truth coefficient vector, σ ( x ) = 1 / (1 + e − x ) , and we have i.i.d.
observations { ( x i , y i ) } ni =1 sampled from the above distribution.
The estimator: We focus on studying logistic regression, which produces a solution that minimizes the following:
[Equation]
[Table]
± 0 .0 .
940 ± 0 .0.
887 ± 0 .0 .
902 ± 0 .0 .
919 ± 0 .0.
882 ± 0 .0 .
880 ± 0 .0 .
887 ± 0 .1.
916 ± 0 .0 .
885 ± 0 .0 .
812 ± 0 .
RoBERTa on MNLI 0.
896 ± 0 .0 .
913 ± 0 .0 .
924 ± 0 .0.
899 ± 0 .0 .
916 ± 0 .0 .
908 ± 0 .0.
899 ± 0 .0 .
903 ± 0 .0 .
902 ± 0 .0.
892 ± 0 .0 .
886 ± 0 .0 .
880 ± 0 .1.
900 ± 0 .0 .
884 ± 0 .0 .
879 ± 0 .
DistilBERT on Banking 0.
908 ± 0 .0 .
915 ± 0 .0 .
913 ± 0 .0.
900 ± 0 .0 .
909 ± 0 .0 .
908 ± 0 .0.
913 ± 0 .0 .
918 ± 0 .0 .
916 ± 0 .0.
901 ± 0 .0 .
897 ± 0 .0 .
904 ± 0 .1.
886 ± 0 .0 .
856 ± 0 .0 .
872 ± 0 .
For k ∈ {− 1 , 1 } , the confidence ˆ P k ( x ) is an estimator of ˆ P ( y = k | x ) , and it takes the form ˆ P k (
[Equation]
Calibration: Here we consider the case where P 1 ( X ) > 1 / 2 , as the case where P 1 ( X ) ≤ 1 / 2 can be analyzed similarly by symmetry.
For u ∈ (1 / 2 , 1) , the signed calibration error at a confidence level u is
[Equation]
We want to show that when X is atypical, i.e., when a ( X ) : = exp( −∥ X ∥ 2 / 2) is smaller 6 , the accuracy P ( Y = 1 | ˆ P 1 ( X ) = u ) would be generally smaller than the confidence u (over-confidence). K. P ROOF OF T HEOREM 3.  Theorem K.1 (Restatement of Theorem 3.1) .
Consider the data generative model with the algorithm described in Section 3.3.
For any K > 1 , suppose we consider the quantiles of a ( X ) , a 1 , a 2 , ..., a K , a K +1 such that P ( a ( X ) ∈ ( a k , a k +1 ])
= 1 /K for k ∈ [ K ] .
In addition, we assume ∥ β ∗ ∥ ≤ c 0 , and d/n = κ for some sufficiently small c 0 , κ > 0 .
Then for sufficiently large n , we have
[Equation]
for k = 2 , .., K .
Proof.
Following Bai et al.
(2021), we have
[Equation]
where cos ˆ θ =
[Equation]
and Z ∼ N (0 , 1) .
According to Sur & Cand`es (2019), we have ∥ ˆ β ∥ → R ∗ =
R ∗ ( κ, β ∗ ) and cos ˆ θ → c ∗ =
c ∗ ( κ, β ∗ ) , for two quantities R ∗ and c ∗ that depend on κ and β ∗ .
We then have
[Equation]
Using the proof of Theorem 3 in Bai et al.
(2021), we have that
[Equation]
where
[Equation]
for two positive constants c 1 , c 2 .
Since when z ∈ [ − 1 , 1] , z · σ ′ ( z ) and − σ ′′ ( z ) are both increasing, we then have C κ ( u ) increasing for ˆ β ⊤ x
= σ − 1 ( u ) ∈ ( − 1 , 1) .
Proving the result for { k = 2 , . .
, K − 1 }
In addition, by our model assumption x ∼ N (0 , I d ) , we have that ∥ x ∥ and x ∥ x ∥ are independent, and x ∥ x ∥ ∼ S where S is a uniform distribution on the sphere in the d -dimensional space.
As the monotonic transformations will not change the events defined by quantiles, and exp( −∥ x ∥ 2 / 2) is a monotonic function in ∥ x ∥ , for the simplicity of presentation we use a ( X ) =
∥
X ∥ in the rest of this proof.
As a result, given ∥ x ∥ = a , we have
[Equation]
where S 1 is the first coordinate of S .
Consequently, if we further condition on the event where ˆ β ⊤ x > 0 (as we assume u > 0 throughout
Section 3.3), we have
[Equation]
where Q ∼ χ 2 p − 1 , Z 1 ∼ N (0 , 1) and they are independent.
Due to the monotonicity of C κ ( u ) on u , we have that for any a 1 > a 2 ,
[Equation]
where the notation d > denotes stochastic dominance.
Consequently, we have
E
[Equation]
for k = 2 , .., K − 1 .
Proving the result for k = K
To complete the proof, it suffices to show that the inequality is also true for K th quantile:
[Equation]
]] , which is equivalent to
[Equation]
In the above inequality, the right hand side can be decomposed into
[Equation]
Denote the α quantile of χ 2 p by χ 2 α,p .
We then have a k = χ 2 kK +1 ,p .
We further decompose the equation into
[Equation]
In the following
[Equation]
we proceed to prove
E
[Equation]
We now use the approximation of the chi-square quantile: when p → ∞ , we have
[Equation]
where z α denotes the α -quantile of a standard normal random variable.
Then
[Equation]
Using the fact that z 1 − 1 K = √ 2 log K + o (1) for K → ∞ , then we have
[Equation]
In addition, for any a ∈ [ χ 2 K + δ K
+1 ,p , 2 p ] and a ′ ∈
[ a K − 1 , a K ]
[Equation]
we have
[Equation]
for some universal constant C .
Therefore
[Equation]
Then
E
[Equation]
+1 − z KK +1 )
− (1 + o (1))
δ
K + 1 · E u [( u − P ( Y = 1 | ˆ P 1 ( X ) = u ))
| a ( X ) ∈
[ a K − 1 , a K ]] .
The last equality uses the fact that P ( a ( X ) ∈ [ a K − 1 , a K ]) = 1 /K, and therefore
E u
[( u [Equation] P [Equation]
Y [Equation] |
[Equation] P 1 ( X ) [Equation]
[Equation] [Equation] [Equation] u
[Equation] [Equation] 1 , a K } ] Then use the fact that | E u [( u − P ( Y = 1 | ˆ P 1 ( X ) = u )) | a ( X ) ∈
[ a K − 1 , a K ]] | =
O (1)
and we choose δ
=
o
(1 / log K )
so δ
K
= o
( | log( [Equation] ) √ log K | ) .
Consequently,
[Equation]
which implies
E
[Equation]
[Equation] ∈
[Equation]
K
[Equation]
[Equation]
a K } ] .
We, therefore, prove equation 12 and complete the proof.
K. T HEORY : A TYPICALITY -A WARE R ECALIBRATION
We analyze atypicality-aware recalibration under the same learning setting and generative model as
Theorem 3.1.
For a predictor ˆ P , let us denote its conditional calibration error at an atypicality level γ by CE γ (ˆ P ) =
E [(ˆ P ( Y = 1 | X ) − E
[ Y | ˆ P ( Y = 1 | X )]) 2 | a ( X ) = γ ] .
Theorem K.2.
Consider the setting in Theorem 3.1.
Let the temperature function ˆ τ ( a ( X ))
=
arg min τ E [ l ( Y, Softmax ( f ( X ) /τ
( a ( X ))))]
where l is the cross-entropy loss, and ˆ P ATS ( X )
=
Softmax ( f ( X ) / ˆ τ ( a ( X ))) .
Then
[Equation]
That is, for all atypicality values γ , The proof can be found in Appendix K.4.
K. P ROOF OF T HEOREM K.
Theorem K.3 (Restatement of Theorem K.2) .
Consider the same setting as Theorem 3.1.
Suppose the temperature function ˆ τ ( a ( X )) =
arg min τ E [ l ( Y, Softmax ( f ( X ) /τ
( a ( X ))))] with l being the cross entropy loss, and let ˆ P ATS ( X ) =
Softmax ( f ( X ) /
ˆ τ ( a ( X ))) .
Then
[Equation]
For a prediction function f , we first define the conditional mean squared error of f at an atypicality level γ by MSE γ ( f ) =
E [( f ( X ) − Y ) 2 | a ( X ) = γ ] , then we have
MSE γ (
[Equation]
( f ) [Equation] E [( f ( X ) − Y ) 2 | a ( X ) = γ ] − E
[( f ( X ) [Equation]
E [ Y | f ( X ) , a ( X ) = γ ]) 2 | a ( X ) = γ ] = E [( E [ Y | f ( X ) , a ( X ) = γ ] − Y ) · (2 f ( X ) − E
[ Y | f ( X ) , a ( X ) = γ ] − Y )
| a ( X ) = γ ] =
E [( E [ Y | f ( X ) , a ( X ) = γ ] − Y ) · ( E [ Y | f ( X ) , a ( X ) = γ ] − Y ) | a ( X ) = γ ] + 2 E
[( E [ Y | f ( X ) , a ( X ) = γ ] − Y ) · ( f ( X ) − E
[ Y | f ( X ) , a ( X ) = γ ]))
| a ( X ) = γ ]
Since
E [ Y E
[ Y | f ( X ) , a ( X ) =
γ ] | a ( X ) = γ ] =
E f ( X ) | a ( X )=
γ E
[Equation]
we have
[Equation]
and therefore
[Equation]
Now that ˆ P ATS (ˆ P ( x ) , a ( x )) is monotonic on the ˆ P ( x ) , we have
[Equation]
implying
[Equation]
[Equation]
Similarly, we have
[Equation]
In the following, we will show that
[Equation]
First, as we consider the binary classification setting, with l being the cross entropy loss, we have l ( Y, Softmax ( f ( X ) /τ
( a ( X )))) =
Y log( σ ( f 1 ( X ) /τ ( a ( X ))) +(1 − {+Y ) log(
+}
− σ ( f 1 ( X ) /τ ( a ( X ))) ,
where σ ( x ) = 1 / (1 + e x ) .
Then, by the definition of ˆ τ ( a ( X )) , we have that ˆ τ ( a
[Equation]
X )
[Equation]
= arg
[Equation]
τ
[Equation]
Taking the derivative on the last line and setting it to zero, we have
[Equation]
implying
[Equation]
This makes the derivative of E [( Y − σ ( f 1 ( X ) /τ
( a ( X ))))
2 | a ( X )] zero and therefore ˆ τ ( a ( X )) is also a minimizer of E
[( Y − σ ( f 1 ( X ) /τ
( a ( X ))))
2 | a ( X )] :
[Equation]
Letting g ( γ ) =
arg min c E [( Y − σ ( f 1 ( X ) /c ))
2 | a ( X ) = γ ] , we have that
[Equation]
, and therefore
[Equation]
As a result,
[Equation]
E
[Equation]
Similarly, we have MSE γ (ˆ P ATS )
≤ MSE γ (ˆ P TS ) , and therefore equation 17 holds.
Combining with equation 15 and equation 16, we have
[Equation]
L L IMITATIONS L. Q UANTIFYING A TYPICALITY
Since we do not have access to the true distribution of P ( X ) , we estimate it through the model, e.g. using the embeddings.
This means we are capturing the atypicality not solely with respect to the training distribution but also the model.
It is possible that a model that does not fit the data well and produces low-quality atypicality estimates.
In general, we observe that our findings hold for large datasets and widely used models, and atypicality gives a semantically meaningful way to group data points qualitatively.
However, initial explorations with smaller datasets resulted in cases with noisy estimates.
Our findings suggest that we can unify the understanding and improve uncertainty quantification and recalibration methods with atypicality, however, practitioners should be careful about incorporating atypicality, as poor atypicality estimates can lead to worse performance.
L. T HEORETICAL A NALYSIS
Following the earlier work (Bai et al., 2021; Sur & Cand`es, 2019), we analyzed the calibration behavior of well-specified logistic regression.
However, our empirical findings suggest that the phenomena are much more broadly applicable.
We suggest that future work can analyze the behavior in more general settings to better understand the dynamics.