Stop Measuring Calibration When Humans Disagree (2022)

Abstract: Calibration is a popular framework to evaluate whether a classifier knows when it does not know - i.e., its predictive probabilities are a good indication of how likely a prediction is to be correct. Correctness is commonly estimated against the human majority class. Recently, calibration to human majority has been measured on tasks where humans inherently disagree about which class applies. We show that measuring calibration to human majority given inherent disagreements is theoretically problematic, demonstrate this empirically on the ChaosNLI dataset, and derive several instance-level measures of calibration that capture key statistical properties of human judgements - including class frequency, ranking and entropy.

## introduction neural text classifiers are becoming more powerful but increasingly difficult to interpret @xcite . in response, the demand for transparency and trust in their predictions is growing @xcite @xcite . one step towards understanding when to trust predictions is to evaluate whether models know when they do not know-i.e., whether predictive probabilities are a good indication of how likely a prediction is to be correct-known as calibration. this is crucial in user-facing and high-stake applications. calibration, and particularly the expected calibration error (ece; @xcite , is widely studied in machine learning and computer vision @xcite , and is gaining increased attention in natural language processing (nlp; @xcite @xcite . an important implicit assumption in the widely used definition of perfect calibration proposed by @xcite is that predictions are either right or wrong-in other words, that the true class distribution, i.e., human judgement distribution, is deterministic (one-hot). however, for many problems, while categories exist, their boundaries are fluid: there exists inherent disagreement about labels. this means that gold labels are at best an idealization-as irreconcilable disagreement is abundant @xcite @xcite @xcite . evidence for this can be found in various tasks, including those which involve linguistic and subjective judgements @xcite . surprisingly, however, while limitations of calibration are studied ( §2), this fundamental assumption is ignored. in this work, we show that popular calibration metrics-such as ece-are not applicable to data with inherent human disagreement ( §3). we propose an alternative, instance-level notion of calibration based on human uncertainty, and operationalize it with several measures that capture key statistics of the human judgement distribution other than matching the majority vote ( §4). finally, we verify our theoretical claims with a case study on the chaosnli dataset, and investigate temperature scaling-a popular post-hoc calibration methodthrough the lens of human uncertainty ( §5). classification a probabilistic classifier approximates π π π(x) with a trained parametric function (e.g., bert; @xcite that maps an input x to a vector f (x) of class probabilities. after training, and given an instance x, we typically map the model's output f (x) to a single decision ŷ ∈ [c]. more often than not, this is the mode of the model distribution: ŷ = arg max c f c (x). the correctness of ŷ is assessed against the observed human 'gold standard' decision y ⋆ = arg max c πc (x). calibration a classifier is multi-class calibrated @xcite if, for all instances mapped to the same vector q, the relative frequency with which c is correct (assessed against the gold standard) is q c for every c: consider a problem with three classes. a model is multi-class calibrated if, for all instances mapped to the same vector, e.g., (0.90, 0.07, 0.03) ⊤ , predicting the first class would result in a correct decision for 90% of these instances, the second class for 7%, and the third class for 3%. estimation of the lefthand side (lhs) of eq(1) by counting is difficult as it requires observing multiple instances mapped to the same probability vector. a weaker notion of calibration (popular in nlp; @xcite is confidence calibration @xcite a model is confidence calibrated if, for all instances mapped to a maximum probability value p (e.g., 0.9), the most probable class under the model is correct for 90% of these instances. expected calibration error is most often used to measure (confidence) calibration in practice. @xcite originally proposed ece for binary classification and @xcite later adapted it to a multi-class setting: ece estimates the confidence calibration errorabsolute difference between the lhs and the rhs of eq( 2) -in expectation by discretizing the probability of the model decision into a fixed number m of intervals (or bins). each prediction vector f (x) is assigned to a bin b m based on its highest probability max c f c (x). the ece is the weighted average of the difference between the average confidence and accuracy per bin. to obtain zero calibration error, if 90 out of 100 instances that received a highest probability between 0.8 and 1.0 are correctly classified, the average confidence on those 100 instances must be 0.9. several recent studies identify and address problems with ece-mostly with its binning scheme and implicit decision rule (e.g., @xcite @xcite @xcite . instead, in this work we identify a fundamental problem in the definition of perfect calibration when applying it to setups where there exists no real gold label. ## calibration & disagreement pathology it is common practice to handle human disagreement with majority voting or other aggregation methods @xcite @xcite . aggregate (gold) labels are then used to evaluate a classifier's accuracy. we now illustrate the problem this poses when measuring calibration. desideratum: any classifier g that, given an instance x, predicts the human judgement distribution g(x) = π π π(x) should be perfectly calibrated. consider the oracle classifier that has access to the mle π π π(x) of π π π(x) for any instance x in a validation set. for each x, the oracle is able to predict the human labeling uncertainty π π π(x). this estimate is unbiased and becomes more precise the more judgements we have access to. by definition, when human majority voting is used, this classifier achieves perfect accuracy-its highest confidence prediction always matches the gold standard. however, according to ece, the oracle classifier is miscalibrated (this is true for other definitions of calibration to accuracy, including multi-class and classwise). recall that the calibration error is the absolute difference between accuracy and average confidence per bin. the accuracy of the oracle classifier is always 1. on data where humans disagree, the average confidence will be lower than 1. this mismatch results in high calibration error (as demonstrated in §5) and exposes a problem with using ece to measure calibration on disagreement data. an important takeaway is that, even if we can train a classifier that perfectly models the human judgement distribution, this classifier would still be severely miscalibrated. to achieve perfect calibration, its probabilities must drift towards an unfaithful representation of human confidence. therefore, we argue that human majority accuracy is a bad estimate of correctness to calibrate against. ## calibration to human uncertainty to get a faithful probabilistic classifier, we expect it to predict the uncertainty the human population exhibits on any given x. notions of calibration to accuracy (e.g., multi-class, classwise, confidence) are defined marginally (i.e., for instances grouped by a property of model predictions such as their probability). instead, we argue for a direct assessment of calibration at the instance level. given x, perfect calibration to human uncertainty requires: with which humans label x as c. with multiple human judgements (whether or not they disagree), the lhs can be estimated by π π π(x)-the relative frequency of the observed labels. assessing the degree to which eq(4) holds in expectation across instances gives us a tool to criticize classifiers in terms of their overall calibration to human uncertainty in a given task. this is appealing because we can assess the trustworthiness both globally (overall calibration) as well as on individual predictions. ## conclusion we demonstrate a fundamental problem with measuring calibration to the human majority vote in settings with inherent disagreement in human labels. we propose an alternative, instance-level notion based on the full human judgement distribution and operationalize this notion with three metrics. we study temperature scaling roberta on the chaosnli dataset using these metrics and conclude that they-and crucially, the ability to inspect them in distribution-provide a more robust and faithful lens to analyze classifier calibration in disagreement settings. human uncertainty can be used to evaluate many other calibration techniques-we only performed a preliminary analysis for temperature scaling-and we encourage the community to look into those, in addition to exploring other datasets with inherent disagreements.