Addressing the Binning Problem in Calibration Assessment through Scalar Annotations (2024)

Abstract: Computational linguistics models commonly target the prediction of discrete—categorical—labels. When assessing how well-calibrated these model predictions are, popular evaluation schemes require practitioners to manually determine a binning scheme: grouping labels into bins to approximate true label posterior. The problem is that these metrics are sensitive to binning decisions. We consider two solutions to the binning problem that apply at the stage of data annotation: collecting either distributed (redundant) labels or direct scalar value assignment. In this paper, we show that although both approaches address the binning problem by evaluating instance-level calibration, direct scalar assignment is significantly more cost-effective. We provide theoretical analysis and empirical evidence to support our proposal for dataset creators to adopt scalar annotation protocols to enable a higher-quality assessment of model calibration.

## introduction with recently released large-scale language models (llms) demonstrating impressive few-shot, zero-shot, and task-agnostic performance @xcite @xcite , there is a boom of interest in deploying nlp-based systems to aid various human decision making @xcite . however, the black-box nature of llms gives little insight into how the predictions are made by these models @xcite , risking user trust in model prediction reliability. a common proposal to address this concern is to explore model calibration @xcite , which requires a model to approximately predict the true label distribution. this evaluation has been adopted by many recent language model benchmarking efforts @xcite @xcite ; these works often consider confidence calibration for classification and adopt expected calibration error (ece) @xcite as the main empirical evaluation metric. ece, along with variants like adaptive calibration error (ace) @xcite , involve binning in their calculation, which groups hard categorical labels into bins to approximate label distributions. this is mainly because many popular nlp tasks are annotated predominantly with categorical labels. however, these empirical evaluations are sensitive to the choice of binning schemes @xcite , and can severely underestimate calibration error @xcite @xcite . instance-level calibration @xcite ) avoids the binning issue and matches model confidence with human annotations at an individual level, as uncertainty from human annotations is a good surrogate for true label distribution @xcite . following this intuition, recent work, particularly in natural language inference (nli), has crowdsourced massive number of redundant labels per instance @xcite , which we call distributed labels. these annotations cater well to the evaluation of instance-level calibration and provide valuable insight into model behavior but are often prohibitively expensive to obtain. in this work, we propose a theoretically sound method for cost-efficient empirical calibration evaluation that can also be measured on an instance level and does not rely on binning schemes. this is done by eliciting scalar labels that score instances along a particular aspect and evaluate whether the predictive distribution is consistent with these scalars. an example that compares categorical, distributed, and scalar annotations can be found in in particular, we can use scalar labels to form a lower bound of the calibration error. we define the expected calibration error w.r.t. d to be notice that we relax the requirement on d by @xcite to allow asymmetric critic function. this is to allow evaluations like confidence-calibration @xcite to be included in this framework as well. however, since p(y |@xmath0, evaluating directly against e d is generally infeasible. one way to approximate pr(y |@xmath1. alternatively, a large number of labels can be collected @xcite to approximate each instance's conditional label distribution, against which model predictive distribution for each instance is evaluated. instance-level calibration @xcite like this does not rely on predefined binning schemes as it requires the model to predict conditional label distribution perfectly. however, the high cost of obtaining these labels makes it very hard to upscale @xcite . often, many practical decisions have to be made during data collection to limit annotation cost @xcite , leading to suboptimal evaluation. we motivate our work by considering a special family of critic functions d that there exists some scoring rule ψ : δ k-1 → r against which the critic function can be written as: that is, the critic function can be calculated by just comparing ''score maps'' of the two distributions. this inspires new annotation potentialities: can we directly annotate the transformed scores ψ(p(x)) instead of the original labels y or the the redder color on the heatmap implies a higher probability of label distribution. the height of the bars corresponds to the human uncertainty scalar labels obtained from unli @xcite . the correspondence between these two sets of labels suggests the existence of a scoring rule that maps chaosnli labels to a scalar with limited information lost. ## unli: a scoring function example in this section, we use unli for a case study on a specific scoring function ψ. for a given nli instance, chaosnli @xcite elicits 100 redundant hard labels per instance to approximate true label distribution pr(y |x), while unli @xcite elicits 2-3 scalar labels per instance to estimate the probability that a hypothesis is true given a premise. despite previously considered mismatched in distribution and unrelated @xcite , we argue that these two labeling schemes are closely related. specifically, figure 1 shows that unli labels preserve an implicit ordering of the chaosnli label distribution. as the probability mass of label distribution for each instance gradually shifts from con to ent through neu, the unli score also increases. also, it is rare for an instance to have a high contradiction and entailment probability at the same time, supporting the intuition that neutral is the intermediate state be-tween contradiction and entailment. these observations suggest that a scoring function ψ from chaosnli to unli can be found which preserves most of the information in the label distribution that we care about, such as: what's the likelihood of a hypothesis, whether some hypotheses are more likely than the others, given their premises, etc. suppose we want to evaluate whether our model is well-calibrated on snli at the instance level, namely, to test whether it provides humanaligned uncertainty for each instance. instead of directly evaluating whether the model's predictive distribution is consistent with the chaosnli label distribution, we can transform model distribution with this ψ and compare it with unli labels to avoid the massive annotation overhead of distributed labels, as described in the following section. 1 4 scalar label for calibration we give theoretical guarantees for using scalar annotations for calibration evaluation in this section. to begin with, we first define the class-wise calibration error with which we assume calibration error is evaluated: definition 1 (class-wise calibration error). consider a k-class classifier p : @xmath2. the classwise calibration error l cce is defined as: remark. this calibration error is fully compatible with the expected calibration error definition in section 2, with a critic function d marked as above. definition 2 below gives a simple function family that maps a discrete distribution to a scalar value. we then show that specific probing tasks on these transformed scalar values provide a useful proxy for evaluating model calibration in the form of a lower bound. here, we only showcase our main theorem that supports direct transfer from ranking to multi-class (@xmath3, and leave additional results that guarantee similar lower bounds from regression or ranking to binary or multiclass classifications and their proofs to appendix a. definition 2 (expected label scoring rule). for a k-way classification problem with label set @xmath4, . . . , y k }, for a given function f : @xmath5: remark. without losing generality, in this paper, we assume f (y) ≥ 0, ∀@xmath6. since we do not require a calibration loss to be calculated against this scoring rule, f does not need to be normalized as for calibration lens in @xcite . it's worth noticing that we are particularly interested in cases where f has an intuitive interpretation in empirical evaluation. for example, given a label set y , f (y) = i[@xmath7, estimating whether the model understands the concept related to a label set reminiscent of the annotation scheme proposed by @xcite and @xcite . the following result is an example demonstrating how comparing induced scalars to their corresponding human uncertainty labels can be used to evaluate model calibration when the probing task on the scalar labels is a ranking problem. previous research has shown when the annotation protocol is properly designed, rankings among instances can be consistent even when individual scores are not @xcite @xcite . we show that empirical loss functions like the pairwise ranking risk defined below can be used to form a lower bound to the original calibration error. for a pair of instances (x, z), (x , z ) ∼ x × r identically and independently distributed (iid), the ranking problem is defined as correctly predicting whether @xmath8. thus a ranking rule is a function r : x × @xmath9, 1}. the ranking risk w.r.t. to a ranking rule r can be defined as: for the original classifier and the corresponding ranking risk regret, we have the following: theorem 1. under mild assumptions (assumption 1), for a reasonably calibrated ordinal classifier p s.t.: for some δ > 0, the relative ranking risk regret to the expected error scoring rule is bounded by: remark. this is particularly useful for challenging annotation tasks where annotators are hard to calibrate; otherwise, we can directly compare with scalar values. regression tasks also provide similar lower bound guarantees that can be found in the appendix a. we also refer to a detailed discussion about how assumption 1 ensures the existence of an intuitive ranking in the appendix a. per the discussion above, to evaluate the calibration of a classifier p, we should compare the figure 2: a hypothetical example of applying definition 2 to an emotion classification task (e.g., goemotions @xcite ). by attaching a single scalar valence value to each class label, we specify how the respective conditional label distribution of instances , , and induces an estimation of its valence, which can then be annotated. we would expect the induced valence score by a calibrated classifier p to correlate well with human annotations. this is reflected by the conformity of rankings (evaluated with ranking risk) and closeness in scoring. it's also worth noticing that our formulation does not require the scalar annotation task to fully recover the classification task, and it's possible that a classification task can be characterized by multiple valid mapping functions. for example, figure 3 shows a different ordering of the same set of instances induced by an ''arousal'' mapping function g. ## pseudo distributed label from scalars this section discusses ways to map scalar labels back to label distributions. this is useful when one wants to augment classification training with scalar annotations, for example when performing distillation @xcite or label smoothing @xcite . if we have parametrized distribution information of scalar annotations through common aggregation techniques @xcite , backmapping is equivalent to trying to quantize a continuous distribution p(y) to a discretized distribution q(y). although there already exist heuristics for allocating probability mass to categories @xcite @xcite , these mappings are generally considered suboptimal @xcite . we propose two more principled ways to do label back-mapping: (1) inference with a neural network; and (2) distribution quantization with fixed support. ## experiments our experiments intend to validate two critical arguments in this paper: (1) scalar labels effectively evaluate models' uncertainty estimation; ## conclusions and future work we show that scalar annotation elicited from individual humans can be a valuable resource for developing calibrated nlp models. both our theoretical and empirical results suggest that scalar annotation is an effective and scalable way to collect ground truth for human uncertainty, and we encourage future datasets to include scalar annotation if applicable. our result provides an interesting perspective for researchers to devise new annotation tasks on traditionally categorical tasks. future research may also look into conditions where scalar label-based uncertainty examination has better guarantees, investigate better ways to robustly collect consistent scalar annotations, and other principled ways to train or evaluate with scalar labels, particularly in areas where direct application of the method isn't immediately available, such as nlp tasks where structured predictions are involved, probably requiring some decisions specific to the task to be made regarding matters such as events of interest @xcite . proof. according to equation 1, the scoring rule s and s can be written as: thus the expectation of the scoring rule mae can be written as: notice that since the classification is binary we have for any @xmath10: similar results can be derived for @xmath11, in the form of the following theorem: theorem 4 (@xmath12. for a k-way classification problem, given a reasonably calibrated classifier p : @xmath13.t.: proof. similar to the last proof we switch in equation 1 to the mae: