Interpreting Predictive Probabilities: Model Confidence or Human Label Variation? (2024)

Abstract: With the rise of increasingly powerful and user-facing NLP systems, there is growing interest in assessing whether they have a good _representation of uncertainty_ by evaluating the quality of their predictive distribution over outcomes. We identify two main perspectives that drive starkly different evaluation protocols. The first treats predictive probability as an indication of model confidence; the second as an indication of human label variation. We discuss their merits and limitations, and take the position that both are crucial for trustworthy and fair NLP systems, but that exploiting a single predictive distribution is limiting. We recommend tools and highlight exciting directions towards models with disentangled representations of uncertainty about predictions and uncertainty about human labels.

## introduction in common language, uncertainty refers to "a state of not being definitely known or perfectly clear; a state of doubt". 1 in statistics and machine learning, uncertainty is taken as a state to be represented @xcite -the state of the world as a function of inherently stochastic experiments or the state of knowledge of an agent observing or interacting with the world-and its mathematical representation requires prescribing a probability measure @xcite . in modern nlp, neural networks are the de-facto standard to predict complex probability measures from available context @xcite : given an input (or prompt), a neural network prescribes a representation of uncertainty over the space of responses (e.g., strings or classes), typically, by mapping the input to the parameter of a probability mass function (e.g., in text classification, inputs are mapped to the probability masses of each outcome in the label space). 1 oxford english dictionary, accessed october 13th 2023. recently, transformer-based large language models (llms) are becoming increasingly powerful and display remarkable abilities on complex classification tasks, leading to an increased deployment in user-facing applications. this motivates the need for models that can signal when they are likely to be wrong (p1; an aspect of trustworthiness), and models that can capture different linguistic and human interpretations (p2; an aspect of language including fairness). in this position paper, we identify that the exact same representation of uncertainty-the predictive distribution over outcomes-is sometimes interpreted as an indication of confidence in model predictions (p1; @xcite @xcite and other times as an indication of variation in human perspectives (p2; @xcite . we hope to provide clarity and accelerate progress by: ## two perspectives on uncertainty consider a user-facing question answering (qa) system. ideally, this model is able to abstain on questions that it is likely to get wrong (a.k.a. selective answering or prediction; @xcite , for which its probabilities should reflect confidence in predictions (i.e., predictive probabilities help us determine whether the model is right or wrong). now consider that various nlp tasks, including qa, are being acknowledged as supporting human label variation @xcite , and that some questions can be underspecified, ambiguous or subjective (there are many such datasets, for qa see for example @xcite and @xcite , and for other tasks see section 3.2). different annotators might therefore provide a different reference answer. from this perspective, probabilities should reflect the relative frequency of each answer assigned to that particular question by the pool of annotators (i.e., predictive probabilities help us determine what answers represent the views of a certain population). these two perspectives on the role of predictive probabilities in fact aim at different sources of uncertainty: uncertainty about model error (e.g., due to imperfect design and estimation) and uncertainty about human labels (e.g., due to label variation in a population). so, if a model predicts a uniform distribution, does this mean that all answers are plausible or that this prediction should not be trusted? ## merits and limitations the predictive distribution for an input x is sometimes taken as a representation of uncertainty about a model's future classification performance ("knowing when it knows"); other times as a representation of uncertainty about label frequency in a population of human annotators (human label variation). we now discuss merits and limitations for each perspective. ## best of both worlds we argue that the desiderata behind both perspectives are equally important for trustworthy and fair nlp systems, but that expecting the predictive distribution to represent both is limiting. rather than calibrating the predictive distribution to better indicate model error, we outline alternative directions to capture uncertainty about predictions (towards more trustworthy nlp) and uncertainty about human perspectives (towards fairer nlp)where the latter can, and in our view should be represented by the predictive distribution. ## conclusion in this position paper, we identified two important perspectives on the predictive distribution in nlp. we believe that the desiderata behind both are crucial for fair and trustworthy nlp systems, but that exploiting the same predictive distribution is limiting. we recommend exiting tools and directions to represent uncertainty about predictions (model confidence) and about label variation (human perspectives). we hope to facilitate a better understanding of uncertainty in nlp, and encourage future work to acknowledge, represent and evaluate multiple sources of uncertainty with principled design decisions.