Predict the Next Word: <Humans exhibit uncertainty in this task and language models _____> (2024)

Abstract: Language models (LMs) are statistical models trained to assign probability to human-generated text. As such, it is reasonable to question whether they approximate linguistic variability exhibited by humans well. This form of statistical assessment is difficult to perform at the passage level, for it requires acceptability judgments (i.e., human evaluation) or a robust automated proxy (which is non-trivial). At the word level, however, given some context, samples from an LM can be assessed via exact matching against a prerecorded dataset of alternative single-word continuations of the available context. We exploit this fact and evaluate the LM’s ability to reproduce variability that humans (in particular, a population of English speakers) exhibit in the ‘next word prediction’ task. This can be seen as assessing a form of calibration, which, in the context of text classification, Baan et al. (2022) termed calibration to human uncertainty. We assess GPT2, BLOOM and ChatGPT and find that they exhibit fairly low calibration to human uncertainty. We also verify the failure of expected calibration error (ECE) to reflect this, and as such, advise the community against relying on it in this setting.

## introduction language models (lms) are trained to assign probability to human-generated text. the typical lm treats a piece of text as a sequence of tokens whose joint probability it factorises autoregressively, with conditional token probabilities predicted from the available context by a neural network @xcite @xcite . an lm can be viewed as a representation of uncertainty about human linguistic production @xcite @xcite , specifically, one that reflects the production variability exhibited by the population(s) who generated the training data. despite how plausible this variability is, lms are not consistently exposed to it at the level of individual contexts (i.e., due to data sparsity, most contexts are unique) leading us to investigate their ability to predict it well. one way to appreciate plausible variability is to ask humans to perform next word prediction: show multiple participants the same prefix of a passage and ask each of them to contribute a word that plausibly extends it. an lm that assigns probability to any next-word candidate similar to the proportion of the human population contributing it as the next word serves as a good proxy to the production variability of that human population-a desideratum @xcite termed calibration to human uncertainty. 1 studying different notions of cal-ibration of text classifiers, @xcite show that the very popular expected calibration error @xcite is flawed in the presence of data uncertainty (e.g., due to the task's inherent ambiguity @xcite ). as data uncertainty is hardly avoidable in language modelling, we must entertain the possibility that ece is not a reliable tool to assess the predictive distributions of an lm, despite its widespread use @xcite @xcite . to assess calibration to human uncertainty, we compare the uncertainty exhibited by lms to the uncertainty exhibited by humans in the next word prediction task (figure 1 )-for which we use provo corpus @xcite , a dataset (in english) with multiple human responses per available context. we analyse three pretrained lms of different sizes and training objectives (i.e., gpt2 @xcite , bloom @xcite and @xcite ) and find that they exhibit low calibration to human uncertainty. we verify ece's unreliability in this setting and advise the community against relying on it as a meaningful notion of calibration of generative models. ## background given context, an autoregressive lm predicts a conditional probability distribution (cpd) over the model's vocabulary of known tokens (i.e., subword units). hence, at this level, an lm can be regarded as a probabilistic multi-class classifier. this motivates research @xcite @xcite assessing the extent to which probabilities predicted by lms are interpretable as 'rate of correctness', a property referred to as calibration @xcite @xcite . a multi-class classifier is said to be confidencecalibrated if its probabilities predict the classifier's accuracy, specifically, if (100 × q)% of its predictions made with probability (close to) q are judged to be correct. the ece estimator @xcite is the average absolute difference between average confidence and frequency of correctness across confidence bins. 2 @xcite uncovered a logical flaw in measuring ece under data uncertaintyof interest, e.g. a specific target audience in a human-machine interaction setting (e.g. @xcite ). ## methodology we compare the uncertainty that lms and humans exhibit in next word prediction. for that, we must represent their uncertainty over a shared space. human distributions. given some context c, we assume that human uncertainty is captured by a single underlying cpd and, hence, regard human responses to the next word prediction task as i.i.d. draws from it. then, given multiple responses, the mle for this cpd assigns probability p(w|c) to word w given c equal to the relative frequency with which humans predict w to follow c. ## experiments data. provo corpus (luke and christianson, 2018) contains 55 passages (50 words long on average) in english from various sources e.g. news, fiction, science. each prefix sequence of all passages (2687 prefixes) is given as context to 40 humans, on average, who predict a one-word completion. we use this corpus to estimate target cpds. ## results main findings. what do tvd differences mean? we measure a difference of around 0.2 tvd units between gpt2's and oracles' means, but, we lack understanding of the practical significance of this difference. to gain some insight, we conduct a controlled experiment. we artificially improve k% of the model's cpds by replacing them by an oracle estimate. we then measure tvd between this artificial improvement and a disjoint oracle allowing us to associate units of tvd with an interpretable rate of improvement (i.e., percentage of plausible cpds). we find that we need to replace about 60% of gpt2's cpds to achieve tvds that distribute similarly to human performance. 9 gold label ## related work there has been work that exploits predictive distributions of lms in various ways. @xcite analyses such distributions and finds that they overestimate the probability of ill-formed sequences. others investigate alternative training signals that minimise the distance between the data and model distributions @xcite @xcite . our work exploits predictive distributions as an uncertainty representation of human linguistic production and study their calibration. several works study how well-calibrated lms are and how to alleviate miscalibration @xcite @xcite @xcite @xcite -the majority using ece to substantiate their findings, whose inadequacy makes us believe that a new round of studies is needed to assess this matter; our work being an example. there is a line of work that stresses the value of obtaining multiple human labels per input @xcite @xcite 2020 ) analyse the miscalibration of lms from a psycho-linguistic lens, and fine-tune an lstm model using multiple labels. our work is an addition to this line of work. ## conclusion our work joins a stream of work acknowledging and better incorporating data uncertainty into evaluation protocols @xcite . in particular, we find empirical evidence for ece's unreliability in this setting and advise further research into calibration of lms not to use it. with a more appropriate tool, we analyse three modern pretrained lms and find that they are not well calibrated to human uncertainty, unlike ece might suggest. we believe that this inability stems from models not being consistently subjected to human production variability during training, and plan to investigate this further in future work.