Surprisal reveals diversity gaps in image captioning and different scorers change the story (2025)

Abstract: We quantify linguistic diversity in image captioning with surprisal variance – the spread of token-level negative log-probabilities within a caption set. On the MSCOCO test set, we compare five state-of-the-art vision-and-language LLMs, decoded with greedy and nucleus sampling, to human captions. Measured with a caption-trained n-gram LM, humans display roughly twice the surprisal variance of models, but rescoring the same captions with a general-language model reverses the pattern. Our analysis introduces the surprisal-based diversity metric for image captioning. We show that relying on a single scorer can completely invert conclusions, thus, robust diversity evaluation must report surprisal under several scorers.

## introduction every person is unique in their linguistic behaviour: the same image may evoke many different yet valid descriptions depending on their different personal contexts, beliefs, attention, and background knowledge. this "natural variability" has been documented in linguistic annotation @xcite , inference @xcite , object naming @xcite , and image captioning @xcite . on the other hand, although vision-and-language models may capture fragmets of individual styles, they are normally not trained with capturing diversity in mind. they are optimised for maximum-likelihood estimation (mle), a function that rewards word combinations already dominant in the training corpus and penalises rarer phrasing @xcite . decoding tweaks (e.g. nucleus sampling @xcite ) or alternative objectives @xcite may mitigate this effect. still, transformer-based models trained with mle now achieve impressive accuracy on caption benchmarks @xcite @xcite . however, such benchmarks assume a single ground truth and often focus on targeting this very benchmark, meeting a certain metric score @xcite . as a result, the diversity of model language is often sidelined -even though prior studies argue that missing variability is critical for evaluating multi-modal models @xcite @xcite . existing diversity metrics for tasks such as image captioning @xcite , rely on surface counts (length, type-token ratio, distinct-n) and lack a principled link to linguistic processing. in this study we examine diversity of image captions through the notion of surprisal, a probabilityweighted and context-sensitive probe. the surprisal, a negative log-probability of a word in context, has been widely used in linguistic analysis to quantify the processing cost of a message @xcite and has proved to be a robust metric across languages @xcite . generic, more commonly used words and phrases across speakers or models are easier for either to produce, hence any potential differences in the measured surprisal between these two groups naturally reveal any differences in diversity of their linguistic behaviour @xcite @xcite . we extend this line of work from text-only settings to image captions. we estimate the probabilities used to measure surprisal using two probabilistic scorers (i) an ngram model trained on a large corpus consisting of a balanced collection of human and model generated image captions (in-domain expert), and (ii) gpt-2 @xcite , a general-purpose transformer trained on broad english. as we do not have access to the true population of all possible captions for images, we follow earlier work @xcite @xcite and approximate the underly-ing distribution with two different language models. inspired by the scorer-sensitivity findings of @xcite , our dual-scorer design tests whether conclusions about diversity change with the change of the evaluator of surprisal. our research questions are: as artificial describers (description generators) we analyse five state-of-the-art vision-andlanguage models and compare their descriptions with human captions. like @xcite , who used linguistic theory to identify syntactic gaps in llm-generated news compared to humanauthored news, we resort to the linguistic theory of surprisal to evaluate multi-modal language, showing how scorer choice interacts with caption distributions. our study reveals a clear difference in surprisal and therefore diversity between human and model captions, and, importantly, that the direction of this diversity flips with the chosen scorer. these differences between evaluation scorers underscore the need for future benchmarks to include surprisal-based diversity scores computed under several, well-motivated scorers. ## experiments and results before moving to our surprisal-based experiments, we first ask if model captions even look like human captions on lexical level. we compute several metrics described in @xcite ; the results are shown in in other words, verbosity does not necessarily translate to greater genuine lexical creativity -instead, it may reflect a tendency to "pad" responses with familiar constructions, a behaviour aligned with next-word prediction training objectives rather than pragmatic efficiency in describing images. overall, the results in for n-gram models, training data for each language-model order n (under "data" column) is the union of the human corpus h and machine-generated texts m produced with the indicated decoding strategy (greedy or nucleus sampling). the same data was used to test pre-trained gpt-2. on the properties and features of the produced captions is also important @xcite . two captions can be similar or different in terms of length and number of types, yet be radically different in how predictable or information-rich they are. look at the example in figure 1 . both captions mention the same objects (monitors, laptops) and share word types, but the model's version explores far less frequent phrases ("workspace setup". "green-themed image") and long chains of modifiers that could carry a different amount of information content. surprisal under scorers can capture these probability-weighted differences that surface counts alone cannot. in other words, surprisal lets us measure not just how many different words ap-pear, but how unexpectedly they are combined. one striking difference is that models tend to overdescribe visual information exhaustively, while humans focus on particular elements. this selectivity may also allow for greater diversity since different humans can choose to highlight different aspects of an image. if humans are behaving pragmatically, they generate optimal (concise) captions, and anything longer would be pragmatically inappropriate. multi-modal llms, by contrast, struggle to capture captioning intent because they produce much longer texts. this is an interesting finding as it raises the question of what constitutes a good model-generated description. we leave this discussion for future work. next, we move to our main experiments. ## conclusion our study provides three main messages. first, using surprisal as a measure of diversity gap in image captioning requires reporting under multiple scorers. doing so will (i) prevent over-interpretation of scores tied to one distribution, and (ii) encourage future models to match human-level variability inside the caption genre rather than describing images in general language. the apparent reversal in variance of surprisal within describer groups is a diagnostic of whose expectations one chooses as their scorer. the two different scorers we employ in this short study should be seen as two complementary metrics. our work is thus helpful when deciding which scorer to choose given a particular task.