Coreference as an indicator of context scope in multimodal narrative (2025)

Abstract: We demonstrate that large multimodal language models differ substantially from humans in the distribution of coreferential expressions in a visual storytelling task. We introduce a number of metrics to quantify the characteristics of coreferential patterns in both human- and machine-written texts. Humans distribute coreferential expressions in a way that maintains consistency across texts and images, interleaving references to different entities in a highly varied way. Machines are less able to track mixed references, despite achieving perceived improvements in generation quality. Materials, metrics, and code for our study are available at https://github.com/GU-CLASP/coreference-context-scope.

## introduction generative models produce text that many perceive as increasingly human-like. however, machinegenerated text conceals important distinctions to which people are sensitive @xcite . work on visual narrative shows that there is still a gap between human and machine ability to generate coherent text @xcite . a key difference between human and machine writing behaviour is the distribution of coreferential elements in english. we find that texts generated in a multimodal task setting have considerably different distributions of transitions between coreference chains, i.e., chains of referring expressions pointing to the same entity. in this work, we describe our methodology, using a set of metrics that we apply to state-of-the-art multimodal language models in a visual storytelling task. all tested models show substantial differences in measured behaviour with respect to human-generated reference texts. this points to a need for a quantitative approach to coherence that accounts for the characteristics of coreference in texts generated from images by machines. a proper handling of reference has implications for the ability of these models to perform multi-modal grounding, reasoning and inference tasks in the way humans expect @xcite . our contributions are: humans build stories in a certain way, focusing on visual events and the logical connections and participants involved in them. event participants take the form of characters in the stories, with some reappearing as the story unfolds. this character introduction and reprisal is precisely what coreference resolution encodes. coreference resolution identifies referring expressions or mentions in a long text and chains them into distinct entities or coreference chains @xcite . the type of mention is also influenced by the salience of a character -whether they are newly introduced or already known @xcite . work on generation of visual narratives is often evaluated on automatic scores such as bleu @xcite or rouge @xcite ), but they correlate poorly with human judgements @xcite . other metrics that look at trigram repetitions @xcite or differences in distributions @xcite have a higher correlation with fluency as perceived by humans. however, these metrics do not capture the tellability of a story. in this study we use surface-level reference patterns to capture key aspects of tellability, including continuity, salience, and character switching. our main contribution is a diagnostic framework for evaluating narrative consistency, demonstrating how coreference-based features can distinguish between human and model-generated stories in a multimodal setting. ## results and analysis we begin by examining the results in humans refer to fewer entities on average (3.85), but each is mentioned multiple times (6.87) suggesting a focused narrative. the chain crossing index (cci) measures how often two chains intersect, excluding overlaps or disjoint chains. a low human cci (0.89) reflects consistent reference to key characters, with less frequent switching between entities. gemini 2.0 flash is closest to humans across all metrics, though it sometimes produces 0 mentions, indicating inconsistency in referencing characters. while its structure appears humanlike, it may omit key entities entirely. in contrast, deepseek-vl2-4.5b generates more entities, longer chains, and more frequent crosschain intersections (high cci), reflecting overgeneration. gpt4o produces fewer chains overall but tracks characters more consistently than deepseek-vl2-4.5b. the results in this trend is reinforced by mcc scores, where humans outperform all models, indicating stronger alignment between text and image character mentions. two-sample t-tests confirm the difference is significant in every case (@xmath0.05), with effect sizes (cohen's d) ranging from 0.45 to 0.83, reflecting moderate to large differences. finally, while average rec across models is similar, their relationship with text length differs. we observe that only models show increasing variation in referring expressions with longer text, e.g., gemini 2.0 flash. human stories remain consistent in how they refer to characters, regardless of length. an interesting trend is that larger models often change characters more frequently than smaller ones, indicating weaker character consistency. however, as to support this, we report pearson correlation between charch and mcc in ## conclusions and future work we found substantial differences in coreferential patterns between llm and human outputs. the model ρ deepseek-vl2-4.5b -0.236** deepseek-vl2-1b -0.263** gemini 2.0 flash -0.189** gpt4o -0.248** internvl2.5-78b -0.251** qwen2-vl-72b -0.238** qwen2-vl-7b -0.188** human -0.040 future work will explore llm attention patterns to better understand biases in reference and coreference generation. we are also actively considering interpretability experiments to probe how models internally represent characters during generation, for example, whether attention aligns with character mentions in images. in addition, we aim to conduct human evaluations to better understand what makes a "good" story.