Burn After Reading: Do Multimodal Large Language Models Truly Capture Order of Events in Image Sequences? (2025)

Abstract: This paper introduces the TempVS benchmark, which focuses on temporal grounding and reasoning capabilities of Multimodal Large Language Models (MLLMs) in image sequences. TempVS consists of three main tests (i.e., event relation inference, sentence ordering and image ordering), each accompanied with a basic grounding test. TempVS requires MLLMs to rely on both visual and linguistic modalities to understand the temporal order of events. We evaluate 38 state-of-the-art MLLMs, demonstrating that models struggle to solve TempVS, with a substantial performance gap compared to human capabilities. We also provide fine-grained insights that suggest promising directions for future research. Our TempVS benchmark data and code are available at https://github.com/yjsong22/TempVS.

## introduction multimodal large language models (mllms) @xcite @xcite have demonstrated remarkable performance in various vision and language tasks. at the same time, the need for standardized evaluation frameworks has become increasingly critical. most existing benchmarks focus on settings involving a single image @xcite @xcite @xcite . while some benchmarks consider multi-image settings @xcite @xcite , they mainly focus on cross-image recognition and reference. to date, relatively little attention has been paid to more complex tasks, such as grounded temporal understanding and reasoning in multiple images. some recent studies have assessed mllms' temporal comprehension across multiple images, but certain limitations remain. first, some tasks can be resolved by relying on a single image rather than a sequence @xcite . for example, given an image sequence, determining whether "pulling a wind-up toy so it continues moving forward or quickly stops" could be answered only based on the final image. second, some tasks depend heavily on commonsense or world knowledge @xcite , such as rearranging a set of shuffled images into the correct sequence of cooking steps. third, some benchmarks @xcite use distractor options absent from the images, allowing the models to infer ground-truth answers based on the presence of objects. these factors may result in the benchmarks failing to truly assess the model's understanding of temporal sequences. in addition, none of the existing benchmarks are designed for multi-event scenarios, making them inadequate for evaluating complex temporal sequences and relations in mllms. as a result, a question arises: do existing mllms really understand time by accurately aligning the order of events in language and image sequences? to address this, we propose tempvs, a benchmark for multi-event temporal grounding and reasoning in visual story image sequences. tempvs contains 2,085 image sequences (9,803 images) across cartoon animations, movies and daily-life albums, with 15,192 multiple-choice questions. tempvs features three temporal understanding and reasoning tasks: event relation inference, sentence ordering, and image ordering (as shown in figure 1 ). these tasks are accompanied by grounding tasks to check whether the model can match the exact image consistent with a single text description. we select image sequences making up visual stories, each containing several events that are temporally related yet relatively independent, in the sense that no event could easily be predicted from preceding events. this makes it hard to resolve the tasks without considering both linguistic and visual modalities. tempvs challenges models to reason about event order in the image sequence and text (e.g., a sentence describing two events using before or after, or a story) and integrate both. we extensively evaluate 38 mllms, including open-source models ranging from 0.5b to 78b (e.g., llava-onevision, internvl2.5, qwen2-vl, phi-3.5-vision, deepseek-vl2, llava-next-video) and the closed-source gpt-4o. we show that tem-pvs is highly challenging for sota models, especially on event relation inference and image ordering tasks. in particular, while models can accurately ground individual events to images, their performance on the main tasks, which require multimodal reasoning with sequences, remains unsatisfactory. we further analyze the impact of the choice of linguistic structure, distance between events, and chain-of-thought prompting. our analysis sheds light on future directions for improvement in architectural design, training objectives, and/or posttraining methods to enhance temporal reasoning. contributions. (1) we introduce tempvs, a new benchmark for evaluating multi-event temporal grounding and reasoning ability in image sequences for mllms. (2) we extensively evaluate 38 mllms from different model families and sizes, highlighting the performance gap compared to human annotators. (3) our findings in evaluation results and fine-grained analysis suggest potential pathways for future improvements. ## related work multimodal large language models progress in large language models (llms) @xcite @xcite has provided impetus to the development of multimodal llms which process both visual and textual information. state-of-the-art mllms @xcite @xcite @xcite @xcite are built upon llms with an integrated visual encoder and a connection module. these models surpass the earlier generation of multimodal models, which were typically based on bert-type architectures, on many downstream tasks @xcite . while some studies have focused on training mllms to interpret multiple images using interleaved image-text datasets @xcite @xcite , their capability to understand and reason about multievent temporal relationships in sequential visual data remains largely unexplored. ## the tempvs benchmark tempvs evaluates models' ability to understand and reason about temporal relations by evaluating the consistency between textual descriptions of temporally related events and the visual event order in an image sequence. to achieve this, we create three main tests (mt): event relation inference, sentence ordering, and image ordering. additionally, to investigate whether a model's difficulty arises from challenges in temporal understanding or from more basic grounding issues, we create a corresponding grounding test (gt) ( §3.1) for the main tests. we present the task curation process and statistics of tempvs benchmark in §3.2. tempvs is built from existing datasets pairing image sequences with narrative captions. since captions may vary in their level of detail, we use both the original captions in the source data, and simplified captions in which the main event is extracted from the caption (we describe this process in §3.2). in what follows, we denote an image sequence s consisting of n images, their corresponding captions, and extracted events as: statement template (@xmath0: e j after e i neg: e i after e j pos: e i before e j neg: e j before e i pos: e j . earlier, e i neg: e i . ealier, e j pos: e i . then, e j neg: e j . then, e i pos: e i . e j . neg: e j . e i . three-event pos: e i before e j , and after that, e k neg: e k before e i , and after that, e j ## conclusion in this paper, we present a novel and challenging benchmark tempvs, designed to assess multimodal, multi-event temporal reasoning abilities of mllms in image sequences. after evaluating 38 advanced mllms, we find that current models typically struggle with reasoning about temporal relations and rearranging shuffled images to the correct order based on a narrative. further analysis of linguistic structures, event distance, and chainof-thought reasoning has shed light on promising avenues for future work. our study contributes to mllm development by uncovering weaknesses in multi-event temporal reasoning in multi-image scenarios, while tempvs provides a valuable resource for further research. limitations despite a careful examination of publicly available repositories, technical reports and papers, we find no evidence that the evaluated mllms were trained on the data included in tempvs. however, for models (such as gpt-4o) that have not fully disclosed or explicitly stated their training data, the possibility of data leakage and contamination remains unclear. this could potentially lead to an overestimation of their advantages. additionally, we focus on multiplechoice questions to ensure structured evaluation and clear correctness criteria, following existing multimodal benchmarks @xcite @xcite @xcite . however, other question types, such as open-ended questions, particularly the evaluation of open-ended generation in multi-image temporal understanding and reasoning, are also worth exploring. we leave these for future work.