Beyond Single Frames: Can LMM s Comprehend Implicit Narratives in Comic Strip? (2025)

Abstract: Large Multimodal Models (LMMs) have demonstrated strong performance on vision-language benchmarks, yet current evaluations predominantly focus on single-image reasoning. In contrast, real-world scenarios always involve understanding sequences of images. A typical scenario is comic strips understanding, which requires models to perform nuanced visual reasoning beyond surface-level recognition. To address this gap, we introduce STRIPCIPHER , a benchmark designed to evaluate the model ability on understanding implicit narratives in silent comics. STRIPCIPHER is a high-quality, human-annotated dataset featuring fine-grained annotations and comprehensive coverage of varying difficulty levels. It comprises three tasks: visual narrative comprehension, contextual frame prediction, and temporal narrative reordering. % , covering various difficulty. Notably, evaluation results on STRIPCIPHER reveals a significant gap between current LMMs and human performance—e.g., GPT-4o achieves only 23.93% accuracy in the reordering task, 56.07% below human levels. These findings underscore the limitations of current LMMs in implicit visual narrative understanding and highlight opportunities for advancing sequential multimodal reasoning.

## introduction in the space between the panels, human imagination takes separate images and transforms them into a single idea. lmms have demonstrated impressive performance across various visual-language tasks such as image captioning @xcite ), text recognition @xcite , visual question answering @xcite and video understanding @xcite . however, the question remains whether these models can truly grasp the implicit narratives embedded in comics-a challenge that goes beyond conventional ocr and text reasoning. existing comic benchmarks @xcite @xcite @xcite have predominantly focus on instance level tasks like object detection and dialogue generation in textheavy comics, allowing lmms to rely on ocr and superficial text processing rather than deep visual comprehension. recent benchmarks @xcite evaluate single-frame comics with an emphasis on surface-level interpretation, neglecting the sequential dependencies of multi-frame narratives. motivated by these limitation, we focus on silent comic strips-compositions devoid of explicit textual dialogue and consisting of 3-8 frames. the transition from single images to sequential data is not just a logical progression-it mirrors realworld perceptual processes and aligns with emerging trends in multimodal research. video-based llms @xcite @xcite also preprocess video input by sampling key frames before performing inference. multi-frame comics occupy a distinctive niche between static images and dynamic videos, conveying complete narratives through a minimal, densely packed sequence of frames. much like keyframes in video analysis, these structured sequences create a controlled yet challenging environment for evaluating temporal understanding, causal relationships, and contextual reasoning in lmms. we introduce stripcipher, a novel benchmark designed to assess the implicit temporal-visual reasoning ability of lmms. stripcipher consists of three challenging tasks: (1) contextual frame prediction task to eval visual contextual reasoning, (2) visual narrative comprehension task to infer overall implicit narrative, moving beyond local cues (3) temporal frames reordering, as shown in figure 1 . we utilized gpt-4o to preprocessing and generate answers and distractors, then employed human annotators double-verify and remove low-quality entries, resulting in 2,170 samples. the dataset includes classic comics like "peanuts" as well as recent published "buni", ensuring diversity. our comprehensive evaluation of 16 state-of-theart lmms on stripcipher reveals a substantial performance gap compared to human capabilities in sequential image comprehension, especially in the frame reordering task. most notably, gpt-4o achieves only 23.93% accuracy in the reordering subtask and trails human performance by 30% in visual narrative comprehension. further quantitative analysis identifies several key factors affecting the sequential understanding performance of lmms, highlighting the fundamental challenges for future lmm development. our key contributions are as follows: we introduce stripcipher, the first benchmark that evaluates lmms on implicit visual narrative reasoning with silent comics. we propose three novel tasks (narrative comprehension, frame prediction, and reordering) that challenge existing lmms beyond single-frame comic understanding. we comprehensive evaluat 16 state-of-the-art lmms, revealing substantial gaps between ai and human capabilities. ## related work large multimodal models llms have demonstrated exceptional performance in various natural language understanding and generation tasks @xcite @xcite . building on the scaling laws of llms, the next generation of lmms has emerged, utilizing llms as their backbone. several closed-source lmms @xcite @xcite , including gpt-4o @xcite , have shown remarkable capabilities in handling complex multimodal inputs @xcite . beyond single-image processing, models such as qwenvl2.5 @xcite and minicpm-o2.6 @xcite ) can handle multiple images, while video-based llms @xcite @xcite preprocess video content by sampling key frames before inference. additionally, state-of-the-art models like gemini @xcite and frameworks incorporating interleaved chain-of-thought @xcite explicitly account for the sequential order of both text and images in their reasoning processes. our approach extends from single to sequential images, mirroring real-world perceptual stage 1: image collection stage 2: data annotation stage 3: cross-examination frame split scrap comic from web task1: what is happening in this comic strip? the cats want to play... ## dataset and task overview to systematically evaluate the narrative comprehension capabilities of lmms on silent comics, we design a suite of tasks arranged in order of increasing complexity. each task is crafted to probe a distinct facet of visual and sequential reasoning: the instructions for three subtasks are presented in prediction: based on the overall story, what is happening in the blank frame (second-to-last)? comprehension: what is happening in this comic strip? what is the implicit meaning? based on your understanding, answer the question. reordering: the sequence of the comic strips provided below is incorrect. your task is to find out the correct order of the comic strips based on the storyline and temporal logical relationship. number each comic strip in the order they should appear, starting from 1. detailed statistics is displayed on ## dataset construction we construct our stripcipher dataset in a multistep crowd-sourcing pipeline, including 1) annotator training, 2) data annotation, and 3) crosscheck examination. an overall demonstration of our dataset construction pipeline is illustrated in figure 2 . ## analysis our analysis addresses the following questions: does fine-tuning with reorder task help? yes, it does. we fine-tune qwen2-vl using 3,160 samples for one epoch. this not only significantly improves performance on the reordering vqa task but also enhances comprehension tasks. to construct the training dataset, we applied data augmentation to 790 images using the reorder task. specifically, we randomly shuffled the sequence of images four times, generating a total of approximately 3,160 distinct samples. for evaluation on the reorder task, we used only the remaining 100 samples. for the comprehension task, we conducted a full test set evaluation, as the training data provided only images without any analytical content. meanwhile, we excluded the frame prediction task from testing due to potential data leakage. the experimental results are presented in the according to the table, qwen2.5-vl significantly benefits from fine-tuning, enhancing both reordering and comprehension tasks. improvements in generation tasks for reordering are greater than in choice tasks, likely because generation instructions are more challenging and unfamiliar, while choice tasks allow educated guesses from provided options. limited training samples also constrain improvements in choice tasks. in contrast, llava only improves in the reorder-choice task after training. does gpt-4o understand sequence images as well as humans? while gpt-4o achieves the highest performance among all tested models, there remains a substantial gap between its capabilities and human performance, particularly in novel tasks like frame reordering. through our preliminary data annotation experiments, we observed that while gpt-4o can comprehend basic comic content and provide interpretations, it frequently generates hallucinated content and struggles with comics that depict unconventional or imaginative scenarios rarely encountered in real life. in the visual narrative comprehension and nextframe prediction tasks, the multiple-choice format allows models to leverage similarity matching between options. our investigation revealed that model performance is heavily influenced by the quality of distractor options. in initial experiments with weak distractors (generated using gpt-4o with instructions to provide distractors with hallucinations, the prompt is followed hallueval), the model achieved accuracy rates up to 90%. upon analysis, we found these initial distractors were too obviously incorrect or irrelevant to the comic con-tent, making the selection task trivial. to address this limitation, we carefully curated a new set of challenging distractors. with these enhanced distractors, performance of gpt-4o decreased significantly to more realistic levels (61.60% for understanding and 69.95% for prediction), better reflecting the true challenges in comic comprehension. the scores obtained from multiple-choice questions with semantically transparent options tend to be inflated. in subsequent reordering tasks, where options lack explicit semantic meanings, coupled with open-ended questions, the scores provided a more authentic assessment of the lmms. does input format of images influence performance? considering the distinct computational pathways that lmms employ in processing individual versus multiple images, we designed following experiments to measure the differential impact of varied input formats using qwen2.5vl as our test case. we compared three input formats: (1) whole image -the entire comic strip as a single image, (2) sequential frames -individually separated frames input in order, and (3) shuffled sequenceseparated frames input in random order. this consistency suggests that while separated frames might theoretically help models extract clearer information from each panel and avoid visual confusion from complex layouts, the current video-like processing mechanism used by lmms for multiple images might not fully capitalize on these advantages. the similar performance with shuffled sequences further indicates that models rely more on individual frame content rather than sequential relationships for comprehension tasks. ## conclusion we present stripcipher, a comprehensive benchmark for evaluating large multimodal models' capabilities in visual comic sequence reasoning. our benchmark comprises meticulously curated § the prompt is displayed in figure