Can you SPLICE it together? A Human Curated Benchmark for Probing Visual Reasoning in VLM s (2025)

Abstract: In this work, we introduce SPLICE, a human-curated benchmark derived from the COIN instructional video dataset, designed to probe event-based reasoning across multiple dimensions: temporal, causal, spatial, contextual, and general knowledge. SPLICE includes 3,381 human-filtered videos spanning 12 categories and 180 sub-categories, such as sports, engineering, and housework. These videos are segmented into a total of 11,423 event clips. We evaluate both human participants and state-of-the-art vision-language models (VLMs) on the task of rearranging these clips into coherent event sequences to assess visual reasoning capabilities. Results reveal a significant gap: VLMs struggle to match human performance. While human-annotated textual descriptions improve model accuracy, they do not affect human performance, suggesting that models rely more on language priors than on visual understanding. Even with annotations, VLMs fall short of human-level reasoning, underscoring persistent challenges in visual reasoning. A deeper analysis across sub-categories shows that VLMs perform relatively better on videos where temporal and causal reasoning are dominant, compared to those where contextual and spatial reasoning are dominant. They also perform better on everyday tasks than on specialized ones.

## introduction transformer-based models @xcite initially focused on pre-training with language data alone @xcite @xcite . they later evolved to multimodal pre-training with the introduction of patchbased training @xcite . since then, vision large language models (vlms) have advanced rapidly, increasingly matching or even surpassing human performance across various domains, including coding, mathematics, scientific knowledge, * these authors contributed equally to this work and reasoning. for example, benchmarks like arc-agi @xcite , where models scored 0% in 2019, now report state-of-the-art models achieving scores between 33% and 55.5% @xcite . while this remarkable progress in reasoning capabilities is essential for enhancing the utility of current ai systems, our understanding of how well these models reason about purely visual sequences remains incomplete. unlike textual reasoning tasks, where progress is well-documented, the field lacks benchmarks that rigorously evaluate visual reasoning without heavy reliance on language priors. alongside advancements in performance, vlms have become significantly more efficient, enabling them to process long videos. for instance, opensource models like qwen2-vl @xcite can understand videos exceeding 20 minutes in length and handle multiple input videos simultaneously. however, despite these capabilities, current benchmarks do not sufficiently probe models' ability to infer event sequences from purely visual cues. leveraging this new capability, we propose an intuitive yet challenging benchmark, splice * (sequential processing for learning and inference in chronological events), where the task is to order shuffled clips, cut from original videos, based on the events depicted in them. splice fills this gap by leveraging a dataset where the correct ordering of clips requires multiple types of reasoning, such as causal, temporal, spatial, contextual, and general knowledge reasoning. unlike previous datasets that rely on automatic or unsupervised segmentations, splice is constructed through a rigorous human curation process. we adapt the coin instructional video dataset @xcite , which was originally created for video understanding and event localization, by extracting 3,600 videos spanning 180 tasks across 12 domains (e.g., vehicles, gadgets, cooking). in the "vehicles" domain, for instance, tasks may include changing tires, lights, or fuses. coin provides event-based annotations, which we use to segment each video into distinct clips before shuffling them. this repurposing necessitates careful filtering and validation to ensure that only meaningful and welldefined tasks remain. consequently, splice eliminates ambiguous or trivial sequences, making it a stronger probe of true visual reasoning. this event-based structure aims to prevent models from relying solely on the first and last frames or other shortcuts, instead requiring deeper reasoning. for instance, a karate practice video may be divided into three clips: an opening salutation, practicing movements, and a closing salutation. since the salutation clips are visually identical but occur at different points, the model must rely on other cues such as breathing, sweat levels, spatial positioning, or background actions to determine their order. in this work, we compare the performance of multiple state-of-the-art models that support multivideo input, including qwen2-vl @xcite , gemini-flash (gemini team, 2024), in-ternvl2.5 @xcite , and llavaonevision @xcite , across three different input settings: videos only, text only, and videos+text. additionally, we provide human performance benchmarks and compare them with the performance of * https://huggingface.co/datasets/prokajevo/ splice-benchmark . the main contributions of this paper can be summarized as follows: ## related work as this work evaluates vlms on various aspects of reasoning, we provide a comprehensive review of reasoning evaluations for vlms, along with tasks similar to our approach, where videos or images were shuffled and reordered. ## types of reasoning cutting, shuffling, and reordering clips may seem like a simple task requiring basic temporal reasoning. however, when applied to event-based instructional videos, it demands a richer set of reasoning skills. below, we outline key reasoning types, with examples from our benchmark: contextual reasoning: understanding the context of actions to predict what follows. e.g., when changing a printer cartridge, opening the door is typically followed by replacing the cartridge and then closing the door. spatial reasoning: interpreting spatial relationships and orientations. e.g., aligning a car to park or tracking an athlete's movement across pole vault stages. general knowledge reasoning: applying everyday knowledge to infer plausible actions. e.g., drying follows washing potatoes, or avoiding a hot plate after oven use. temporal reasoning: grasping the correct sequence of events. e.g., assembling a sofa by attaching legs before placing the mattress. causal reasoning: recognizing cause-effect links. e.g., mixing flour and water forms dough; cutting a bike chain results in it falling loose. ## results the performance of the models compared to humans is shown in in terms of binary accuracy, where a prediction is considered correct only if it exactly matches the ground truth, humans score 84.86%, while the highest-performing model, gemini-2.0-flash-exp, scores 51.08%, and random accuracy is 21.14% when using video-only input. this demonstrates that although the model performs well above random accuracy, it still lags behind human performance. in contrast, while human performance did not benefit from the additional text modality, models showed substantial improvement. even on videos that humans misordered, models rarely outdo them. for instance, qwen2-vl-72b solved none of the 57 seven-clip videos humans got wrong, while gemini-2.0-flash-exp solved only three. likewise, out of 77 six-clip videos misordered by humans, qwen2-vl fixed two, and gemini-2.0-flash-exp five, highlighting the persistent gap in visual reasoning. ## conclusion in this paper, we presented a novel, human-curated benchmark designed to assess multiple facets of visual reasoning, including temporal, causal, contextual, visual-spatial, and general knowledge. we evaluated various open-source and closed-source vlms under different input modalities and compared their performance against human participants. despite improvements from combining video and text (highlighting the value of cross-modal alignment), all models lag significantly behind human performance, especially without human-annotated descriptions. moreover, open-source models lag further behind their closed-source counterpart, revealing a persistent gap in visual reasoning. the low performance of several models, with some scoring just above random chance, highlights the bench-mark's effectiveness as a rigorous probe of visual reasoning. our reasoning analysis further reveals that models perform better on videos dominated by causal and temporal reasoning than on those requiring contextual or spatial reasoning. in the future, we aim to incorporate voice to enhance cross-modal alignment and assess how models integrate audio with visual reasoning.