Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation (2025)

Abstract: Internal world models (WMs) enable agents to understand the world’s state and predict transitions, serving as the basis for advanced deliberative reasoning.Recent large Vision-Language Models (VLMs), such as GPT-4o and Gemini, exhibit potential as general-purpose WMs. While the latest studies have evaluated and shown limitations in specific capabilities such as visual understanding, a systematic evaluation of VLMs’ fundamental WM abilities remains absent. Drawing on comparative psychology and cognitive science, we propose a two-stage framework that assesses **perception** (visual, spatial, temporal, quantitative, and motion) and **prediction** (mechanistic simulation, transitive inference, compositional inference) to provide an atomic evaluation of VLMs as WMs. Guided by this framework, we introduce **WM-ABench**, a large-scale benchmark comprising 23 fine-grained evaluation dimensions across 6 diverse simulated environments with controlled counterfactual simulations. Through 660 experiments on 15 latest commercial and open-source VLMs, we find that these models exhibit striking limitations in basic world modeling abilities. For instance, all models perform at near-random accuracy when distinguishing motion trajectories. Additionally, they lack disentangled understanding—e.g., they tend to believe blue objects move faster than green ones. More rich results and analyses reveal significant gaps between VLMs and human-level world modeling.

## introduction world models (wms) in agents provide internal representations of the external world @xcite . they simulate how the current state transforms to the next @xcite , enabling agents to perform deliberative planning by predicting probable future states and choosing the most favorable one @xcite @xcite @xcite . different environments operate under distinct mechanisms and dynamics-for example, the transition mechanics of autonomous vehicles, surgical robots, and rover-based spacecraft vary significantly. prior work handled this diversity by building environment-specific world models, but they failed to adapt across various real applications. recent large-scale vision-language models (vlms; @xcite @xcite , inter alia) enhance generalist llms with visual semantics and encapsulate extensive knowledge of world dynamics, making them promising potential world models for general domains. unlike video generative world models which directly generate the next states, vlms can reason in their latent representation space and forecast via language. however, their language grounding and world simulation capabilities may still be insufficient in various aspects. for example, existing benchmarks have revealed their vulnerability in visual or spatiotemporal perception @xcite @xcite and future state prediction driven by intuitive physics @xcite @xcite . these limitations underscore the need for a more systematic evaluation. a robust world model must integrate multiple fundamental abilities in perception and prediction, yet previous studies that assess these aspects in isolation provide only a partial view. to address this, we propose an atomic evaluation framework, systematically testing the essential aspects (and their interactions) of a vlm's internal world model from first principles. with theories and evidence from comparative psychology and cognitive science @xcite @xcite , we first present a systematic conceptual framework to formalize the functioning of world models (figure 1 ). we decompose the process into two stages: (1) perception stage, which involves visual, spatial, temporal, quantitative, and motion perceptions @xcite @xcite ; and (2) prediction stage, which involves mechanistic simulation, transitive inference, and compositional inference @xcite @xcite . following this framework, we create wm-abench, the world model atomic benchmark that covers 23 fine-grained dimensions (figure 2 ) of world modeling and over 100,000 instances curated from 6 different simulators @xcite @xcite @xcite . by systematically manipulating environmental factors and simulating counterfactual actions, we generate incorrect states for models to differentiate from the correct ones to ensure controlled studies. we also measure human performance to verify the fairness and solvability of our problems. we conduct 660 experiments on 15 state-of-theart vlms and find that, while they excel in certain aspects of visual and quantitative perception, they are surprisingly limited in many other dimensions (figure 1 ). specifically, vlms exhibit (1) weak perception of space, time, and motion; (2) insufficient knowledge of physical causality in intuitive physics and agentic actions; (3) limited transitive and compositional reasoning capabilities, showing near-random accuracy. our further analyses reveal a lack of independent and robust world representations in vlms, e.g., mistakenly associating color with speed. these findings reveal significant gaps between current vlms and human-level world modeling, shedding light on the need for deeper understanding, grounding, and reasoning over the perceptual world and its transition mechanisms before vlms can truly serve as generalist world models. ## the dual-stage conceptual framework we view world modeling as a two-stage process of perception and prediction @xcite . in the first stage, agents form internal representations of the current state by sensing and encoding environmental stimuli. in the second stage, agents use these internal representations to extrapolate future states, refining their model whenever new ground-truth observations arrive. this dual-stage framework explains (1) how raw sensory signals are converted into compact world representations; and (2) how these representations then guide forward simulations. the two stages of wms are core functions in agents' advanced planning and decision-making. we brief our formulation here and leave the in-depth discussion in appendix a. ## the wm-abench benchmark following the above taxonomy, we design and implement corresponding tasks for benchmarking vlms as wms. figure 2 presents a visual overview of our complete benchmark tasks, offering readers a high-level understanding of the framework. ## related work world models. world models (wms) predict how the current state transitions to the next, based on prior states and actions @xcite . traditionally, people train framelevel video-generative models specializing in some narrow domains. for instance, in robotics, wms enable model-based reinforcement learning and trajectory prediction @xcite ; in autonomous driving @xcite , they facilitate path planning; and in gaming @xcite @xcite ,c), they power interactive simulations. meanwhile, recent work @xcite has explored whether video-generation models can serve as world simulators that go beyond mere pixel-level synthesis. we instead investigate whether vision-language models can capture world dynamics from large-scale training data, enabling them to function as generalist world models. ## conclusion our study provides the first atomic evaluation of vlms' internal world modeling abilities with a cognitively inspired framework. while vlms excel in scenarios with pronounced differences, they struggle with 3d and dynamic perception, fail to differentiate subtle physical distinctions, and exhibit failures in understanding world transitions of transitive and compositional scenarios.