🧠 Mental World Modeling (Project Page)

From simulating physical scenes to simulating the minds that act within them.

TL;DR: A world model can reconstruct the physical scene correctly and still predict the wrong human action.

Why? Because human decisions are shaped not only by objects, geometry, and physical dynamics, but also by hidden mental and social variables: what someone has observed, knows, believes, wants, intends, feels, and considers socially permissible.

We introduce Mental World Modeling (MWM), a general framework that makes these mental variables part of the world state itself—rather than treating them as post-hoc explanations.

The world’s next state is not only physical. It is also mental.

The missing half of a world model

Most existing world models focus on the physical substrate of the world: What objects and agents are present? Where are they? How will the visible scene evolve?

But two physically identical scenes can produce completely different actions when the people inside them hold different beliefs, goals, emotions, relationships, or social obligations.

MWM therefore maintains a coupled physical–mental world state. For every target agent, it:

1. represents both the physical environment and the agents’ latent mental states;
2. renders a target-specific partial observation—what that person can actually see, hear, know, and infer;
3. simulates how each candidate action changes both the physical world and the mental-social world.

MENTIS: an inspectable implementation

We instantiate the framework in MENTIS, a training-free and fully inspectable baseline that forces an LLM-based system to operate as a mental world model.
MENTIS decomposes decision prediction into explicit stages:

state parsing → target-observation generation → action decomposition → coupled transition simulation → branch-level evaluation → final decision

What do the experiment results show?

The results reveal a consistent pattern:

• Full MWM achieves the strongest decision-prediction performance for every evaluated model.
• Removing the mental channel degrades all models.
• The gains are largest in interpersonal situations, where hidden beliefs, intentions, emotions, and norms determine the action.
• Explicit intermediate structure substantially improves multimodal reasoning.
• The largest remaining bottleneck is transition simulation: predicting how an action changes the coupled physical–mental world.

Why does this matter?

• A physically possible action is not necessarily the action a person will take.
• The same movement can represent help, pressure, deception, politeness, avoidance, or trust depending on the mental and social state surrounding it. This distinction matters for embodied assistants, human–AI collaboration, education, care, interactive agents, and any AI system expected to operate around people.
• MWM does not claim direct access to private consciousness. Mental states are treated as approximate, task-relevant hypotheses that must remain inspectable, revisable, and responsibly used.