Language-Informed Synthesis of Rational Agent Models for Grounded Theory-of-Mind Reasoning On-the-fly (2025)

Abstract: Drawing real world social inferences usually requires taking into account information from multiple modalities. Language is a particularly powerful source of information in social settings, especially in novel situations where language can provide both abstract information about the environment dynamics and concrete specifics about an agent that cannot be easily visually observed. In this paper, we propose Language-Informed Rational Agent Synthesis (LIRAS), a framework for drawing context-specific social inferences that integrate linguistic and visual inputs. LIRAS frames multimodal social reasoning as a process of constructing structured but situation-specific agent and environment representations – leveraging multimodal language models to parse language and visual inputs into unified symbolic representations, over which a Bayesian inverse planning engine can be run to produce granular probabilistic judgments. On a range of existing and new social reasoning tasks derived from cognitive science experiments, we find that our model (instantiated with a comparatively lightweight VLM) outperforms ablations and state-of-the-art models in capturing human judgments across all domains.

## introduction making sense of any real social situation requires integrating many different sources of information. language, in particular, can fundamentally recast our understanding of the social environment around us. being told the abstract dynamics underlying a social institution, from the rules of american football to the norms of drive-through restaurants, gives us a useful overarching picture of people's goals and intentions in unfamiliar settings. other times, language can provide specifics about particular people and environments. hearing that a friend tends to get hungry around midnight and keeps a spare stash of chocolate in the highest pantry shelf, for instance, gives new meaning to a few glimpses of someone bumbling around the kitchen in the dark. these tidbits of socially-relevant information from language allow us to draw much richer, more flexible, and often quite situation-dependent conclusions about the behavior we see. how do we integrate language with perceptual information to support this kind of grounded but often highly ad-hoc social reasoning, in which language can flexibly restructure our inferences about the agents we observe? this setting poses particularly acute challenges for two dominant flavors of computational work in social reasoning. for approaches that cast social reasoning as principled inferences over structured models of agents and environments, as in many symbolic ai and cognitive science frameworks (e.g., @xcite , this setting tests the scalability and breadth that can be achieved using (usually hand-engineered) symbolic models of particular environments and domains. for approaches that use large-scale neural models trained on language and visual inputs, like recent llm and vlm-based systems (e.g., @xcite , this setting tests the generalizability of complex decision-making and latent inference over particularly novel inputs. ongoing evaluations suggest that each component of grounded, ad-hoc social reasoning poses challenges for both dominant approaches @xcite @xcite -challenges that only compound in the harder multi-modal setting. in this paper, we develop language-informed rational agent synthesis (liras), a framework that can integrate linguistic and visual inputs to draw ad-hoc, probabilistic inferences about agents' mental states in grounded settings. our approach achieves this by using neural models to parse language and visual inputs into unified symbolic representations, which support automatic bayesian inference and inverse planning. impor- ## language-informed rational agent synthesis (liras) in this paper, we consider how to solve ad-hoc social reasoning tasks that are grounded in visual observations of an agent taking actions over time. each task (l, q, v 1:t ) is defined by a linguistic description l of the agent and its environment, a social inference query q expressed in natural language, and a sequence of t + 1 video frames v 0:t showing how the agent interacts with the environment over time. given (l, q, v 0:t ), liras produces @xmath0:@xmath1.g. goals, beliefs, etc.) based on the query q. since probabilistic social reasoning lacks a "ground-truth" answer @xcite , we assess task performance based on how human-like the ratings r 1:k are, measuring correlation with a dataset of collected human responses (see section 3). to solve these tasks, we propose language-informed rational agent synthesis (liras, figure 1 ). liras differs from social reasoning methods which rely on either prompting a single llm @xcite or scaffolding of llm calls within a structured probabilistic inference procedure @xcite @xcite . instead, given a language description l, liras synthesizes a rational model of an agent and its environment -which assigns likelihoods to an agents' actions by solving the (partially observable) markov decision process ((po)mdp) @xcite that describes the agent's decision problem -and draws probabilistic inferences about mental states from the agent's actions via bayesian inverse planning with respect to that agent model @xcite @xcite . this process is decomposed into four components: (i) synthesis of an environment model (fig. 1b ); (ii) synthesis of a rational agent model (fig. 1c ); (iii) parsing environment states and agent actions (fig. 1d ); (iv) mental state inference via sequential inverse agent modeling (siam) (fig. 1a , right), a flexible engine for bayesian inverse planning that extends sequential inverse plan search @xcite . we describe each component below. ## results liras demonstrates human-like reasoning on social reasoning tasks across domains. we compare liras and baselines against human data. while the state-of-the-art multimodal reasoning model openai o3 exhibits stronger alignment with human judgments than lighter-weight models such as gemini flash, its performance still lags significantly behind that of liras and shows a much weaker correlation against human judgments (average @xmath2.63) on these classical cognitive social reasoning domains. these findings underscore a key limitation of contemporary vision-language models: even when extensively pre-trained and fine-tuned for complex reasoning tasks, they still face challenges in achieving humanlike multimodal social reasoning, particularly when prompted to interpret visual scenes conditioned on complex linguistic information about the domain. these results show that grounded theoryof-mind reasoning with both language and visual inputs is challenging for most state-of-the-art foundation models -even in classic relatively simple social reasoning domains, echoing recent findings by @xcite . human vs gemini 2.0-flash human vs gpt-4o human vs openai o3 human vs liras ``the player no longer possesses the key after unlocking the door.````a ll keys can be reused and they don't disappear after unlocking a door.`@xmath3. a key motivation for this work is to computationally account for how language shapes inferences about so to illustrate, we also show a qualitative example in figure 2 . in this example, there are four gems, a, b, c, d. the same visual stimulus were tested under dkg-single and dkg-reuse variants. in the single condition, the openai o3 model finds a and b to be similarly likely, where humans and liras both rate gem b to be more likely. this is because the agent would have gone to get the closer blue key(s) on the left, if they are aiming for gem a, c, or d. under the dkg-reuse condition, where the participants are told that each key can be reused after unlocking doors, human judgments change significantly. liras is able to capture this shift in probability distribution, reasoning that now the agent is equally likely to go for gem a or gem b, since the top right blue key can now unlock both blue doors to get to gem a. on the other hand, all other baseline models judge only gem b to be the most likely option. ## discussion in this paper, we propose liras, a multimodal model synthesis architecture capable of synthesizing agent and world models "on the fly", from visual and linguistic inputs, to reason about agents' mental states from observation. we test our model on a variety of popular cognitive science domains with different aspects of social reasoning under different settings. like people, our model flexibly adapts to variations in the underlying rules of the scenario at hand. our work provides a computational account of how language can help construct an ad-hoc world model for people to contextualize and interpret other agents' behavior. on the other hand, despite using gemini 2.0 flash, a lightweight vlm, in the model architecture, liras is able to match or outperform stateof-the-art multimodal reasoning models on most social reasoning tasks, and significantly exceeds the performance of the same gemini 2.0 flash model that liras uses for model synthesis. by constructing these ad-hoc world and agent models on-the-fly, our study is a step towards a generalized cognitive model capable of human-like flexible social reasoning navigating the social world and form more effective human-ai thought partners @xcite . ## limitations our work is not without limitations. first, our current approach is restricted to discrete domains and does not extend to continuous spaces, reflecting a major limitation inherent in the pddl framework. additionally, modeling multiagent scenarios remains challenging, particularly in competitive settings; our framework cannot yet adequately capture the complexities of multiagent interactions. while we found that a single set of prompts can handle all four tested domains, the generalizability of this approach to novel domains is not guaranteed, due to possible issues with domain-specific syntactic or semantic mismatches. in addition, the liras model currently parses gridworld domains by enumerating each cell, which does not generalize to more complex visual inputs. lastly, our model depends on explicitly provided linguistic information-such as a clearly enumerated action space and well-defined transition dynamics (e.g., specifying that agents cannot walk through buildings). in contrast, humans can often infer such rules implicitly from context, drawing upon commonsense knowledge to build mental models of new environments without explicit instructions. to address these limitations, future research could focus on scaling up and improving generalization by fine-tuning vision-language mod-els (vlms) with more diverse training examples across broader domains. this could enable more robust handling of new task domains on-the-fly. furthermore, enhancing the model's capacity to infer implicit domain constraints -possibly by incorporating structured priors -could make this approach more generalizable to cases where the linguistic information is ambiguous or incomplete.