M o M ent S : A Comprehensive Multimodal Benchmark for Theory of Mind (2025)

Abstract: Understanding Theory of Mind is essential for building socially intelligent multimodal agents capable of perceiving and interpreting human behavior. We introduce MoMentS (Multimodal Mental States), a comprehensive benchmark designed to assess the ToM capabilities of multimodal large language models (LLMs) through realistic, narrative-rich scenarios presented in short films. MoMentS includes over 2,300 multiple-choice questions spanning seven distinct ToM categories. The benchmark features long video context windows and realistic social interactions that provide deeper insight into characters’ mental states. We evaluate several MLLMs and find that although vision generally improves performance, models still struggle to integrate it effectively. For audio, models that process dialogues as audio do not consistently outperform transcript-based inputs. Our findings highlight the need to improve multimodal integration and point to open challenges that must be addressed to advance AI’s social understanding.

## introduction throughout our lives, we continuously generate hypotheses about other people's emotions, knowledge, and a range of other mental states; these hypotheses guide how we understand and interact with others. this ability, known as theory of mind (tom) @xcite , is essential for interpreting behavior at the individual level and fundamental to coherent human social interaction @xcite . humans rely on more than just language to express their mental states. gaze, facial expressions, body posture, gestures, and vocal cues all play an important role in communicating how we feel and what we think. this combination of verbal and non-verbal cues provides relevant multimodal information to infer mental states of others @xcite @xcite . for artificial agents, this information can serve as multimodal input that enhances socially intel-question why does the woman tell the little girl that she cannot come on the journey? tom abilities non-literal communication ## related work prior benchmarks for tom broadly fall into two categories: text-only and multimodal. traditional text-only benchmarks, such as tomi @xcite and hi-tom @xcite , predominantly focus on probing models' ability for nested belief tracking and logical inference through text stories lacking realistic social context. tombench @xcite expands beyond belief tracking alone, incorporating a broader taxonomy of social and pragmatic tom tasks (e.g., faux-pas detection, persuasion, hidden emotions, desires) within everyday textual scenarios. despite this richer coverage, it remains constrained by its purely textual format, lacking multimodal information critical to human social understanding @xcite . multimodal approaches such as mmtom-qa @xcite present procedurally-generated videos of single actors in household tasks, primarily evaluating goal and belief inferences without meaningful social interaction or emotional complexity. similar to the text-only evaluations discussed above, this setup fails to reflect the depth and nuance of genuine human social behavior, limiting its applicability in evaluating socially intelligent ai systems @xcite . from the social intelligence perspective, social iqa @xcite probes social and emotional intelligence of models through multiple choice questions that require reasoning about social motivations, reactions, and actions based on specific situations. sotopia @xcite evaluates how models navigate complex social scenarios and achieve social goals. emobench @xcite measures emotional intelligence by assessing models' ability to understand and apply emotional knowledge in complex social scenarios. however, these works are again limited to text-only evaluations and do not measure tom directly. social genome @xcite (based on socialiq2 @xcite ) addresses the evalua-+ moments annotation pipeline desires nlc percepts knowledge beliefs intentions emotions assigned tom abilities: assigned short films: question annotator from our annotators pool iteratively refined with our reviewing feedback loop annotated tom dataset distractor reviewer from our research team question reviewer from our research team distractor annotator from our annotators pool & llm copilot q a an no tat ion 6 a n n o ta ti n g t o m d is tr ac to rs . 9 r efi n in g th e q a d is tr ac to rs . ## r efi ni ng the qa based on fe ed ba ck . human reviewers focused on the overall quality, while human annotators leveraged the llm copilot to minimize shortcut effects. tion of social interaction understanding in vlms through video-based multiple-choice questions, but videos are limited to 60 second clips, and evaluation is not designed to evaluate tom. furthermore, @xcite observed a strong bias in the representations of correct and incorrect answer candidates, where llms can achieve high accuracy with no context at all. given the limitations in prior work, there is a need for evaluating tom within realistic multimodal settings, capturing authentic social interactions beyond goals and beliefs alone @xcite . ## dataset design recognizing the limitations of previous benchmarks, we design moments based on two core principles: (1) an established taxonomy of socially relevant tom abilities -emotions, non-literal communication, desires, intentions, knowledge, percepts, and beliefs-to evaluate tom beyond the commonly addressed belief and goal probing abilities, and (2) long-form videos with real human actors that provide sufficient context and multimodal signals (e.g., facial expressions, gaze, body language, speech tone) to characterize interpersonal dynamics and mental states. this section outlines our taxonomy for probing different tom abilities, the video selection process, and the annotations included in each question. ## annotation methodology creating multiple-choice questions for this task is challenging. annotators must understand different tom abilities, find relevant moments in short films, and write clear questions. making good distractors is also difficult because humans often create distractors that models can easily guess without seeing the video context, as observed by @xcite in other multimodal social understanding datasets. to address these challenges, we conducted two pilot annotation rounds (see appendix a.2) before launching the main annotation phase. findings from the pilots helped us refine our pipeline to address the cognitive demands of tom question creation, reduce annotation biases, and ensure question quality. the final methodology included carefully structured annotation phases, refined guidelines, and a custom-built platform to support robust distractor generation. ## experimental evaluations we conduct experiments to evaluate the performance of current multimodal models in inferring mental states and to identify the factors that influence their performance. specifically, we aim to answer: (i) how well do these models perform overall and across different tom abilities? (ii) to what degree does visual information and context length impact performance? and (iii) how effective is our llm-in-the-loop distractor creation platform at mitigating answer set biases? to this end, we report model accuracies on moments, ablate the effect of the visual modality and context window length, and assess performance in a no-context setting against baselines lacking bias-mitigation mechanisms. ## open challenges for future model development our evaluations on moments suggest that current limitations in multimodal tom performance may stem not only from the reasoning capabilities of large language models, but also from how these systems access and process multimodal evidence. our findings point to several technical factors that likely limit models' ability to reason about mental states in socially rich scenarios. in this section, we outline four open challenges that, if addressed, could foster progress toward building better social multimodal agents. ## conclusion we introduced moments, a benchmark that probes seven tom abilities in realistic, long-form videos. it contains over 2,300 human-annotated mcqa items with substantially reduced biases in answer sets compared to prior datasets. from baseline experiments with video, audiovisual, and speech llms we observe: (i) visual input offers consistent yet modest gains, indicating underutilization of visual cues; (ii) using audio inputs does not yield a noticeable improvement over transcripts, suggesting that current models still have struggle to effectively integrate this modality; and (iii) performance tends to drop on extended context windows, highlighting limitations in long-range video reasoning. based on these results, we identify several open challenges that likely constrain progress on multimodal tom tasks, ranging from multimodal alignment and audio processing to frame selection and reasoning over multimodal evidence. addressing these issues will be essential for developing ai systems capable of truly understanding, predicting, and responding to human mental states in realworld social settings.