MMT o M - QA : Multimodal Theory of Mind Question Answering (2024)

Abstract: Theory of Mind (ToM), the ability to understand people’s mental states, is an essential ingredient for developing machines with human-level social intelligence. Recent machine learning models, particularly large language models, seem to show some aspects of ToM understanding. However, existing ToM benchmarks use unimodal datasets – either video or text. Human ToM, on the other hand, is more than video or text understanding. People can flexibly reason about another person’s mind based on conceptual representations (e.g., goals, beliefs, plans) extracted from any available data. To address this, we introduce a multimodal Theory of Mind question answering (MMToM-QA) benchmark. MMToM-QA comprehensively evaluates machine ToM both on multimodal data and on different kinds of unimodal data about a person’s activity in a household environment. To engineer multimodal ToM capacity, we propose a novel method, BIP-ALM (Bayesian Inverse Planning Accelerated by Language Models). BIP-ALM extracts unified representations from multimodal data and utilizes language models for scalable Bayesian inverse planning. We conducted a systematic comparison of human performance, BIP-ALM, and state-of-the-art models, including GPT-4. The experiments demonstrate that large language models and large multimodal models still lack robust ToM capacity. BIP-ALM, on the other hand, shows promising results, by leveraging the power of both model-based mental inference and language models.

## introduction theory of mind (tom) is the cognitive ability to ascribe hidden mental states (e.g. goals, beliefs, and desires) to other individuals based on their observed behavior. a hallmark of human social intelligence, tom serves as the foundation for a wide range of social interactions and a pillar of commonsense reasoning @xcite . systems designed to safely and productively interact with humans in an open-ended manner, such as assistive robots (e.g., @xcite @xcite , ai teachers (e.g., @xcite , and autonomous vehicles (e.g., @xcite , would greatly benefit from incorporating tom reasoning capabilities. the recent advancements in machine learning, especially in the realm of large language models (llms), have spurred increased interest in assessing these models' aptitude for tom reasoning (e.g., @xcite @xcite @xcite @xcite @xcite @xcite . many of these assessments use either text-based or video-based benchmarks inspired by classic tom experiments in the cognitive science literature @xcite . while the recent tom benchmarks provide welldesigned, cognitively informed tools, they share several notable limitations. one such limitation is the dependence on massive training data, which raises the concern that these models work by finding data patterns in a way that deviates from humanlike tom reasoning (e.g., @xcite @xcite . this paper focuses on a different but related limitation: these benchmarks rely on unimodal data, either in the form of videos (e.g., @xcite , or textual descriptions of actions and environments (e.g., @xcite @xcite . but tom reasoning goes beyond merely text comprehension or video understanding. it is about forming a causal model of another person's mind, which connects mental variables to possible actions @xcite @xcite . such a model can infer mental what's inside the apartment: … the kitchen is equipped with a microwave, eight cabinets, … inside the microwave, there is a cupcake. there is a wine glass and an apple on one of the kitchen tables. there are water glasses, a bottle wine, a condiment bottle, and a bag of chips in inside the cabinets. … actions taken by emily: emily is initially in the bathroom. she then walks to the kitchen, goes to the sixth cabinet, opens it, subsequently closes it, and then goes towards the fourth cabinet. ## the bip-alm model to infer the mental state of a person based on video and text inputs, we propose a novel machine theory of mind method, bayesian inverse planning accelerated by language models (bip-alm), which builds on bayesian inverse planning (bip) @xcite . prior works have shown that bip can reverse engineer human tom reasoning in simple domains. bip-alm extends bip by (1) building unified representations about a scene, a person's actions, and the mental state hypotheses from multimodal inputs, and (2) finetuning a language model to efficiently conduct inverse symbolic planning, based on unified symbolic representations. figure 3 provides an overview of the method. we first extract symbolic representations of the states and actions from both the video and the text. we then align and fuse representations extracted from different modalities, to form a unified representation of the event and the physical scene. this unified representation allows us to use a principled method to infer a person's mental state based on inputs from any given modality. we then use an inverse symbolic planner to compare the two hypotheses extracted from the question and produce the answer. we introduce each module below and provide more details in appendix c.