O pen T o M : A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Models (2024)

Abstract: Neural Theory-of-Mind (N-ToM), machine’s ability to understand and keep track of the mental states of others, is pivotal in developing socially intelligent agents. However, prevalent N-ToM benchmarks have several shortcomings, including the presence of ambiguous and artificial narratives, absence of personality traits and preferences, a lack of questions addressing characters’ psychological mental states, and limited diversity in the questions posed. In response to these issues, we construct OpenToM, a new benchmark for assessing N-ToM with (1) longer and clearer narrative stories, (2) characters with explicit personality traits, (3) actions that are triggered by character intentions, and (4) questions designed to challenge LLMs’ capabilities of modeling characters’ mental states of both the physical and psychological world. Using OpenToM, we reveal that state-of-the-art LLMs thrive at modeling certain aspects of mental states in the physical world but fall short when tracking characters’ mental states in the psychological world.

## introduction theory-of-mind (tom), the awareness that others perceive the world differently and the capability of keeping track of such differences, is at the core of social interactions @xcite . studies in cognitive science have designed numerous false-belief tests to investigate human tom capabilities @xcite @xcite . one such test is the sally-anne test @xcite , in which anne stealthily moves an object that is initially known to both sally and anne. this covert action causes sally to have a false belief that the object is still in its initial location. consequently, individuals taking the test are required to reason about "where will sally look for the object?" this story features two protagonists: sam (observer) and amy (mover); and an entity-of-interest: rubber duck. there are two containers involved: a basket and amy's backpack. each narrative within opentom is followed by three types of questions, namely questions regarding the location (loc) of an entity, questions that involve multi-hop reasoning (mhop), and questions about the characters' attitude (att). to study neural theory-of-mind (n-tom) foot_1 , machines' capabilities of performing tom reasoning, researchers have applied human tom tests such as the sally-anne test to benchmark large language models (llms) @xcite @xcite @xcite @xcite . however, using human tom tests for evaluating llms is problematic because stories in human tom tests lack certain elements found in real-life scenarios. specifically, the characters do not have personality traits or preferences. additionally, their actions are not motivated (e.g. why would anne want to move the object?). furthermore, the narratives of many existing n-tom benchmarks are generated using a template-based approach @xcite @xcite , which results in overly-structured and ambiguous narratives (see appendix a.1). the structured context makes existing benchmarks susceptible to overfitting, while the ambiguities may lead to an underestimation of a model's true n-tom capabilities. to this end, we introduce openbook-qa dataset for tom (opentom). following previous works' success in generating high-quality data using llms @xcite @xcite , we generate opentom stories using a four-stage humanin-the-loop generation pipeline ( §2.1). our pipeline includes (1) endowing characters with preferences and personality traits, (2) generating intentions and the corresponding enctions @xcite , (3) constructing story plot and producing narratives using llms, and (4) revise and refine stories by human annotators. based on the opentom narratives, we formulate questions that cover characters' mental states of both the physical world (e.g., the location of an object) and their psychological states (e.g. character's attitude towards a particular action). see figure 1 for examples. we evaluate opentom dataset on a range of llms including llama2-chat @xcite , mixtral-8x7b-instruct @xcite , @xcite , and gpt-4-turbo (openai, 2023) under a zero-shot setting. we also test two prompting techniques, namely chain-of-thought (cot) @xcite and simulated-tom (simtom) @xcite . additionally, we fine-tuned a llama2-chat-13b model to serve as the fine-tuning baseline. our results show that, while fine-tuning and advanced prompting techniques improve models' n-tom reasoning capabilities, their performance in deducing the psychological states of characters is still far from human performance (section 3.3). we summarize our contributions as follows: the omissions of characters' personality, intention, and enaction in existing n-tom benchmarks makes it difficult to construct questions that inquire characters' mental states of the psychological world. to address this, each of the characters in open-tom stories is personified and acts with an intention (appendixa.2). recognizing that llms are good at utilizing spurious correlations such as lexical overlaps @xcite , we take extra effort in mitigating the potential spurious cues in opentom stories ( §2.5). ## experiments following the convention of previous n-tom studies, we focus on evaluating zero-shot performance of llms @xcite @xcite . ## detailed result analysis to further investigate llms' n-tom capabilities, we conduct in-depth analysis on llms' faithfulness in answering loc coarse and loc f ine questions ( §4.1), performance discrepancy of modeling the mental states of different character roles ( §4.2), and lack of capability in modeling characters' mental state of the psychological world ( §4.3). ## conclusion we introduce opentom, a comprehensive n-tom benchmark featuring long narratives with realistic characters and events, and a diverse range of questions that cover both physical and psychological aspects of n-tom. our evaluation of llms' n-tom capabilities on opentom reveals that while state-of-the-art llms perform well on some n-tom tasks, they are still far from human-level performance on tasks requiring emotion deduction.